IOS-XE is takes the best elements out of Internet Operating System (IOS) which has its roots in a closet at Stanford, and combines them with the most successful open source technology ever – Linux. Cisco is leveraging Linux virtualization technologies such as Kernel Based Virtual Machines to protect against operating system failures as well as to allow for In Service Software Upgrades (ISSU).

To really appreciate this, we first have to dive down into the overall architectural changes of the ASR1000. The largest change that Cisco has made was to implement separate forwarding and control planes. In the past, Cisco routers would have the processes responsible for forwarding traffic, and the processes responsible for configuring the router running on the same root operating system. The side effect of this is that if you want to upgrade the root operating system of your router, you are going to have interrupt the traffic flowing through it to do so, or have a physically separate route processor to take over while you rebooted. This is a big headache operationally, and effectively forced engineers to design in separate physical chassis to meet high uptime requirements.

What Cisco has done to address this, was to mirror changes made in their storage and carrier routing portfolios. Both of those product lines utilize the operating system to push commands into advanced processors that exist on the line cards themselves. The ASICS on the line cards are designed to work in a distributed fashion, so that production traffic never goes into up into the router processor (or sup engine). This in effect ensures that the control and forwarding planes can exist as independent elements.

If you look at the graphic above, you will notice 3 main zones. The upper zone is what we would normally describe as the control plane. This is where the higher level functions such as your routing processes, ssh daemons, snmp daemons, and shells live. In short, if you you configure or read something, you are going to do it here. The only time traffic flows through this plane is when you are doing a thing called process switching. keep in mind this is a rare occurrence and usually occurs because of an oversight in your network designs.

By separating the control and forwarding planes, this allows Cisco to basically run a management station on the router, that programs chip sets in the line cards on the fly. This in my opinion is where the true power of this architecture comes through. By separating the two functions the software engineers are free to utilize powerful open source technologies such as Kernel-based Virtual Machines, and the Linux kernel, while letting the integrated circuit engineers design blazing fast chips which allow full functionality at line rate.

What benefits should we receive from a virtualized control plane? First, in larger routing and switching chassis (including the top end of the ASR1000 line) you normally have physically redundant route processors (RP)/ supervisory engines(SUP). The operating systems on these RP’s synchronize many things, including configuration, process state, routing tables, security associations and much more. The primary reason for this, is if you have a failure in the active RP, you can failover to the standby RP without interrupting traffic flows.They also can be used to streamline the software upgrade process by only upgrading one RP at a time, and then gracefully transferring traffic to it. Once proper operation is verified, the backup RP can be brought up to the same code revision.In any production environment this is highly desirable, and helps immensely in the battle for five nines.

The ASR1000 takes the redundant RP concept seen in high end chassis, and allows you to implement redundant upgrades, as well as protection against software failure, with only one physical route processor. This is done by utilizing Linux kernel virtualization. Instead of running the control plane directly on the production hardware, a small kernel is inserted. Booting from that are two copies of IOS-XE. These run independently, and synchronize state and configurations just as if you had two physically separate route processors. What this means in operational English, is that where in the past, you would have to either have two devices, or a larger device with redundant RP’s to upgrade without disruption, you can now have that same ease of maintenance, in a much smaller (and at the end of the day, less total cost) package.

Below this is the forwarding plane.It plugs into to a high speed interconnected fabric which all line cards and RP’s are redundantly connected to. In the diagram above, this is the bottom level. Items in this plane include buffer memory, Cisco Express Forwarding (CEF) ASICS, and now the new QuantumFlow processor. This is normally where you would find your DCEF enabled line cards, fibre channel and Nexus7000 line cards, as well as the modules for the ASR1000 routers. When properly utilized, traffic should be relatively isolated to this tier, and function independently from the control plane.

The shining star of the ASR1000′s forwarding plane is a group of chips that is referred to as QuantumFlow. The QuantumFlow architecture itself merges Cisco’s strength in integrated circuit design, with its strengths in IOS software design. In the past, Cisco would design ASICS’s for specific functions, and then write commands down into them. This has worked very well, until they point that a new feature came out that couldn’t leverage the fixed configuration of an older ASIC. Your choice at that point was generally to process switch for that feature (which is slower, and honestly bad form), or upgrade your cards to the newer ASIC design. The QuantumFlow chipset approaches this problem from a new angle. The first chip in the set (Popeye) is designed to be field programmable in C, as well as no fixed internal pipelines. This combined with utilizing 40 cores running between 900 and 1200 megahertz allows the programmers to utilize parallel processing techniques to utilize an immense amount of processing power in real time.

To put things into perspective, remember when you got your first multi core laptop or desktop. You were able to say watch a DVD, as well as compile code at this same time, while continuing to have a responsive workstation. Now imagine what you could do with a 40 core processor. This is the kind of power that we are talking about. Now imagine, that not only is your workstation immensely powerful, but you could also offload common jobs such as running daily builds, or encoding videos to another machine (or in this case processor.

In the ASR1000 this processor is called Spinach (yellow are in the graphic above). And of course just like the cartoon, Popeye’s potential really comes to light when combined with Spinach. Spinach is a separate chip, that is used a a traffic manager. This chip handles queueing and quality of service, ensuring that the proper packets arrive at the proper time, as well as interconnecting with cryptographic offload engines so it can equally apply services to encrypted flows.

At the end of the day, the most important question is not how fast something is, or how cool it is. The question is what can it do for me? By leveraging this new architecture the ASR1000 is now able to do line rate inspection of traffic using Network Based Application Recognition (NBAR), Support 128,000 queues for deep quality of service, secure and encrypt data using zone based firewalls and embedded crypto engines, segregate traffic using MPLS, integrate advanced voice and video functionality, as well as providing fulling Netflow v9 support for all of the above. It provides all of these services in an always on solution utilizing Linux virtualization, as well as leveraging an flexible chip set architecture that allows for field programmable improvements in the future.

My hope is that after reading this article that you are in a better to understand how Cisco is leveraging open source technology and integrated circuit designs to improve the foundation of the internet. In upcoming articles I will be discussing design scenarios utilizing this features in this product, as well as highlighting other areas where Cisco is embracing both open source technology, as well as open architectures that can properly leverage projects such as Linux, Ntop, Wireshark and more. If this article has you interested in learning more about some of the technologies mentioned today, then I encourage you to check out some of the links below, or shoot me and email to be highlighted in a future readers questions article.

Learn more about Linux Kernel-based Virtual Machines

Learn more about Cisco’s ASR1000

Learn more about Cisco QuantumFlowSimilar Posts:

• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Application Extension API notes – Cisco Live 2008
• Zone based IOS firewalls
• Arista Networks – Their approach to cloud networking
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware

--Colin McNamara

Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers

GoDaddy, one of the nations largest registrars and hosting providers is distributing bloated and possibly insecure code that will cause the average user to more then double their hosting costs. In the pages below you will learn exactly what GoDaddy is doing to your server, how their support staff will try to upsell you, and the steps you need to take to ensure proper operation of your Virtual Dedicated Server.

Background

One of my new years resolutions this year was to consolidate hosting accounts into one virtual server (hosted). I had my domains, and my old hosting with Godaddy already so it was a no brainer to try out one of GoDaddys VDS (Virtual Dedicated Servers).

I went with their 29.99 a month package, with Centos5, unlimited domains, 10Gb disk, and 256 MB of memory. This should be perfectly fine for hosting a couple MySql driven sites, and a couple gallery instances. Let me emphasise this is only handling 4 active domains, two of which only have static HTML.

Provisioning was a breeze, from order to shell account only took 4 hours. I was provided with shell access, pre-configured yum repositories, and this web control panel – simple control panel, or TurboPanel (seems to have two different names). I was able to pop into GoDaddy’s control panel interface with a direct link from their server manager console, and was setting up domains in no time. (Let me throw this caveat out though, don’t buy this product for your mom’s hosting…. the documentation is horrible, and by horrible I mean non-existant).

So I get my server all set up a couple weeks ago, transfer all my files from 2 Cups Solutions and set up my new www.colinmcnamara.com site. Things go just fine, I changed over to wordpress as a CMS and am totally thrilled. My applications and email are working perfectly. Plus, I have a shell account at GoDaddy which is a very handy thing to have as a network engineer. Things are going so well, that I give my buddy Rick a Christmas present and get ricksdavis.com and el-cinco.net for him, and host it on my GoDaddy VDS.I go ahead and purchase the domain through godaddy’s domain manager. This is obviously GoDaddy’s core competency, and goes flawlessly as usual. Next step, I go into the Turbo Panel web interface. Let me give you a little background on TurboPanel. This is the “free” equivalent “to plesk or cpanel. It is actually ok for automating your domain provisioning, though if you are a normal skill level user, I would recommend paying the extra 9.99 a month for Cpanel.

Server Error

I open up my TurboPanel interface and go to provision ricksdavis.com into the domain manager. It comes back with the least descriptive error I have ever received.

Server Error
We are sorry, the system has encountered an error while processing your request.

Home

If you continue to receive this error, please contact your system administrator.

Your URL: /domain/edit.do

Error details:

CommandFailedException: Unable to get min/max uids
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUids:825
at c.g.t.f.systems.user.LinuxUserSubsystem.getMinUid:780
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUserInfo:670
at c.g.t.f.systems.user.LinuxUserSubsystem.getUserInfo:646
at c.g.t.w.actions.domain.ActionDomainEdit.process:84
at c.g.t.w.actions.AbstractSpringAction.execute:118
…
at c.g.t.w.filters.AuthorizedResourceFilter.doFilter:38
…
at c.g.t.w.filters.RequestPopulationFilter.doFilter:117
…

This is the most descriptive error ever right? it tells you what is wrong, has a link to the support system, and gives you actionable information…. I would say a resounding NO. This is a classic example of why friends don’t let friends program in Java. When I got this error last night, I was scratching my head. As an engineer, the first thing I will look at is the last change to the system. Coincidentally I had installed awstats two nights before, and looking at my change logs, I saw that I had upgraded my perl version. So, with no fast response to the support email from GoDaddy I chose to put in a server re provision request (fully automated) and restore from my backups. That process took about an hour, but afterwards I was back online with no errors. Eureka! I found it (I thought). I provisioned Rick’s domain, wordpress, gallery2 etc and then went to bed.

GoDaddy Support Response

Fast forward to this morning, and I finally recieve an email reply from godaddy support. The email is pasted below-

(I have replaced the agents name with John Doe. Tech support is a hard job and I see no reason to highlight him specifically)

Before re-provisioning the server, I went ahead and tried the old three finger salute (reboot) the error still existed. So even if this email would have came to me on time, it would not have helped.

But that is besides the point. lets dig into the solution

The agent suggested the following fix -

Restart tomcat – which I was NOT using for any of my web applications (not an ejb guy)

/etc/init.d/tomcat55 restart

Restart TurboPanel (or simple control panel now – they haven’t updated their init scripts)

/etc/init.d/turbopanel restart

GoDaddy tried to upsell me, instead of fixing their code

He gives the standard, run less stuff on your server speech (remember, I only have 4 domains on this server)

Now here is the kicker – To remedy this issue long-term, you could either setup a server with 512mb RAM, or upgrade to a Dedicated server.

What the heck is with that? I should not need an upgrade with only 4 domains on a server. Especially when in the setup the default Cpanel implies support of 30 domains. Is this Tech Support or a Sales Call?

But, since my server was working fine I don’t pay much attention to the email and move on with my life.

Fast forward an hour, and I am show Rick how to access all the features of his new site, and I figure that I need to change an email account on his domain to forward to his old account. Fine, this should take two minutes. I log onto the TurboPanel interface to put the email forward in.. and there it, a big useless error screen. What the heck is with that?

So this time I actually read the email, and try the fix. Results = nothing. GoDaddy offers shell access so I log in, run top and filter for %memory used. Low and behold, there is only 8142 bytes of memory free, and a Java process owned by root is using 300Mb of virtual memory, and 132Mb of real memory, and Tomcat is using 115Mb of virtual memory and 86Mb of real memory.

Let me translate this into English – Godaddy’s control panel application was using 218 Megabytes of the 256 Megabytes of memory I had purchased. That left me with 34 Megabytes of memory . Let me clarify this, I had paid for a virtual server with 256Mb of ram, up to 1000Mb bursted (which I think is their code for swap). I am only running 4 domains on this server, and two pop3 email servers. This should not be a problem.So what is the cause of the problem? I can sum it up, crappy Java programming. Someone decided to write this program in Java (probably easier to outsource) instead of optimising it to run on lean systems. Their code effectively takes up all the available memory. And on top of that, they are are running a webserver process as root… yes as root. It is like asking for your server to get hacked.

Now, that I am done ranting, let me highlight how to fix this problem.

If you are on Windows use the following procedure to get shell access to your GoDaddy VDS -

• you will want to download a ssh client called putty – Download Here
• Copy this file to your desktop, double click putty.exe , and you should see something like this -

• In the host name field I have www.yourdomain.com . replace yourdomain with your domain name.
• Click on the open button on the bottom right, and a shell should pop up, along with a warning that looks like this (you can click OK to the warning)

• Now skip past the linux section

If you are running Linux or Unix start here -

• Open up a command line terminal
• ssh using your godaddy simple control panel username example -

ssh your-godaddy-control-panel-username@www.yourdomain.com

• If this is your first time connecting to this server, you will be prompted to accept an unkown ssh key into known_hosts, choose yes to accept

Both Windows, Linux and Unix Continue Here –

• Enter your the password you use to access your GoDaddy Simple Control Panel Interface
• You will be presented with what looks like a DOS window, this is called a secure shell terminal. Type in the username you use to access your Godaddy Simple Control Panel and then hit enter

• Now type in the password that you use to access your Simple Control Panel Interface and hit enter

• Congratulations, if you see the window below you are now shelled into your virtual dedicated server.

Change to the Root user

• Now that you are shelled into a Linux device you need to escalate your privileges to get administrator level access. In the Unix world this user is known as Root. You can change to this user, and get full system privileges by using the following command.

su – root

• You will be presented with a password prompt, enter in the same password that you have used to log into your Simple Control Panel Interface
• You are now root, be careful with what command you enter under this user, as you can do some damage if you are careless

Clean out your servers memory

• Most recent Redhat direvatives (including Centos) utilize a tool called yum to add and remove packages. This is also true with your linux servers at GoDaddy.
• you need to install a tool called memhog, it is part of a package called numactl that is normally used to assign specific process’s to specific cpu’s in a multi-core system. We will be using it today to fix GoDaddy’s memory hogging application

yum install numactl

• choose yes to all the prompts, and numactl will automatically be downloaded and installed on your server

Stop GoDaddy Simple Control Panel, and Tomcat, and tell them not to start automatically when your server restarts.

• Tomcat is a special type of webserver for Java based applications. Godaddy uses it to run their control panel interface. 99.99999 percent of users will not need to use Tomcat. If you do need to use Tomcat then you are a technical user and will know what to do.
• In Linux, server applications are called daemons. The are executed by init scripts. We will use these scripts to turn off these server applications
• Turn off the TurboPanel daemon ( this is the process that runs your simple control panel web interface

/etc/init.d/turbopanel stop

• Next we need to turn off Tomcat

/etc/init.d/tomcat55 stop

• Now that we have these services turned off, we need to make sure that they don’t come back when we reboot the server. We can do this by using the chkconfig command.
• Stop the Simple Control Panel Interface from starting automatically by executing the following command

chkconfig turbopanel off

• Stop the Tomcat server from starting automatically by executing the following command

chkconfig tomcat55 off

Clean the mess GoDaddy made of your servers memory

• A couple steps back we installed numactl. The executable we wanted out of this package is memhog. Issue the following command to take your memory back. This command will overwrite 200 megabytes of your memory, allowing the rest of your applications to get access to that memory.

memhog 200m

• This will Clear out the memory that GoDaddy’s application took over, and allow the rest of your daemons to run fine.

Great, my server is running better now. But I want to use my Simple Control Panel Interface. How do I do that?

• Easy, all you need to do is temporarily start the turbopanel daemon. When you are done making changes, you can turn it off again

/etc/init.d/tomcat55 start

/etc/init.d/turbopanel start

• When you are done, don’t forget to turn it off

/etc/init.d/turbopanel stop

/etc/init.d/tomcat55 stop

*** UPDATE *** *** UPDATE ***

I managed to stay with Godaddy and use their Virtual Dedicated Servers for 12 months. However I have moved to a new hosting provider (rimuhosting) in the spring of 2009. My last straw with GoDaddy occurred when I noticed that even with my memory utilization under control my applications were performing poorly. I shelled into verify and found that the virtual disk I/O was severly limited.

Performance issues –

Simple command to determine directory size on GoDaddy VPS

[root@ip-72-167-15-128 home]# time du -hs
1.3G    .
real    0m49.013s
user    0m0.011s
sys    0m0.062s

Simple command to determine directory size on Rimuhosting VPS
[root@colinmcnamara home]# time du -hs
1.3G    .
real    0m0.343s
user    0m0.050s
sys    0m0.290s

Translated into english, it took 49 seconds to execute a simple command on my GoDaddy server, and it too 0.34 seconds (under 1 second) to execute the same command on my virtual server at Rimuhosting.

On top of that, I was paying $32 dollars a month to GoDaddy for a server with 256 megs of ram as well as a FTP backup account. When I created my short list of hosts, I noticed that I was not getting the best deal by staying with GoDaddy.

Pricing at Rimuhosting

• Virtual Dedicated Server 190 Megabytes RAM – $19.95
• Virtual Dedicated Server 400 Megabytes RAM – $29.95
• Virtual Dedicated Server 900 Megabytes RAM – $39.95
• Virtual Dedicated Server 1150+ Megabytes RAM – $49.95

One thing you may notice, is that for the same price as the base GoDaddy VDS ($29.95) you get 400 Megabytes of RAM instead of 256. (and they won’t put on a memory sucking turbopanel app) On top of that, things that I had to pay extra for at GoDaddy such as backup space, backup mail relays, and DNS service came bundled for free.

I made the choice to move to Rimuhosting, and the service has been superior to GoDaddy in every single way. If you get sick of GoDaddy like I did, you might want to check Rimuhosting out, I think you’ll like them.Similar Posts:

• What should I do this thanksgiving break?
• Zone based IOS firewalls
• Routers can email you when they go down
• New features in VMware 3.1
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Identity aware networking using Cisco TrustSec

--Colin McNamara

Why GoDaddy Linux Virtual Dedicated Hosting Sucks & How to Fix It

YOUSERVER=`cat /var/lib/YaST2/you/config | grep LastServer= | sed ‘s/LastServer=”//g; s/;;”//g’`
YOU64=$YOUSERVER/$PLATFORM/update/SUSE-CORE/9/rpm/x86_64
YOU32=$YOUSERVER/$PLATFORM/update/SUSE-CORE/9/rpm/i586

if [[ `uname -i` = x86_64 ]]; then
echo $YOU64/kernel-source-$KERNELVER.x86_64.rpm

else
echo $YOU32/kernel-source-$KERNELVER.i586.rpm
fi

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• I found a great tutorial on PHP + MySQL+Flash integration
• PHP / MySQL / FLASH website integration
• Why can’t I post from docs.google.com to blogger beta?
• Study on Perspective – Set 1
• Taking a closer look
• Gall stones update – Kaiser sucks

--Colin McNamara

Why does SUSE linux make it so rough?