Please join me in welcoming John McCool to his new position as the leader of (in my opinion) Cisco’s most strategic business units.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

John McCool chosen as Jayshree Ullal’s replacement to lead Cisco’s Data Center Switching and Services Group (DSSG)

Please join me in thanking Jayshree for all the positive contributions she has given to Cisco and the industry, and wishing her the best in her future endeavors.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Thanks and farewell to Jayshree Ullal

Legal implications of a breach

Losing control of personal data means means more then just replacing a tape in your backup rotation. Laws vary from state to state, however generally you are required to contact the identity holders who were breached, as well as fund some sort of remediation. This has huge implications on consumer confidence, and at the end of the day stock price of your company. In some cases, such as ChoicePoint a company can be completely decimated by a breach.

Data protection regulations

There are an ever increasing number of regulations that concern the control of sensitive data. These can vary from laws focused on patient data, to financial data, to personal identification data. The most most well known laws are HIPPA, GLBA, and Sarbanes Oxley (SOX). Past that there are laws that pop up every day at the state and municipality level that further increase the requirements and expense of dealing with a breach. In short, it is becoming an expensive and in some cases criminal offense to lose control of your sensitive.

What you can do to protect your backup tapes

First things first, putting a lock on that Iron Mountain box is just not good enough. You must assume that no matter what, a determined attacker will get physical access to your tapes. So many times companies thing that just because their data format is unique or proprietary that an attacker won’t be able to access it. The cold reality is that any format can be read, and yours is not that special.

The only way to be assured that your data is safe is to encrypt it with a complex cipher. In short, you need to treat your data the same way on tape as you would if it was sitting on a public ftp site (with anonymous access enabled). Luckily Cisco has a technology that allows you to encrypt and decrypt your data coming on and off tape. This technology is storage media encryption.

Cisco Storage Media Encryption (SME)

Cisco’s Storage Media Encryption (SME) technology allows for the seamless encryption of your data flows on and off your backup tapes using AES256 standard encryption. Whether you have VSANS segregating your data, a core / edge architecture, or Virtual Tape Libraries (VTL), you can use SME to protect your data at rest, removing the possibility of an attacker getting access to your critical data.

Storage Media Encryption works by leveraging a multifunction chipset available in the 18/4 module that comes default with the 9222i and is an option for the 9500 series director class SAN switches. Chipset has a couple functions, including line rate encryption of iSCSI and FCIP data streams at gigabit speeds, as well as line rate encryption of data as it streams your tape or virtual tape library’s (VTL).

Want to learn more ?

SAN and NAS, Oreilly Press - In the classic Oreilly style by W. Curtis Preston, this book is a great starting place to understanding the fundamentals of San and Nas architectures that many people are likely to face.

Storage Media Encryption for Cisco MDS SAN Switches - http://www.cisco.com/en/US/products/ps8502/index.html . Cisco has lumped together a couple good data sheets here, though I may have to write a future article taking a deap dive on what really drives SME.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Encrypting your backup tapes with Cisco Storage Media Encryption (SME)

This switch answers a fundamental problem that has been presented by blade centers and VMware. The problem is increasing density of 10 Gig Ethernet, as well as the creation of SAN islands to provide storage access to VMware ESX clusters.  The nexus 5020 provides a solution that address both of these challenges, as well as supporting Fibre Channel Over Ethernet (FCOE) for the eventual move to a consolidated data center fabric in the years to come.

Want to learn more ?

Mastering VMware Infrastructure

Nexus 5020 Video Data Sheet

Unified Data Center Fabric whitepaper

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Nexus 5020 - Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel

This causes an interesting question to come up in many customers - Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco - Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware - Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe - Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Challenges integrating VMware into Cisco networks

IOS-XE is takes the best elements out of Internet Operating System (IOS) which has its roots in a closet at Stanford, and combines them with the most successful open source technology ever - Linux. Cisco is leveraging Linux virtualization technologies such as Kernel Based Virtual Machines to protect against operating system failures as well as to allow for In Service Software Upgrades (ISSU).

To really appreciate this, we first have to dive down into the overall architectural changes of the ASR1000. The largest change that Cisco has made was to implement separate forwarding and control planes. In the past, Cisco routers would have the processes responsible for forwarding traffic, and the processes responsible for configuring the router running on the same root operating system. The side effect of this is that if you want to upgrade the root operating system of your router, you are going to have interrupt the traffic flowing through it to do so, or have a physically separate route processor to take over while you rebooted. This is a big headache operationally, and effectively forced engineers to design in separate physical chassis to meet high uptime requirements.

What Cisco has done to address this, was to mirror changes made in their storage and carrier routing portfolios. Both of those product lines utilize the operating system to push commands into advanced processors that exist on the line cards themselves. The ASICS on the line cards are designed to work in a distributed fashion, so that production traffic never goes into up into the router processor (or sup engine). This in effect ensures that the control and forwarding planes can exist as independent elements.

If you look at the graphic above, you will notice 3 main zones. The upper zone is what we would normally describe as the control plane. This is where the higher level functions such as your routing processes, ssh daemons, snmp daemons, and shells live. In short, if you you configure or read something, you are going to do it here. The only time traffic flows through this plane is when you are doing a thing called process switching. keep in mind this is a rare occurrence and usually occurs because of an oversight in your network designs.

By separating the control and forwarding planes, this allows Cisco to basically run a management station on the router, that programs chip sets in the line cards on the fly. This in my opinion is where the true power of this architecture comes through. By separating the two functions the software engineers are free to utilize powerful open source technologies such as Kernel-based Virtual Machines, and the Linux kernel, while letting the integrated circuit engineers design blazing fast chips which allow full functionality at line rate.

What benefits should we receive from a virtualized control plane? First, in larger routing and switching chassis (including the top end of the ASR1000 line) you normally have physically redundant route processors (RP)/ supervisory engines(SUP). The operating systems on these RP’s synchronize many things, including configuration, process state, routing tables, security associations and much more. The primary reason for this, is if you have a failure in the active RP, you can failover to the standby RP without interrupting traffic flows.They also can be used to streamline the software upgrade process by only upgrading one RP at a time, and then gracefully transferring traffic to it. Once proper operation is verified, the backup RP can be brought up to the same code revision.In any production environment this is highly desirable, and helps immensely in the battle for five nines.

The ASR1000 takes the redundant RP concept seen in high end chassis, and allows you to implement redundant upgrades, as well as protection against software failure, with only one physical route processor. This is done by utilizing Linux kernel virtualization. Instead of running the control plane directly on the production hardware, a small kernel is inserted. Booting from that are two copies of IOS-XE. These run independently, and synchronize state and configurations just as if you had two physically separate route processors. What this means in operational English, is that where in the past, you would have to either have two devices, or a larger device with redundant RP’s to upgrade without disruption, you can now have that same ease of maintenance, in a much smaller (and at the end of the day, less total cost) package.

Below this is the forwarding plane.It plugs into to a high speed interconnected fabric which all line cards and RP’s are redundantly connected to. In the diagram above, this is the bottom level. Items in this plane include buffer memory, Cisco Express Forwarding (CEF) ASICS, and now the new QuantumFlow processor. This is normally where you would find your DCEF enabled line cards, fibre channel and Nexus7000 line cards, as well as the modules for the ASR1000 routers. When properly utilized, traffic should be relatively isolated to this tier, and function independently from the control plane.

The shining star of the ASR1000’s forwarding plane is a group of chips that is referred to as QuantumFlow. The QuantumFlow architecture itself merges Cisco’s strength in integrated circuit design, with its strengths in IOS software design. In the past, Cisco would design ASICS’s for specific functions, and then write commands down into them. This has worked very well, until they point that a new feature came out that couldn’t leverage the fixed configuration of an older ASIC. Your choice at that point was generally to process switch for that feature (which is slower, and honestly bad form), or upgrade your cards to the newer ASIC design. The QuantumFlow chipset approaches this problem from a new angle. The first chip in the set (Popeye) is designed to be field programmable in C, as well as no fixed internal pipelines. This combined with utilizing 40 cores running between 900 and 1200 megahertz allows the programmers to utilize parallel processing techniques to utilize an immense amount of processing power in real time.

To put things into perspective, remember when you got your first multi core laptop or desktop. You were able to say watch a DVD, as well as compile code at this same time, while continuing to have a responsive workstation. Now imagine what you could do with a 40 core processor. This is the kind of power that we are talking about. Now imagine, that not only is your workstation immensely powerful, but you could also offload common jobs such as running daily builds, or encoding videos to another machine (or in this case processor.

In the ASR1000 this processor is called Spinach (yellow are in the graphic above). And of course just like the cartoon, Popeye’s potential really comes to light when combined with Spinach. Spinach is a separate chip, that is used a a traffic manager. This chip handles queueing and quality of service, ensuring that the proper packets arrive at the proper time, as well as interconnecting with cryptographic offload engines so it can equally apply services to encrypted flows.

At the end of the day, the most important question is not how fast something is, or how cool it is. The question is what can it do for me? By leveraging this new architecture the ASR1000 is now able to do line rate inspection of traffic using Network Based Application Recognition (NBAR), Support 128,000 queues for deep quality of service, secure and encrypt data using zone based firewalls and embedded crypto engines, segregate traffic using MPLS, integrate advanced voice and video functionality, as well as providing fulling Netflow v9 support for all of the above. It provides all of these services in an always on solution utilizing Linux virtualization, as well as leveraging an flexible chip set architecture that allows for field programmable improvements in the future.

My hope is that after reading this article that you are in a better to understand how Cisco is leveraging open source technology and integrated circuit designs to improve the foundation of the internet. In upcoming articles I will be discussing design scenarios utilizing this features in this product, as well as highlighting other areas where Cisco is embracing both open source technology, as well as open architectures that can properly leverage projects such as Linux, Ntop, Wireshark and more. If this article has you interested in learning more about some of the technologies mentioned today, then I encourage you to check out some of the links below, or shoot me and email to be highlighted in a future readers questions article.

Learn more about Linux Kernel-based Virtual Machines

Learn more about Cisco’s ASR1000

Learn more about Cisco QuantumFlow

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers

Mike Writes -

“Hi Colin,
I’m an avid reader of your blog and had a question that I figured you could answer. I don’t have CCIE knowledge like I’m sure a lot of your readers do. I have worked for the same company for 6 years and during that time had been promoted into the Network Group where I was sent through class and earned my CCNA. The company I worked for decided to relocate across the country and so I have been looking for a new job. Finding a new job doesn’t seem to be that big of a deal but I noticed a lot of job descriptions are asking for BGP experience. We didn’t use BGP at my last job and I thought BGP is used primarily by ISPs for routing between Autonomous systems? If that is the case why do so many non-ISP companys list BGP experience in Networking job descriptions? What are they doing with it? Shouldn’t the ISP be doing the BGP routing for them?
Thanks!
-Mike”

Well Mike there are 3 primary reasons why a company would require (or want) BGP knowledge from its candidates.

Scenario 1. The company has an redundant Internet edge.

In this case lets call our company sample_company. Sample_company has its website hosted in a publicly facing DMZ and wants to make sure that its web servers are available in the case of an ISP failure. Normally in this case the company would request and Autonomous Systems Number (ASN) from ARIN and would get assigned a block of publicly routeable IP address’s (normally /24) that they can advertise. Sample_company would then peer with multiple ISP’s for example one connection to AT&T and the other to Sprint. Sample_company would advertise their ASN through both these ISP’s, and in the case of a failure of one of their ISP’s, the rest of the Internet would be able to calculate a path to sample_company’s web servers via the backup ISP.

Scenario 2. The company is utilizing MPLS for its WAN connectivity.

From a customer perspective MPLS is a private BGP based WAN where all edge devices connected to the MPLS provider utilize BGP to inject and learn routes. One note, some providers do support advertisement of routes via OSPF and even EIGRP now, but the most common scenario is to use BGP as your internal WAN protocol while running MPLS. One trend I am starting to notice, is that since companies are already using BGP on the MPLS WAN, they have started utilizing BGP as their primary routing protocol for their sites to avoid running multiple routing protocols and having to redistribute into BGP to cross the WAN.

Scenario 3. The company is using MPLS inside their data centers for segregation of business units.

In essence they are using the same tools and technologies that MPLS service providers are, however applying it inside of their data center and campus networks. In this case, BGP is the routing protocol necessary to carry the routes between the seperate MPLS VPN’s that are running inside the corporate data center. While this sounds pretty complicated, it actually simplifies many of the designs that you would normally implement to attain the same goals.

Learn more about BGP - Of course, there are many other reasons why you may see BGP on a job listing, but I think the previous covers the most common. If you are curious, and want to learn more about BGP I recommend buying Routing TCP/IP volume 2 by Jeff Doyle. This covers many great scenarios and configuration examples in EGP protocols. It is also written in plain English which can be a challenge with many technical books.

Learn more about MPLS in the enterprise - If you are feeling like learning about how you can implement MPLS inside of your own enterprise network then I would recommend buying Network Virtualization by Kumar Reddy and Victor Moreno. I was lucky enough to have Rick Davis translate the whole idea of utilizing MPLS in a campus environment into plain English for me a couple years back. From that point I was able to really expand my knowledge base and start asking the right questions from a firm foundational understanding of the technology. Kumar and Victors book took my understanding to the next level, showing how to incorporate many very cool features to make a MPLS network stand on its head if you want to. I can say (and actually have said to Kumar Reddy) that this book redefined my data center designs for large corporate and enterprise customers. I really recommend that you add this to your collection.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Reader question - Why are corporations looking for BGP experience?

Cisco TrustSec starts at the edge by negotiating a secure link if both hosts support it (802.1ae). This is similar to wireless encryption schemes, where a secure handshake is established and the L2 path become impervious to sniffing. This is user configurable, and to my knowledge the asics available to support line rate encryption are currently only on the Nexus 7000 blades.

The next step is to start 802.1x negotiations. For the people not familiar with 802.1x, it is a way of passing username / password information from your computer up into the network infrastructure. Once this is completed, the switch can not only utilise tools like NAC to place you into the appropriate quarantine, or access vlans, but it also know knows your identity.

Now the “network” is aware of your identity, a new level of granular security control can be deployed across your infrastructure. These security policies can map into “user x can connect to webserver y” instead of being restricted by ip and port. This allows you to utilize true roles based administration similar to what you use in your Windows and Unix file systems, but now you can do this across the network.

How is this done ? I like to think of this as a mix between dscp and mpls tags. Which in a nutshell means that when traffic enters the network it is tagged with a small amount of additional “identity: information which is retained as it traverses the network. This information can be used to augment or completely replace your current ACL based security controls in a way that enables you to more effectively comply with complex regulatory environments such as PCI, SOX, GLBA and HPPA.

Over the past few years we have learned how to leverage intelligence in the the network by utilizing tools like QOS, MPLS VPN’s, and many others. Expect to add Cisco TrustSec to your quiver of tricks to address the ever growing compliance needs faced by today’s network designers.

Learn more about Cisco TrustSec

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Identity aware networking using Cisco TrustSec

I was wrong, the Green Data Center is not about saving baby seals, it is about saving cold hard cash. Saving the world is just a nice side benefit.

That being said, saving cold hard cash is a very important discussion item in any IT Operations group as they are normally seen as a cost center. For them, a penny saved is literally a penny earned. Not only can you save money by not paying for power, but PG&E will actually has a budget to pay you NOT to use their power. Most people here this and get a puzzled look on their face. “why would the power company, who makes money on power, not want me to buy it from them?” The answer is that Californians use more power then PG&E can produce at peak times. When they have to buy it from another state it can cost them 10 times or more then they charge us. This is the reason why PG&E will pay you to use less. Each penny they give to the consumer for saving a watt, saves them 4 pennies (80% return on investment).

Great, PG&E saves money by giving it to me. How do I get this cash? Well there are a couple ways to get this.

1. Incentives for new buying new energy efficient servers
2. Rebates for moving to virtualized servers
3. Rebates and incentives for moving to thin client desktop systems
4. Audit teams for cooling and power if your Data Center is 10,000 square feet or more
5. Incentives for airflow control systems
6. Incentives for high efficiency UPS and power distribution systems
7. Technical services for cooling system evaluation (PG&E funded)

That is a pretty comprehensive list of how to get money from the power company, but you can save even more money buy not using the power in the first place. Not unsurprisingly this starts with the server.

First thing you can do, is virtualize, virtualize, and virtualize some more. For most people this means VMware. For others this may mean Xen, or Microsofts virtualization product. Whatever flavor you chose, the key message is to consolidate from many servers to few. A server sitting “idle” still pulls 50% of its max current. Now, howe many servers do you have that are just sitting there? My guess is a large amount. By virtualizing these servers, you allow them to be stacked onto high performance server that can be run at a higher utilization. This lowers the over all power utilization for your DataCenter. Another side benefit is that ever watt that you remove from a server, you get another watt removed from your cooling.

These same virtualization techniques can also be applied to your network devices, which account for 6 to 12 percent of your datacenters power draw.

Ask yourself a few questions

• ” Do I need 4 different firewall clusters?”. It is likely that these are leftovers from organic growth, and that you could consolidate them into virtual firewalls on a more efficient chassis (ASA comes to mind).
• ” Do I need to maintain physically separate infrastructure?”. There are technologies like MPLS, VFR-Lite, Virtual Switching and more that allow you to consolidate onto a shared network infrastructure, taking a service provider approach to providing transport in your network.
• ” Am I running old inefficient gear?”. Power supplies have increased in efficiency over the last few years. There may be a good return on investment for you to upgrade.
• ” Can I consolidate into larger chassis?”. Ask the question, which is more efficient - a closet full of 3560’s or a 4507? There is efficiency in scaling out.

I hope that reading this has caused you to ask some questions, and maybe look at the larger impact of your network operations on both the ecosystem and your operational expenses. With these questions in hand, you might want to talk to PG&E and your Cisco / HP parter about going “Green” in the data center.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Moving towards a Green Data Center - Truth behind the hype

For new readers to this site, please you might enjoy the following articles -

Who is Colin McNamara ?

Usability features on the Nexus 7000

Overview of NX-OS 4.0 (the operating system for the Nexus 7000)

Why did I choose to get a CCIE in storage networking?

What does it take to pass the CCIE exam?

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Featured on Network World - 20 useful sites for Cisco networking professionals

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

What do you do for fun?

What useability enhancements do you feel are the most beneficial?

1. A separate, IP enabled, Management Interface. This has been a long time coming. The out of band management interface is very similar to a Ilo card in the HP world. it is effectively a supercharged console server that happens to site on the backplane of the sup engine. I am sure whoever pushed this feature through is going to get flowers one day from a Tech who DIDN’T lock himself out because the management interface was effectively a separate system.
2. Finally, a functionally USB Interface that I can transfer IOS (well, now NX-OS) images through. Everyone has a USB key nowadays, even my Grandmother has one, it will make life so much easier when I can have a 4 gig key with me that has most IOS / NX-OS  versions and my common configs and just pop them right in.
3. The integrated Cabling system is CLEAN. I love that it forces you to reserve the appropriate space for cabling, and that there finally is the possibility to avoid the flying spaghetti train wreck we see so often in Data Centers.
4. Front to back Cooling. The cooling design is well thought out. I liked the fact that it draws from directly above the front floor and exits rear top.. This should help out in raised floor data centers that have a large temperature gradient as you move to the top of the rack. It also negates problem of having multiple 6500 chassis side to side and having warm air blowing from the exhaust of one 6500 to the intake of another 6500.
5. Fan Slots are now placed where it is IMPOSSIBLE to cover with cables. I would say 7 out of 10 times when I walk into a new customers Data Center I find that there are cables run directly over the fan tray with no slack. That is not a failure in design per say, but it could have been avoided. With the Nexus 7000 fan trays in the back the problem is solved before it is created.
6. Power supplies are in the back . FAR away from the data cabling. It never fails that 20 amp circuits get uncomfortably close to copper cabling. By moving the power supplies to the back side of the chassis, this becomes a mute point and we remove any shadow of a doubt about EM interference causing craziness in our cabling.
7. This one sounds really mundane, but a quick heads up grouping of status lights. In the past these were normally in a position where you had to squat down to see them, or they are obscured by cables. Buy putting them on the front of the cable tray assembly it ensures these will always be visible.

What can we focus on now to make it a better platform?

1. One thing that worried me a little was the placement of the compact flash cards in the supervisory module. For those how haven’t it up close look at this picture of the chassis  and look for the Grey cover midway up the sup modules in the center slots. Behind them are two flash cards, one for system partition extension, and one to dump log files into. Having these cards available are great features however I could see an operational process of security rotating out the log partitions, or more likely and engineer pulling the flash card after dumping some data for analysis to it, and then pulling the wrong card by accident. Having a simple strap (like the screw downs for power supply plugs) or something similar would go along way towards mitigating that risk.
2. Continue with the spirit of innovation that has defined Cisco over the years. Cisco has consistently came out with or acquired and integrated many great products that directly address the needs of the market place into the product line (MARS, ASA, AireSpace, TelePresence, MDS, ACE, Etc) but frankly the last GAME CHANGING product that set the industry on its heals and forced everyone to rethink how we utilize technology to accelerate business as a whole was the acquisition of Selsius and the introduction of VOIP as an enterprise class product to the world. I remember having the hair stand up on my arms from the excitement of going up against Avaya and Nortel back then and fighting that uphill battle, educating customers and peers about this “new thing called VOIP and how CallManager (now Unified Communications Manager) is your ticket towards productivity.

   When we talk about the Virtual DataCenter, I/O Virtualization (FCOE) and VFrame Automation it is not just another incremental improvement of existing technology. It is a paradigm shift, a leap ahead, a GAME CHANGER. I get the same chills that I did when VOIP was new because I know that those are technologies that will force us to rethink how we approach computing and data systems. These technologies are to the Data Center what IP telephony was to the PBX, and Cisco is the only company with technologies and engineering know how in all the verticals necessary to pull this off.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Usability features in Cisco’s Nexus 7000

As with any vendor specific event, there is a mix of for public knowledge and for private consumption content, so I generally choose not to talk or write about subjects that may have been covered in the VT unless I can find some public documentation on that subject. So don’t expect to find any juicy pre-release information or gossip here. I don’t want to have the NDA police knocking at my door, and its just not cool to let stuff slip. So, I will generally avoid the subject.

What I can tell you is this - There is a lot of buzz about the Nexus 7000. It is a rocking platform, and we spent the majority of a day going over it. I can’t share much more then I did the night before the VT just yet (will wait till I get lab access to one) but I can share this.

Yes, most people are proud of their shots with Tom Cruise, or Oprah, Richard Stephens or BSD Girl. But I can Top that.. I have a picture of me and the Nexus 7000.

The DC Channels team was nice enough to take us down to the DataCenter and Network Applications (DNA) lab. Where the Nexus 7000 has taken its new throne. They allowed to ooh and ahh and poke and prod it. Weirdly enough, the one thing that struck most was the attention to detail that went into the physical design of this chassis. It is not only good looking, but has some super usability enhancements that really impressed me.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Me and the Nexus 7000 last week at the Data Center VT

GoDaddy, one of the nations largest registrars and hosting providers is distributing bloated and possibly insecure code that will cause the average user to more then double their hosting costs. In the pages below you will learn exactly what GoDaddy is doing to your server, how their support staff will try to upsell you, and the steps you need to take to ensure proper operation of your Virtual Dedicated Server.

Background

One of my new years resolutions this year was to consolidate hosting accounts into one virtual server (hosted). I had my domains, and my old hosting with Godaddy already so it was a no brainer to try out one of GoDaddys VDS (Virtual Dedicated Servers).

I went with their 29.99 a month package, with Centos5, unlimited domains, 10Gb disk, and 256 MB of memory. This should be perfectly fine for hosting a couple MySql driven sites, and a couple gallery instances. Let me emphasise this is only handling 4 active domains, two of which only have static HTML.

Provisioning was a breeze, from order to shell account only took 4 hours. I was provided with shell access, pre-configured yum repositories, and this web control panel - simple control panel, or TurboPanel (seems to have two different names). I was able to pop into GoDaddy’s control panel interface with a direct link from their server manager console, and was setting up domains in no time. (Let me throw this caveat out though, don’t buy this product for your mom’s hosting…. the documentation is horrible, and by horrible I mean non-existant).

So I get my server all set up a couple weeks ago, transfer all my files from 2 Cups Solutions and set up my new www.colinmcnamara.com site. Things go just fine, I changed over to wordpress as a CMS and am totally thrilled. My applications and email are working perfectly. Plus, I have a shell account at GoDaddy which is a very handy thing to have as a network engineer. Things are going so well, that I give my buddy Rick a Christmas present and get ricksdavis.com and el-cinco.net for him, and host it on my GoDaddy VDS.I go ahead and purchase the domain through godaddy’s domain manager. This is obviously GoDaddy’s core competency, and goes flawlessly as usual. Next step, I go into the Turbo Panel web interface. Let me give you a little background on TurboPanel. This is the “free” equivalent “to plesk or cpanel. It is actually ok for automating your domain provisioning, though if you are a normal skill level user, I would recommend paying the extra 9.99 a month for Cpanel.

Server Error

I open up my TurboPanel interface and go to provision ricksdavis.com into the domain manager. It comes back with the least descriptive error I have ever received.

Server Error
We are sorry, the system has encountered an error while processing your request.

Home

If you continue to receive this error, please contact your system administrator.

Your URL: /domain/edit.do

Error details:

CommandFailedException: Unable to get min/max uids
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUids:825
at c.g.t.f.systems.user.LinuxUserSubsystem.getMinUid:780
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUserInfo:670
at c.g.t.f.systems.user.LinuxUserSubsystem.getUserInfo:646
at c.g.t.w.actions.domain.ActionDomainEdit.process:84
at c.g.t.w.actions.AbstractSpringAction.execute:118
…
at c.g.t.w.filters.AuthorizedResourceFilter.doFilter:38
…
at c.g.t.w.filters.RequestPopulationFilter.doFilter:117
…

This is the most descriptive error ever right? it tells you what is wrong, has a link to the support system, and gives you actionable information…. I would say a resounding NO. This is a classic example of why friends don’t let friends program in Java. When I got this error last night, I was scratching my head. As an engineer, the first thing I will look at is the last change to the system. Coincidentally I had installed awstats two nights before, and looking at my change logs, I saw that I had upgraded my perl version. So, with no fast response to the support email from GoDaddy I chose to put in a server re provision request (fully automated) and restore from my backups. That process took about an hour, but afterwards I was back online with no errors. Eureka! I found it (I thought). I provisioned Rick’s domain, wordpress, gallery2 etc and then went to bed.

GoDaddy Support Response

Fast forward to this morning, and I finally recieve an email reply from godaddy support. The email is pasted below-

(I have replaced the agents name with John Doe. Tech support is a hard job and I see no reason to highlight him specifically)

Before re-provisioning the server, I went ahead and tried the old three finger salute (reboot) the error still existed. So even if this email would have came to me on time, it would not have helped.

But that is besides the point. lets dig into the solution

The agent suggested the following fix -

Restart tomcat - which I was NOT using for any of my web applications (not an ejb guy)

/etc/init.d/tomcat55 restart

Restart TurboPanel (or simple control panel now - they haven’t updated their init scripts)

/etc/init.d/turbopanel restart

GoDaddy tried to upsell me, instead of fixing their code

He gives the standard, run less stuff on your server speech (remember, I only have 4 domains on this server)

Now here is the kicker - To remedy this issue long-term, you could either setup a server with 512mb RAM, or upgrade to a Dedicated server.

What the heck is with that? I should not need an upgrade with only 4 domains on a server. Especially when in the setup the default Cpanel implies support of 30 domains. Is this Tech Support or a Sales Call?

But, since my server was working fine I don’t pay much attention to the email and move on with my life.

Fast forward an hour, and I am show Rick how to access all the features of his new site, and I figure that I need to change an email account on his domain to forward to his old account. Fine, this should take two minutes. I log onto the TurboPanel interface to put the email forward in.. and there it, a big useless error screen. What the heck is with that?

So this time I actually read the email, and try the fix. Results = nothing. GoDaddy offers shell access so I log in, run top and filter for %memory used. Low and behold, there is only 8142 bytes of memory free, and a Java process owned by root is using 300Mb of virtual memory, and 132Mb of real memory, and Tomcat is using 115Mb of virtual memory and 86Mb of real memory.

Let me translate this into English - Godaddy’s control panel application was using 218 Megabytes of the 256 Megabytes of memory I had purchased. That left me with 34 Megabytes of memory . Let me clarify this, I had paid for a virtual server with 256Mb of ram, up to 1000Mb bursted (which I think is their code for swap). I am only running 4 domains on this server, and two pop3 email servers. This should not be a problem.So what is the cause of the problem? I can sum it up, crappy Java programming. Someone decided to write this program in Java (probably easier to outsource) instead of optimising it to run on lean systems. Their code effectively takes up all the available memory. And on top of that, they are are running a webserver process as root… yes as root. It is like asking for your server to get hacked.

Now, that I am done ranting, let me highlight how to fix this problem.

If you are on Windows use the following procedure to get shell access to your GoDaddy VDS -

• you will want to download a ssh client called putty - Download Here
• Copy this file to your desktop, double click putty.exe , and you should see something like this -

• In the host name field I have www.yourdomain.com . replace yourdomain with your domain name.
• Click on the open button on the bottom right, and a shell should pop up, along with a warning that looks like this

• Now skip past the linux section

If you are running Linux or Unix start here -

• Open up a command line terminal
• ssh using your godaddy simple control panel username example -

ssh your-godaddy-control-panel-username@www.yourdomain.com

• If this is your first time connecting to this server, you will be prompted to accept an unkown ssh key into known_hosts, choose yes to accept

Both Windows, Linux and Unix Continue Here -

• Enter your the password you use to access your GoDaddy Simple Control Panel Interface
• You will be presented with what looks like a DOS window, this is called a secure shell terminal. Type in the username you use to access your Godaddy Simple Control Panel and then hit enter

• Now type in the password that you use to access your Simple Control Panel Interface and hit enter

• Congratulations, if you see the window below you are now shelled into your virtual dedicated server.

• Now that you are shelled into a Linux device you need to escalate your privileges to get administrator level access. In the Unix world this user is known as Root. You can change to this user, and get full system privileges by using the following command.

su - root

• You will be presented with a password prompt, enter in the same password that you have used to log into your Simple Control Panel Interface
• You are now root, be careful with what command you enter under this user, as you can do some damage if you are careless

Clean out your servers memory

• Most recent Redhat direvatives (including Centos) utilize a tool called yum to add and remove packages. This is also true with your linux servers at GoDaddy.
• you need to install a tool called memhog, it is part of a package called numactl that is normally used to assign specific process’s to specific cpu’s in a multi-core system. We will be using it today to fix GoDaddy’s memory hogging application

yum install numactl

• choose yes to all the prompts, and numactl will automatically be downloaded and installed on your server

Stop GoDaddy Simple Control Panel, and Tomcat, and tell them not to start automatically when your server restarts.

• Tomcat is a special type of webserver for Java based applications. Godaddy uses it to run their control panel interface. 99.99999 percent of users will not need to use Tomcat. If you do need to use Tomcat then you are a technical user and will know what to do.
• In Linux, server applications are called daemons. The are executed by init scripts. We will use these scripts to turn off these server applications
• Turn off the TurboPanel daemon ( this is the process that runs your simple control panel web interface

/etc/init.d/turbopanel stop

• Next we need to turn off Tomcat

/etc/init.d/tomcat55 stop

• Now that we have these services turned off, we need to make sure that they don’t come back when we reboot the server. We can do this by using the chkconfig command.
• Stop the Simple Control Panel Interface from starting automatically by executing the following command

chkconfig turbopanel off

• Stop the Tomcat server from starting automatically by executing the following command

chkconfig tomcat55 off

Clean the mess GoDaddy made of your servers memory

• A couple steps back we installed numactl. The executable we wanted out of this package is memhog. Issue the following command to take your memory back. This command will overwrite 200 megabytes of your memory, allowing the rest of your applications to get access to that memory.

memhog 200m

• This will Clear out the memory that GoDaddy’s application took over, and allow the rest of your daemons to run fine.

Great, my server is running better now. But I want to use my Simple Control Panel Interface. How do I do that?

• Easy, all you need to do is temporarily start the turbopanel daemon. When you are done making changes, you can turn it off again

/etc/init.d/tomcat55 start

/etc/init.d/turbopanel start

• When you are done, don’t forget to turn it off

/etc/init.d/turbopanel stop

/etc/init.d/tomcat55 stop

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Why GoDaddy Linux Virtual Dedicated Hosting Sucks & How to Fix It

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

DataCenter SVP Jayshree Ullal interviewed about the Nexus 7000

One feature that is new, and frankly extremely exciting is Virtual Device Contexts. Each virtual device runs with its own process, vs the use of tagged differentiators in technologies such as VRF-Lite. This provides for paravirtualized management instances, and clear lines of delineation for both software and hardware for a resource that can be shared between different groups within an enterprise.

Chassis that run NX-OS will support In Service Software Upgrades (NSSU) to allow operations groups to upgrade operating systems with zero downtime. This is accomplished through a combination of modular software architecture, and the decoupling for the control and forwarding planes.

One of my favorite features in SAN-OS is the embedded is fabric analyser. This is a tool that can sniff management traffic without having to plug in a sniffer, or provision a span port. You can dump in real time to a tcpdump like interface in the command line, output to a local file, or map to the ip of a wireshark instance that layer 3 access to the management port. Cisco again has taken the best of SAN-OS and bundled it with NX-OS. You will be able to remotely span management traffic without having to set up rspan, or trudge down to the datacenter to set up a sniffer.

Now, your router can call home right now so that is not a totally new feature. Smart Call Home was released recently into IOS. But that still doesn’t stop it from being a great feature. This allows you to configure NX-OS powered devices to mail an xml formatted troubleshooting email to TAC, and / or your support staff. This has been proven to drop the average time to resolution from 16-30 hours to 6 hours.

Now the drum roll…… All IP routing features are VRF aware. This has been a point of contention with me for a while. As Cisco and the market in general has embraced virtualization as an answer to pressing business concerns of leveraging shared infrastructure, while retaining security controls segregating disparate environments technologies such as MPLS and VRF within the datacenter have become more and more prevalent. That is great, however it never fails that the feature you need at that moment always seems to be coming out in the NEXT IOS release. With Cisco NX-OS 4.0 this is no longer a question.

Now, if I was a CIO and I was reading about all these new technologies that Cisco was pushing with NX-OS, I would frankly be cautious, and rightfully so. The thing is, most of these features are not new, they have been in use, and in production under the most stringent uptime conditions in the world - storage networking. They have been tried and tested on Cisco’s MDS line of storage networking switches. So get comfortable, get educated, but most importantly get on board for DataCenter 3.0.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Cisco NX-OS 4.0 | Next Generation Internet Operating System

Highlights of the the Nexus 7000’s features are -

• 15 Terrabit per second backplane
• Support for 40 and 100 gig ports in the future
• Seperate control and data planes
• link layer encryption
• front to back airflow (FINALLY available in a non NEBS chassis)
• Lossless non blocking fabric (VOQ enabled)
• Fibre Channel, Infinaband, and Ethernet blades in one unified platform
• Cisco Data Center Network Manager (MDS Fabric Manager on steroids)
• Virtual Device Contexts (Network Systems virtualization, the next level past VRF route tags)

You can learn more about this switch in upcoming articles, and at http://www.cisco.com/en/US/products/ps9402/index.html

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Cisco Nexus 7000 DataCenter switch released - Welcome to DataCenter 3.0

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Russ White - CCDE interview - Part 2

you suck at photoshop 1

you suck at photoshop 2

you suck at photoshop 3

you suck at photoshop 4

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Quite possibly the funniest and most educational PhotoShop tutorials ever.

For the engineering perspective on this, the CCDE is equivalent  to the CCIE. However, the CCDE is focused on design and architecture rather then implementation. Where the CCIE (R&S, Voice, Security, Service Provider, Storage) is focused on implementation, the CCDE is focused more on the pre-sales design and architecture efforts. I am personally looking forward to the lab being released, as it provides a certification to validate the skill set needed to be an sales engineer on Enterprise accounts, or to be a network architect at an Enterprise corporation.

It is funny how small a world it is. Eplus (the company I work for) CEO - Phil Norton was quoted on Cisco’s press release -

“Certifications provide a stamp of approval that validates the quality of our organization’s employees,” said Phil Norton, chairman, CEO and president of ePlus. “The CCDE isn’t about operations; it’s about recognizing the value of network designers and honoring their core skills that provide a real value to our business and our customers.”

My gut feel when I first got invited to the CCDE beta program was that this will become a requirement for the Channel. I think Phil’s statement cements that gut feel into a reality. Obtaining a CCDE will become similar to the CCIE - a check box that you must attain to work with the top VAR’s out there. This makes me extremely grateful that I was lucky enough to be invited into the beta group to be allowed first crack at this gem of a certification.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Cisco Certified Design Expert - CCDE - officially released by Cisco

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Russ White - CCDE interview

According to surveys, 57 percent of Americans shower every day.

Hugh Hefner has a species of rabbit, the Sylvilagus palustris hefneri, named after him.

The bull’s-eye on a dartboard should be exactly 5 feet, 8 inches off the ground.

Teenagers are 50 percent more susceptible to colds than people over 50.

Sir Isaac Newton died a virgin.

Studies have shown that the larger a man’s testicles, the more likely it is he’ll cheat on his mate.

The consumption of beer in a lavatory is forbidden in Manitoba, Canada.

Elevators and escalators kill about 30 people, and injure about 17,000, each year in the United States.

Mother Teresa’s real name was Agnes Gonxha Bojaxhiu.

In ancient Rome it was considered a sin to eat the flesh of a woodpecker.

Typically, sex between snakes lasts between six and 12 hours.

“Symphorophilia” is sexual arousal derived from arranging a disaster, crash or explosion.

The average American worker uses 11 Post-It notes a day.

There are five cities named Las Vegas in the United States. There are two in Puerto Rico.

The oldest existing governing body is in Althing, Iceland. It was established in 930 A.D.

The band Creedence Clearwater Revival was originally named The Blue Velvets.

Men are twice as likely to contract leprosy as women.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

A little trivia to start the week

More making of…, originally uploaded by Ashley McNamara Photography.

I got bought a 15mm fisheye lense, and of course, Ashley can put it to much better use then me. This photo is of her friend Ingrid getting henna art done on her belly.

Post from: Colin McNamara - CCIE 18233 , RHCE, CCVP, GIAC-GCIH, GEEK

Ashley stole my new Lense…

Kylie Launching in Yosemite while sledding, originally uploaded by Colin_McNamara.

This is the best vacation ever, 4 days in Yosemite over Christmas. This may turn into a yearly thing.

Post from: Colin McNamara - CCI