In the past couple years, VMware has changed from a product hidden in development and testing environments to a full fledged enterprise computing platform. It brings many benefits to the companies that implement it, however with those benefits come changes to the access layer of your data center. Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers – Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco – Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware – Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe – Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Arista Networks – Their approach to cloud networking
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Resume – Colin McNamara, CCIE #18233
• New features in VMware 3.1

--Colin McNamara

Challenges integrating VMware into Cisco networks

For those who haven’t heard yet, I passed my CCIE Lab on June 14th. Now I can officially put CCIE #18233 after my name.

Anyone who has been down the same path understands how long of a road this is.
I started down this path in the spring of 1999 as the 432nd student ever in Cisco’s Networking Academy (which I promptly dropped out of to move to the bay area). I grew my networking skills all the way to passing my CCIE Written in 2001. I attempted my Route Switch lab in 2002, however I got called away to war for a year. Being away from it all for a year really cramped my style technically. In that time many of my certs expired, and I lost much of the momentum I had built of the past half decade.

Since I got back to the real world I had focused on work to the detriment of my certifications. I really did some great things, however I really neglected my resume.
In July of 2006 I had worked 2038 hours that year (for those not mathematically inclined, that is a full work year, in 1/2 a year) . At that point I decided it was time to stop neglecting my certifications. At that point I dedicated 8 hours each saturday, along with two nights a week to studies. By December of that year I had Certified on a few of the technologies that I had tons of experience in.

I managed to get my RHCE, Cisco Storage Support and Design Specialist, update my old IP Telephony specialist certifications to CCVP, get my Netapp Certified Administrator, and pass my CCIE Written for storage networking all by the end of the year. This spring I finished my Netapp Certified Expert and scheduled my Storage lab for early summer.

Anyone who knows me well knows how closely I track my time. That time tracking extends to my training. I tracked my training (reading, lab practice, testing, etc) just like any other part of my professional life. I spent around 150 hours studying for my Design, Support specialist certs, along with reading the recommended books of the CCIE reading list, and around 300 hours preparing for my lab exam. That is 300 hours configuring every possible combination and permutation of technology that could be setup, and then refining my speed in configuring those technologies until i got to the point where speed as well as brains would be an advantage in the lab.

So now that I have my CCIE, whats next? Well, oddly enough.. I am thinking of getting my second CCIE cert. In my office I will be the Jr guy by only having a single CCIE (on of our guys has all five). I also need to take my VMware certified professional cert, and probably get my HP Master Accredited Storage Engineer. I guess I am just a glutton for punishment.

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”Similar Posts:

• Where was Colin in 2010?
• About Colin McNamara
• Updated CCIE numbers
• Why was Storage Networking my first CCIE? And What did I do to prepare?
• And it begins again – On the road to my CCIE in Storage
• Update on the Cisco / Nuova connection

--Colin McNamara

What does it take to pass the CCIE exam?