Colin McNamara - CCIE 18233 , RHCE, GCIH, CCVP, GEEK

John McCool was chosen to succeed Jayshree Ullal as the leader of the DSSG yesterday. John comes with a rich development background on both the 4500 and 6500 series platforms, as well participating in internet standards bodies.

Please join me in welcoming John McCool to his new position as the leader of (in my opinion) Cisco’s most strategic business units.

Jayshree Ullal anounced today that she will be leaving her post as Senior Vice President in charge of Data Center, Switching, and Security groups. Jayshree has earned a reputation inside and outside of Cisco as a person who could take charge and get things done. First coming to Cisco as an engineer with the crescendo acquisition, she has directed some of Cisco’s most successful units culminating with the realization of the DC 3.0 vision.

Please join me in thanking Jayshree for all the positive contributions she has given to Cisco and the industry, and wishing her the best in her future endeavors.

IT staff at the University of Miami are having a very bad week. They are having to deal with the fact that two million private health records were stolen from from them. While it wasn’t directly their fault that their backup tapes were stolen from a off site storage providers transport van. The responsibility does fall on their shoulders to protect sensitive data no matter who has access to the physical media.

Legal implications of a breach

Losing control of personal data means means more then just replacing a tape in your backup rotation. Laws vary from state to state, however generally you are required to contact the identity holders who were breached, as well as fund some sort of remediation. This has huge implications on consumer confidence, and at the end of the day stock price of your company. In some cases, such as ChoicePoint a company can be completely decimated by a breach.

Data protection regulations

There are an ever increasing number of regulations that concern the control of sensitive data. These can vary from laws focused on patient data, to financial data, to personal identification data. The most most well known laws are HIPPA, GLBA, and Sarbanes Oxley (SOX). Past that there are laws that pop up every day at the state and municipality level that further increase the requirements and expense of dealing with a breach. In short, it is becoming an expensive and in some cases criminal offense to lose control of your sensitive.

What you can do to protect your backup tapes

First things first, putting a lock on that Iron Mountain box is just not good enough. You must assume that no matter what, a determined attacker will get physical access to your tapes. So many times companies thing that just because their data format is unique or proprietary that an attacker won’t be able to access it. The cold reality is that any format can be read, and yours is not that special.

The only way to be assured that your data is safe is to encrypt it with a complex cipher. In short, you need to treat your data the same way on tape as you would if it was sitting on a public ftp site (with anonymous access enabled). Luckily Cisco has a technology that allows you to encrypt and decrypt your data coming on and off tape. This technology is storage media encryption.

Cisco Storage Media Encryption (SME)

Cisco’s Storage Media Encryption (SME) technology allows for the seamless encryption of your data flows on and off your backup tapes using AES256 standard encryption. Whether you have VSANS segregating your data, a core / edge architecture, or Virtual Tape Libraries (VTL), you can use SME to protect your data at rest, removing the possibility of an attacker getting access to your critical data.

Storage Media Encryption works by leveraging a multifunction chipset available in the 18/4 module that comes default with the 9222i and is an option for the 9500 series director class SAN switches. Chipset has a couple functions, including line rate encryption of iSCSI and FCIP data streams at gigabit speeds, as well as line rate encryption of data as it streams your tape or virtual tape library’s (VTL).

Want to learn more ?

SAN and NAS, Oreilly Press - In the classic Oreilly style by W. Curtis Preston, this book is a great starting place to understanding the fundamentals of San and Nas architectures that many people are likely to face.

Storage Media Encryption for Cisco MDS SAN Switches - http://www.cisco.com/en/US/products/ps8502/index.html . Cisco has lumped together a couple good data sheets here, though I may have to write a future article taking a deap dive on what really drives SME.

Cisco released an addition to their Nexus series data center switching line, the Nexus 5020. The Nexus 5020 packs 1.04 terrabit of switching capacity into a 2 ru top of rack chassis. Inside this chassis you have 40 10 Gig Ethernet ports, as well as modular slots that can accept 12 extra 10 Gig ports, or 8 port Fibre Channel cards for a total of 56 available ports

This switch answers a fundamental problem that has been presented by blade centers and VMware. The problem is increasing density of 10 Gig Ethernet, as well as the creation of SAN islands to provide storage access to VMware ESX clusters.  The nexus 5020 provides a solution that address both of these challenges, as well as supporting Fibre Channel Over Ethernet (FCOE) for the eventual move to a consolidated data center fabric in the years to come.

Want to learn more ?

Mastering VMware Infrastructure

Nexus 5020 Video Data Sheet

Unified Data Center Fabric whitepaper

In the past couple years, VMware has changed from a product hidden in development and testing environments to a full fledged enterprise computing platform. It brings many benefits to the companies that implement it, however with those benefits come changes to the access layer of your data center. Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers - Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco - Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware - Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe - Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.