So many people linked, tweeted and posted on Facebook about my VMworld presentation submition #3221 Built to Fail. I just wanted to say thanks to everyone for spreading the word. I should find out on June 1st whether I am presenting.

Again, thanks to everyone for all of your support.Similar Posts:

• How to succede in 2007 – By Tim O’Reilly
• VMworld 2009 Schedule
• Witch fire in San Diego – Notes from an evacuee
• New Buzzword Enterprise Service Bus
• New features in VMware 3.1
• What does it take to pass the CCIE exam?

--Colin McNamara

Thank you for voting (VMworld Presentation)

The Problem

Apparently when the UCS pod’s firmware was updated to 1.3(1c) an interesting bug was encountered. What happens is when you associate your service profile to a server which has had another service profile associated with it previously you a config failure error stating that the local disk config you are applying is destructive to the physical server and violates a requirement.

I headed over to the local disk policies section of UCSM to see what was going on. Once I got there I noticed a new check box labeled “Protect Configuration”.

It’s a lab, which means that change controls should be followed but rarely are. Following that mantra and went ahead and unchecked the “Protect Configuration” box on one of the commonly used local disk policies and saved my changed. A little background on HOW I am shelled into this lab. I am shelled in via a VMware View server that has component installed on blades 1 and 2 of this chassis. And, yes the local disk configuration that I made a change to was used by these profiles.

A little while later, after all the servers went through a reboot and settings application cycle and a coworker kicked a View server in the butt I was back in. The first thing I did was try to apply my service profile. I got the same error message.

I went back to the local disk config profile section to see if maybe the change hadn’t applied. But, when I edited the profile, it showed that the local disk config should not be in a protected configuration.

However when I went into a service profile to change the same local disk policy, you will notice that this same local disk config was still showing up as a protected configuration.

The Solution

When you encounter a bug, the first thing you do if the obvious isn’t working is check the release notes. This bug did show up int he 1.3 release notes, with a tag of “no workaround”. I don’t believe in not having workarounds so proceeded to bang my head up against a wall trying time consuming things like rediscovering servers, recovering bios’s, creating custom firmware packs and more.

After all that failed, I tried the simple things (you like the reverse order?). My excuse was that I assumed that TAC who had created the bug had already tried this. I guess I should assume less often…

The simple thing, that I should have done from the start was create a brand new local disk profile. I named this profile FIX-PERSIST-ANY, choosing to use any local disk config and expressly making it non-persistent. I then applied the same local disk policy that was on previously to my service profile, and then once applied changed it to FIX-PERSIST-ANY.

At that point I whatever flag that had gotten stuck in the UCSM database got unstuck and I was able to get my service profile applied.

Moral of the story

Just because a bug shows no workaround, it doesn’t mean you can’t fix it. It just means that the engineer who submitted the bug wasn’t able to.

Now I can finally get around to what I was originally trying to do today, getting scripted PXE installs of ESXi 4.1 working.Similar Posts:

• Cisco Nexus 4000 Blade Switch
• Cisco introduces the C-Series Rack Servers
• Cisco Unified Computing System Quoting and Configuration with Netformx
• Where was Colin in 2010?
• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem

--Colin McNamara

Fixing UCS Config Failures due to local disk config requirements

Intellectual capital driving the cloud

It is wise to follow the movements of thought leaders in Silicon Valley. Why is that? Because when enough smart people land at the same company, it is only a matter of time something great happens. This “human network” of intellectual capital has been the seed of many successful tech companies, and will continue to be true in the future.

One of these tech companies with a wealth of intellectual capital is Arista Networks. There are A LOT of ex Cisco folks walking the halls of Arista. Many of them come from the Granite Systems acquisition (Cisco’s 4500 platform). This platform, while designed with line card oversubscription to keep it between the 3560 and 6500 platforms in price and performance has an extremely elegant internal architecture. Case in point, the 4500 platform has had in service software upgrade (ISSU) for over two years, something that the 6500 still struggles with.

Now that this team, and key leaders from Cisco and other tech companies are putting together a network platform, what can they do? And more importantly, what will they do?

Before I dive into that answer, I think it is important to take a quick overview of the two major camps of network platform development, and what the advantages and drawbacks of each method is.

Creating your own ASICS in house

The first way is to create your own ASICS that handle switching and security functions. In this case, you are effectively a chipset manufacturer, who then bundles your own chipsets into routing, switching and security platforms. On one hand, developing your own ASICS can give you a competitive advantage by rolling in features that are not available to your competitors.

On the downside however because of the high cost of developing these chipsets you are forced to design for a very long lifecycle (7+ years). Another downside is that if you have any problems with manufacturing, you cannot just call up another supplier and change your sourcing strategy because you are that supplier. In the case of any Fab issues you are forced to slip your product delivery dates.

Utilizing market silicon

The second way is to utilize routing, switching, and security ASICS that are commercially available through many manufacturers and wrap your own software and chassis integration around them. This is commonly referred to as “market silicon”. In this case, your focus is end to end integration of commodity ASICS and most importanly creating  software differentiation to add value to your product.

The positives aspects of this model is that you are not locked into your own chipset design time lines. If your primary chipset supplier has a Fab issue, then you can easily change your supplier and hit your deployment time lines.

The downsides of this model is that every single networking manufacture in the world has access to the same chipsets. This forces a vendor to differentiate through better software, support, and integration of these “Market Silicon” ASICS into a superior platform.

Who uses what?

With all the talk of Market Silicon being evil, the reality is that the major networking manufacturers use a mix of home grown ASICS and market silicon to drive their products. I can’t say who uses what, but feel free to crack open your switch and take a look at the chipsets on the line cards. Don’t be surprised if you can find some market silicon sprinkled here and there. Now that doesn’t mean that these platforms are bad, it just means that for certain functions it is cheaper to source ASICS externally then to create them in house.

How does Arista approach this problem?

Aristas focus is to create an extensible network operating system that can manage and enable multiple switching ASICS and switching platforms (VMware Virtual Network Distributed Switch – vNDS).

Extensible Operating System (EOS/vEOS)

Arista created a new operating plaform, based on Linux that manages both the physical and virtual implementations of switching devices (ASIC and Virtual Switches). It is called the Extensible Operating System. This operating system has hooks into all the ASICS and vSwitches that it supports. Most importantly it provides one single operating system for all supported platforms both physical and virtual.

sysDB

Core to the functionality of EOS is the sysDB. What is the sysDB? It is a custom real time database written specifically for the interaction of individual system processes. These include routing, switching, security, management processes. By centralizing all of this information in a central location the time to react to events is minimized . This is especially true when compared to classic networking implementations where independent processes keep independent state.

vEOS

Virtual Extensible Operating system is just that – A virtualized instance of the items mentioned above. This can be run inside a vmware virtual machine. It is the same operating system, database, and daemons that run on Arista’s physical hardware. The only difference that it happens to run inside of your virtual infrastructure.

You may ask the question, why would you want to take a network operating system / hardware combination and split it apart?

vEOS and VMware Virtual Distributed Network Switch

EOS and vEOS have implemented a hook into VMware’s vNetwork Distributed Switch (vNDS) API. In effect, you can think of the vNDS as just another ASIC to the operating system. Instead of connected through a device driver, EOS and vEOS connect in through an XML API. This accomplished the function of both retrieving status and performance information that the vNDS provides, and creating policies inside EOS and publishing them into your VMware switching infrastructure.

If you have an Arista switch directly northbound of your ESX servers, you get this monitoring and configuration feature for free. If you don’t have Arista switches, (say you have Cisco, HP, Juniper or Foundary) you can use vEOS (the virtual instance) and pay a fee to get a cli interface into the VDS.

vEOS vs Nexus 1000V

This is a likely to be a highly contested item, complete with competing bumper stickers. In my opinion it isn’t that big of a deal. The reason being is that the 1000v and Arista’s vEOS implementation are completely different. Cisco’s 1000V is a dedicated piece of code running on your ESX servers that handles switching differently then VMware’s vNDS. Arista’s implementation of EOS and vEOS is more of a management interface to VMwares vNDS. vEOS does not replace the switch inside VMware, it configures and monitors it through the vNetwork API.

When comparing the two products head to head, the discussion is really a VMware vNDS vs Nexus 1000v discussion. If you have already decided to move to the 1000V because of the feature differential between the native vNDS then nothing really changes.

This doesn’t mean that vEOS does not add value. In smaller environments where the 1000V is not an option, or in an intercloud situation where state needs to be passed between disparate network instances vEOS’s vNDS implementation can be very valuable. If the vNDS features are all you need, but you would prefer a CLI for your VMware switching and cannot justify the expense for the 1000V licenses, then Arista might be right for you.

Want to learn more?

Arista Networks – Extensible Operating System

Andy Bechtolsheim‘s opinion on Market ASICs

VMware Virtual Network Distributed Switch

Cisco Systems – Nexus 1000V

Similar Posts:

• Jayshree Ullal takes the helm of Arista Networks
• Cisco Nexus 4000 Blade Switch
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Cisco’s Cloud Computing Offering
• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers

--Colin McNamara

Arista Networks – Their approach to cloud networking

If you happen to be in the same class, or see me passing in the hall, feel free to pull me aside and say hi.

Tuesday
10:00 AM-11:00 AM	EA3605 Room 302	Virtualizing Tier 1 Applications: The Value of the vSphere Internal Cloud as a Better Platform for Apps
11:30 AM-12:30 PM	VM4800 Room 110	The “Next Generation Data Center” for Telecommunication Companies
1:00 PM-2:00 PM	SS5240 Room 134	Engineering Developments Enabling the Virtual Datacenter – VMware, Cisco and EMC
2:00 PM-3:30 PM	EA3234 Room 104	Virtualizing SQL Server in a VMware vSphere environment
4:00 PM-5:00 PM	VM2472 Room 303	Introduction to VMware vCenter Chargeback
6:00 PM-7:00 PM	EA1820 Room 310	Virtualizing Critical Healthcare Applications

Wednesday
10:00 AM-11:00 AM	TA3286 Room 132	Applications in the Cloud: Getting off the ground
11:30 AM-12:30 PM	EA2583 Room 110	HPC/Grid Computing and Virtualization
1:00 PM-2:00 PM	TA1962 Room 121	How and Why we Upgraded Herning Kommune’s Production Environment to vSphere 4.0 at GA
2:30 PM-3:30 PM	TA4100 Room 303	Internal Clouds: Customer perspective and implementations
4:00 PM-5:00 PM	VM3881 Room 309	Business Objects SAP Virtual Infrastructure Lab Manager Deployment and An Overview of the Best Practices and Process of Migrating Between Network Ranges

Thursday
9:30 AM-11:30 AM	LAB11 Nob Hill A	VMware vCenter Chargeback
11:30 AM-12:30 PM	TA4881 Room 132	Designing Dynamic Data Centers with NetApp and VMware
12:30 PM-2:30 PM	LAB09 Salon 4	VMware vCenter AppSpeed
2:30 PM-3:30 PM	EA3241 Room 301	Beyond Infrastructure as a Service: Developer and Runtime Services with VMware and our Partners
4:00 PM-5:00 PM	EA3481 Room 301	Virtualization of Analytic Databases

Similar Posts:

• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• Cisco’s Cloud Computing Offering
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Interesting TechWise TV episode on virtualization
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)

--Colin McNamara

VMworld 2009 Schedule

Why do I say that? It is simple, every server that moves from a corporate data center into a cloud provider is a switchport and fibre channel port (and now server) that is not purchased from Cisco. More so, each system that is moved into the cloud hurts secondary sales of security and content switching products.

The promise of enterprise cloud computing

The ability to dynamically scale enterprise compute workloads while only running a “right sized” private infrastructure is top of every CIO’s mind. This is the promise of cloud computing in the enterprise space. However, right now most cloud offerings are too new, and lack the critical integrations with VMware or XenSource (the two most common enterprise virtualization platforms) to make a serious dent in Cisco’s revenue stream. But fast forward 12 to 16 months and the kinks will be worked out. Projects that would previously have required new capital infrastructure will be restructured to use cloud providers as an operational expense. This will present a real threat to Cisco’s revenue moving forward.

John Chambers and his team of technologist are not new to this game, this is not the first threat to Cisco’s sales model. And I am sure that it won’t be the last. So if I was in their shoes, what would I do? (and more specifically, what do I think they are doing)

Create a compute platform that can power the cloud at a much lower cost that my competitors

Cisco publicly announced their computing offering, the Unified Computing System in March of this year. The promise of the UCS is to minimize power, cooling, capital costs and management overhead of data center compute. Looking at this new product line from an enterprise sales perspective it makes sence. For Cisco to continue with their growth plans they had to choose to enter the Compute or Storage markets, with the compute (server) market being the logical step.

While the Unified Computing System is well placed as an enterprise computing platform, I think there is a larger goal in mind. The large goal is to make a platform that can be shared by Cisco’s largest enterprise clients in their emerging private clouds, as well as by Cisco itself for it’s own cloud offering. By producing their own servers, with technology that Cisco alone has access too (memory expansion / hypervisor bypass) Cisco sets themselves up to have both lower hardware costs in their own cloud, as well as lower operational costs (power/cooling). This will provide Cisco with higher margin at the same price point as their competitors.

Distribute application aware network devices at customer locations

Cisco already has a significant edge over any competitive cloud offering. A vast majority of enterprise customers already run Cisco routers, switches and firewalls. If Cisco decided to say, port the TCP optimization code from their WAN acceleration platform into IOS, and configure it to work with their own cloud offerings this would give them an immediate leg up on the competition. Combine this with the existing WAAS auto discovery and Cisco could conceivably automatically integrate a cloud based caching offering with a customer’s onsite devices.

Create an application centric cloud security model that can be integrated with virtualization platforms

Last year Cisco announced a new approach to security called Cisco TrustSec. This technology includes a change from layer 4 based acl’s to an application focused role based implementation. This is applicable in the cloud environment because it provides a standard integration for controlling the access to and mobility of applications as they travel between public and private clouds.

An interesting side bar, is the fact that when integrating public and private clouds, there will always be applications that you want to keep on your internal cloud. The easiest way to do this is to put some sort of meta information on the virtual server containing a flag that this server should only run on the private cloud. With VMware there are fields that are used for DRS that can house just such data. I would not be surprised that with all the work that Cisco and VMware have been doing together if this was not implemented with vSphere (Virtual Infrastructure 4).

Learn as an organization how to profit from a SaaS model

I think this last piece of the puzzle has been overlooked by many people. Cisco already has in house experience dealing with a massive Software as a Service (SaaS) offering – Cisco WebEx. In acquiring WebEx Cisco also acquired the talent and technology behind the worlds largest collaboration platform. Cisco should be able to take the lessons learned from running and improving this platform, and apply them to their upcoming cloud offering.

Summary

Cisco has to go to market with a Cloud offering to maintain long term viability as a company. When they do they will have the benefit of lower cost of building and operating the grids that their cloud offering will run on. They will be able to leverage millions of Cisco network devices in their current install base as well as provide application centric security integrated with these same devices. And most importantly they will be able to use the lessons learned from running WebEx to ensure flawless delivery of an upcoming cloud computing offering.Similar Posts:

• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• VMworld 2009 Schedule
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Cisco releases Nexus 1000V virtual switch for VMware
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

--Colin McNamara

Cisco’s Cloud Computing Offering

Because VMware is central to Data Center architectures, I decided to do a deep dive over the past week. This is not my first foray into virtualization, I have used vmware workstation since 1999 and got my first exposure to Storage Networking in 2004 with ESX. Read an article from 2005 about me dealing with network issues in ESX. However VMware has has added so many interesting and relevant features that I found it was good to do a ground up review on. I have to say, that review was helpful. There are many features that as a network designer that should have been in the forefront of my mind, that I hadn’t grasped the full potential of.

As always, after a deep dive into game changing technology I am filled with more questions then answers. But the answers I do have are enough to try my luck at an exam. With an afternoon free, I decided to try my luck at the VMware Certified Professional Exam (VCP-310). Apparently my deep dive worked, because I walked away with a new cert for the binder

Now with a better understanding of the value adds that VMware Infrastructure 3 (Vi3) has in the Data Center, I need to deep dive again on VMware Infrastructure 4 (Vi4) and figure out how Virtual Data Center OS (VDC-OS) and VMware based application virtualization tie into the classic methods of application virtualization (load balancing and content switching) as well as lay down some common network architectures utilizing the Nexus and converged data center fabrics for the move towards cloud computing in the enterprise. I’ll keep you posted on the results of these deep dives, I am sure it will be interesting.

Needless to say, it is a fun time to be a Data Center geek…Similar Posts:

• Interesting TechWise TV episode on virtualization
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• I’ll be at Cisco Live 2008 (networkers) in Orlando all week
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• VMworld 2009 Schedule

--Colin McNamara

Where is Colin ? Passing the VCP exam (VMware Certified Professional)

• None Found

--Colin McNamara

Quoted on ZDnet – Shameless self promition

Altor networks goal is to provide a single pane view of communications within your ESX clusters, as well as ease access control list creation and deployment. With this single pane virtualization customers should be able to decrease the time needed resolve availability and security issues, allowing virtual enviornments to continue to scale.

This is a sign that we can look forward to many other software vendors adding Nexus 1000v  support to their existing product lines. I wonder who is next ? NetQOS maybe ….

Want to learn more ?

Altor NetworksSimilar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Interesting TechWise TV episode on virtualization
• Arista Networks – Their approach to cloud networking
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)

--Colin McNamara

Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware

The boundary between server team and network team responsibilities has become “fuzzy”

Cisco address’s this issue by putting a switch that can be managed via the same methods common to other network devices inside the ESX cluster. This switch runs the same code that has become standard on Cisco’s Nexus series of Data Center switches – NX-OS.

Prior to adoption of virtualization, when there was a connectivity problem with a host it was quite common for the network team to verify functionality down to the switch port. The server team would do the same. This allowed for each team to focus on areas that met their core competancy. Once we moved from a real switch port, to a dumb bridge inside ESX, lots of finger pointing resulted.

Now, with a Nexus 1000V sitting virtually inside the ESX clusters, the boundary between network and systems teams has been re-estabilished. Now when there is a problem with a host inside an ESX cluster, the network team can use the same day to day troubleshooting tools available to them in other portions of the network to resolve issues faster, and with less finger pointing.

Security controls have been moved further away from the hosts then we would like

A best practice for applying security policy is to apply controls as close to the source as possible. Think of this analogy – Your kids are blasting Radio Disney from their computer. Which of the following do you do?

A. Turn down the speakers at the source

B. Distribute earplugs to all members or the household

Of course, the obvious action is to go to the source, and apply a control (turn down the volume, and tell the kids to clean their rooms). The same principle is valid on the networking side. The best practice is to apply security policies such as VLAN ACL’s and TrustSec policies directly to the switchports that host your switches. Before the Nexus 1000V this was impossible to do in ESX, and forced many environments to move security controls further up into the distribution layer. The side effect of this was that now the security stance from host to host inside ESX clusters was diminished.

The Nexus 1000V brings something called port policies to the table to address this. What these are is pre-configured application security descriptions that are available to you systems administrators to apply in a point and click fashion. Once these policies are applied to the virtualized host, they follow the host where ever it is moved in your virtual cluster.

Provisioning and integrating the networks of VMware ESX clusters with classic networks for most is challenging at best

I wrote an article in march about this specific issue in my post – Challenges integrating VMware into Cisco networks . The core of this issue is that in general that the network integration portions of VMware ESX clusters is not really designed to address server teams , or network teams. In fact, you need to be pretty savy with both portions to successfully integrate VMware clusters into your network. In the real world, you generally find people that are good at one or the other, not both.

By putting a Nexus 1000V in your VMware clusters, you know give the networking teams something they can understand without having to learn Linux, and how it handles bridges (key to understanding ESX networking). With a Cisco switch running virtually inside your clusters, network teams can follow standard core / distribution / access models with the access layer now residing inside the ESX clusters. The network teams can also leverage their existing LAN switching skills for integrating the virtual switches in the clusters with the existing Data Center switching fabrics.

With these roadblocks addressed, Cisco is moving to further the DC 3.0 vision

To realize the DC 3.0 vision, the network inside of VMware clusters had to be under control, and follow the same architectural guidelines that the rest of our network is subject to. With the Nexus 1000V this is now a reality. The next steps withing the DC 3.0 vision to are to extend virtualization and mobility throughout our storage fabrics, and to continue to extend virtualization to the network as a whole, as well as focusing on application virtualization and acceleration to truly realize the vision of cloud computing in the data center.

On the storage virtualization side, Cisco will be using a technology called FlexAttach to enable virtual and physical hosts to change locations in the datacenter without storage team intervention (more on this in a near future post). And on the application virtulization and acceleration side, expect Cisco to continue to enhance it’s existing Application Control Engine (ACE) and Wide Area Application Services (WAAS), and further integrate these into their virtualization offerings.

Want to learn more ?

Introduction to VN-Link network services – Cisco.com

Nexus 1000V overview – Cisco.com

VMware distributed vNetwork switch demo – VMware.com

Challenges integrating VMware into Cisco networks – colinmcnamara.com

Douglas Gourley speaking about how Cisco and VMware will drive Cloud Computing in the Data CenterSimilar Posts:

• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?

--Colin McNamara

Cisco releases Nexus 1000V virtual switch for VMware

Sadly, I have the unfortunate situation of being tagged as an insider (work in the partner community). So I have to play nice, and cannot reveal any juicy tidbits. Suffice it to say, that Jashree Ullal and Doug Gourlay weren’t pulling anyones leg in the past two years when the DataCenter 3.0 vision was established.

Stay Tuned

–ColinSimilar Posts:

• Me and the Nexus 7000 last week at the Data Center VT
• Thanks and farewell to Jayshree Ullal
• Usability features in Cisco’s Nexus 7000
• CCIE Party 2008 Recap – Cisco Live Networkers 2008
• Colin has left ePlus Technology
• Sun Project Backbox

--Colin McNamara

BIG Cisco – VMware announcement – 1:30 Pacific time

He writes about “the VLAN leaking myth” and how it encourages clients to utilize physically separate network infrastructure in the DMZ’s. Now first things first, I wouldn’t call VLAN leaking a myth. At one time it was a very real and serious vulnerability that was exploited by overflowing the capacity of the switch you were attacking, and causing it to “downgrade” from switch to a hub. Once this happened you now had access to previously protected devices, as well as having the ability to sniff data as it passed through the shared hub backplane.

As he mentions though, this is 8 years ago. Most switches have evolved to the point where backplanes far exceed the traffic that could ever be injected into their switchports. Even beyond backplane enhancements there are many ways to further firm up your security stance – Virtual Device Contexts, not using Layer 3 SVI’s on a DMZ VLAN, utilizing PVLANs, using port security, virtual routing instances, and many more. Of course, there are still many other attack vectors that still remain, but can be mitigated by utilizing features built into the majority of enterprise switches available today.

I think the real question is not “are VLANs safe in a DMZ”. The important question is have you mitigated the probability of compromise (the actual threat) to levels that are acceptable to your business. This question remains whether you have a standalone switch or not. So many times we hear about risk risk and more risk. But risk alone is meaningless in a business context. What is important is combining risk with likelihood. For that I like to use a simple table to come up with the true threat.

For example, as I drive to Fry’s there is the risk of me dying due to a car crash. The impact of me dying is very high (risk) however the likelihood of an accident is low, and furthermore I reduce (mitigate) the latent risk (threat) by wearing my seat belt. So all in all the threat of me dying on my way to Fry’s is pretty darn low.

In a business context this may be that I have public facing web servers and network devices in my DMZ. The impact of them being compromised is that my public image may be tarnished for a short time, and my end users may lose productivity if they are not able to VPN into work, or access the Internet while on premise. I mitigate this risk by using firewalls and both host and network based Intrusion Prevention Systems as well as implementing best security practices on my network and systems devices. The latent risk (threat) remaining is at a level that is acceptable to the business leaders, so the system is allowed.

One question that I have seen coming up more often as we move towards fully virtualized data centers is centered around commingling of virtual infrastructure. There are some hard questions which challenge some practices that we have held true over the years.

• Should you allow sharing of physical memory on a host virtual machine between an internal and DMZ server?
• Should you allow virtual infrastructure from multiple security zones to share a storage array or cluster of arrays?
• Should you allow multiple virtual switches in different security zones commingling on the same ESX or Hyper-V cluster?
• Should you allow virtual firewall and load balancing instances protecting internal and external zones to reside on the same hardware?
• Should you allow virtual routing instances from multiple zones to share a physical infrastructure?

In the past world of standalone systems, the additional cost of providing a wholly separate infrastructure for DMZ environments was relatively low. Each system generally had internal disk, or at most direct attached storage. Network devices themselves were scaled down to support one chassis one function. This fit quite neatly into the Enterprise Composite Network model that was quite common from 1999-2003.

Now, many data centers have moved to the Service Oriented Network Architecture (SONA). In this model the cost of a virtualized data center is primarily focused on foundation elements such as the virtual storage and virtual fabrics, virtualized network, and virtual systems elements. The cost of providing additional virtualized services off these elements is low, however the cost of duplicating the physical infrastructure is quite high on both the capital and operational levels. This is forcing the technical and executive leadership at many companies to take a long hard look at the true threats they are facing in previously physically separate security zones such as DMZ’s, Financial and other secure zones. In the end, they are having to decide whether the threat remaining after their security controls is worth duplicating hundreds of thousands of dollars worth of infrastructure or not.

These are hard questions, with really no single good answer. My gut feel is that over the next few years we will continue the move towards the fully virtualized data center where components such as memory, PCI-X buses, storage and network devices are even further decentralized. This will make the cost of duplicating the infrastructure more and more significant, causing consolidated data center (or compute) fabrics to be the norm. At this point the discussion will move away from securing zones by creating separate infrastructure, to providing end to end security, starting integrated application level security, maybe with TrustSec or a dirivative, all the way down to securing the data at rest on disk. For the time being however, the best we can do is sit down and do an honest appraisel of our security stances, mitigate what we can, and do our best to design data center architectures that provide the flexibility of implementing whatever choice the technical and business leaders agree on.Similar Posts:

• Moving towards a Green Data Center – Truth behind the hype
• Cisco’s Cloud Computing Offering
• About Colin McNamara
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

In the past couple years, VMware has changed from a product hidden in development and testing environments to a full fledged enterprise computing platform. It brings many benefits to the companies that implement it, however with those benefits come changes to the access layer of your data center. Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers – Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco – Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware – Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe – Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Arista Networks – Their approach to cloud networking
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Resume – Colin McNamara, CCIE #18233
• New features in VMware 3.1

--Colin McNamara

Challenges integrating VMware into Cisco networks

What useability enhancements do you feel are the most beneficial?

1. A separate, IP enabled, Management Interface. This has been a long time coming. The out of band management interface is very similar to a Ilo card in the HP world. it is effectively a supercharged console server that happens to site on the backplane of the sup engine. I am sure whoever pushed this feature through is going to get flowers one day from a Tech who DIDN’T lock himself out because the management interface was effectively a separate system.
2. Finally, a functionally USB Interface that I can transfer IOS (well, now NX-OS) images through. Everyone has a USB key nowadays, even my Grandmother has one, it will make life so much easier when I can have a 4 gig key with me that has most IOS / NX-OS  versions and my common configs and just pop them right in.
3. The integrated Cabling system is CLEAN. I love that it forces you to reserve the appropriate space for cabling, and that there finally is the possibility to avoid the flying spaghetti train wreck we see so often in Data Centers.
4. Front to back Cooling. The cooling design is well thought out. I liked the fact that it draws from directly above the front floor and exits rear top.. This should help out in raised floor data centers that have a large temperature gradient as you move to the top of the rack. It also negates problem of having multiple 6500 chassis side to side and having warm air blowing from the exhaust of one 6500 to the intake of another 6500.
5. Fan Slots are now placed where it is IMPOSSIBLE to cover with cables. I would say 7 out of 10 times when I walk into a new customers Data Center I find that there are cables run directly over the fan tray with no slack. That is not a failure in design per say, but it could have been avoided. With the Nexus 7000 fan trays in the back the problem is solved before it is created.
6. Power supplies are in the back . FAR away from the data cabling. It never fails that 20 amp circuits get uncomfortably close to copper cabling. By moving the power supplies to the back side of the chassis, this becomes a mute point and we remove any shadow of a doubt about EM interference causing craziness in our cabling.
7. This one sounds really mundane, but a quick heads up grouping of status lights. In the past these were normally in a position where you had to squat down to see them, or they are obscured by cables. Buy putting them on the front of the cable tray assembly it ensures these will always be visible.

What can we focus on now to make it a better platform?

1. One thing that worried me a little was the placement of the compact flash cards in the supervisory module. For those how haven’t it up close look at this picture of the chassis  and look for the Grey cover midway up the sup modules in the center slots. Behind them are two flash cards, one for system partition extension, and one to dump log files into. Having these cards available are great features however I could see an operational process of security rotating out the log partitions, or more likely and engineer pulling the flash card after dumping some data for analysis to it, and then pulling the wrong card by accident. Having a simple strap (like the screw downs for power supply plugs) or something similar would go along way towards mitigating that risk.
2. Continue with the spirit of innovation that has defined Cisco over the years. Cisco has consistently came out with or acquired and integrated many great products that directly address the needs of the market place into the product line (MARS, ASA, AireSpace, TelePresence, MDS, ACE, Etc) but frankly the last GAME CHANGING product that set the industry on its heals and forced everyone to rethink how we utilize technology to accelerate business as a whole was the acquisition of Selsius and the introduction of VOIP as an enterprise class product to the world. I remember having the hair stand up on my arms from the excitement of going up against Avaya and Nortel back then and fighting that uphill battle, educating customers and peers about this “new thing called VOIP and how CallManager (now Unified Communications Manager) is your ticket towards productivity.

   When we talk about the Virtual DataCenter, I/O Virtualization (FCOE) and VFrame Automation it is not just another incremental improvement of existing technology. It is a paradigm shift, a leap ahead, a GAME CHANGER. I get the same chills that I did when VOIP was new because I know that those are technologies that will force us to rethink how we approach computing and data systems. These technologies are to the Data Center what IP telephony was to the PBX, and Cisco is the only company with technologies and engineering know how in all the verticals necessary to pull this off.

Similar Posts:

• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Moving towards a Green Data Center – Truth behind the hype
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Usability features in Cisco’s Nexus 7000

* Solid State Drive (SSD) boot support
As initially discovered last month, VMware will make available a special version of ESX Server (mentioned with terms like ESX Lite and Embedded ESX) for OEM vendors, to be installed into bootable Solid State storage devices (flash drives, etc.). This option will allow creation of ESX Server hardware appliances for easy jumpstart, granting smaller form-factors and improved reliability.
Dell, IBM and possibly other vendors will offer this option at announcement time in Q3 2007.
* DMotion
Unofficially introduced with ESX Server 3.0.1, in its first version DMotion is a special VMotion operation only capable of moving running virtual machines from an ESX Server 2.5.x host to a new ESX Server 3.x., without shared SAN LUN mandatory requirement.
In ESX Server 3.1 this capability will be extended, allowing hot migration of running virtual machines between ESX 3.1 hosts through the Ethernet cable.
* Patch management system for host and virtual machines (Update Manager 1.0)
ESX Server 3.1 will finally introduce an automated patch management system called Update Manager. This solution will be able to update both host itself and virtual machines (both Microsoft Windows and Red Hat Enterprise Linux).
Update Manager will look for available updates from Shavlik Technologies website (a possible acquisition after IPO), and will allow VI administrators to decide which patches to deliver to virtual machines.
Before applying them, Update Manager will take a snapshot and will even rollback automatically if something goes wrong.

(this product was originally codenamed VM Integrity and its developement started more than one year ago, when virtualization.info discovered it in June 2006)
* VMware Consolidate Backup (VCB) and VMware Converter 4.0 integration
VirtualCenter 2.1 will now allow restoring VCB images with an integrated version of VMware Converter, which reaches 4.0 release number.
* Server consolidation advisor
VirtualCenter 2.1 will expose a server consolidation assistant able to analyze which physical machines should be converted in virtual ones, and where to move existing VMs among available hosts.
(note that with this feature VMware is further extending competition with PlateSpin, covering both features with PowerRecon and PowerConvert)
* Guest OS disaster recovery capability
VirtualCenter 2.1 will be able to recognize a failure inside a virtual machine and restart it through VMware HA module.
* Support for VMware Server 2.0
VirtualCenter 2.1 will be finally able to seamless manage both ESX Server and VMware Server 2.0 hosts.
* Lockdown Mode
ESX Server 3.1 will expose a new security feature to completely disable local administrative account after a VirtualCenter 2.1 takes remote control.
* Power saving capability (Distributed Power Management)
VirtualCenter 2.1 will introduce a new resources utilization analysis feature, able to verify when a physical host can be powered off, VMotion-ing its virtual machines on other hosts without impacting performances.
* Support for Cisco Discovery Protocol (CDP)
VirtualCenter 2.1 will be able to recognize and use CDP to discover physical and virtual network topologies.
It stays unconfirmed if ESX Server 3.1 will already expose new virtual network architecture, allowing 3rd party virtual switches, as it will be announced by Cisco CEO at VMworld 2007.
* Support for 10Gbit Ethernet network cards
* Support for TCP/IP Offload Engine (TOE) network cards
* Support for network load balancing algorithms
* Support for 200 hosts and 2000 virtual machines
* Support for 128GB RAM per host and for 64GB RAM per virtual machine
* Support for SATA storage devices
* Support for N_Port ID Virtualization (NPIV)
* Support for VCB over iSCSI SANs
* Support for IPv6 in virtual networking
* Support for Para-virtualization guest OSes

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• Interesting TechWise TV episode on virtualization
• Cool new features in 12.2(33)SXH
• VMworld 2009 Schedule

--Colin McNamara

New features in VMware 3.1

Now this system work pretty good, except when we have to do usability testing. Its just not the same on a XP image, as it is from your laptop, on wireless, at home.. etc etc. So we decide we need some sort of VPN solution into our Exchange staging environment. After not alot of thought, I decide to set up a linux PPTP server. Its a protocol that pretty much everybody can use, pretty lightweight, and free .

I have set these boxes up before, and its not terribly hard. Until I had to do it on VMware. Let me set the stage for you – Redhat AS 3.3 / VMware Esx server / VMware tools installed / 1 nic . One NIC generally isn’t best for a VPN server, so I used Virtual Center to deploy a 2nd NIC to my PPTP box.
For anyone reasonably familiar with Redhat or Manadrake, we are used to installing well known hardware and seeing Kudzu add it on boot. Not this time. I rebooted, Kudzu came.. Kudzu went. I ran Kudzu manually no dice.

I know I know, auto anything never works… So I moved on to manually defining this network adapter. The first thing to check – is it actually plugged in. LSPCI reports the adapter as plugged in, and recognises it as a pcnet32 adapter. Normally I would take this as a positive sign, but not today. I issue IFCONFIG -a , network adapter is still not found. I was definately feeling a bit frustrated at this point. To be sure it wasn’t me, I bugged a friend and a co-worker (remind you of a game show?). Anson, who runs the VMware enviornment tried removing VMware tools, adding them…. still no dice. Shad tried to help through Yahoo, although to no avail.

By this time, I am frustrated, feeling like a moron, and ready for a break.
I went to get lunch from the Cafe, and settled down to eat at my desk.
Funny thing, how when you aren’t thinking about a problem is when you normally think of the solution. Let me give you a little background. Earlier when we were troubleshooting this problem, I googled this usenet post -
http://content.ix2.net/arc/t-4236.html . This fellow describes his headaches with debian, in which he had a very similar problem.

His problem, along with mine was that the proper modules werent loading for his network devices. He had tried, along with me to add the module listing to /etc/modules.conf. The one thing however that I hadn’t tried was manually loading pcnet32 using modprobe.

needless to say, modprobe pcnet32 now sits in my rc.local file. Its lame, but it works.

–Colin
Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• New features in VMware 3.1
• Dragged to the hospital
• Cisco releases Nexus 1000V virtual switch for VMware
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Challenges integrating VMware into Cisco networks
• My CCIE Storage Shopping List

--Colin McNamara

Redhat AS3.3 and VMware ESX network issues