“it looks like one of the key features not on the list of components for the California boxes is going to be a red discount pen”

Timothy references sources who have obtained a price list and shared it with “El Reg” . I wish Timothy would have contacted an actual Cisco Unified Computing System Advanced Technology Partner, because any partner that is involved in the launch could have explained to him the concepts of List price (List), Manufacturers Suggested Retail Price (MSRP), and Purchase or Buy price.

In this article I want to dispel the myths of server and network manufacturer pricing, demonstrate the true cost of building a data center with blade systems, and at the end provide a cost comparison between legacy server vendors options and Cisco’s Unified Compute System.

First, lets go over some the basic concepts of vendor pricing. At the end of this you should understand the difference between list price, manufacturers suggested retail price, and purchase price.

List Price

List price is a high level number that Cisco publishes weekly in its global price list. The purpose of this list price is to provide a uniform price list across all product sets that Cisco offers. The most important thing about list price is NOBODY EVER PAYS LIST PRICE. Let me repeat that again NOBODY EVER PAYS LIST PRICE. Are we clear? This is similar to list price on a car on the car lot. All list price provides is a starting point where a Cisco partner and a customer can negotiate a common discount and end up with something close to (generally at or below depending on technology type and yearly spend) MSRP.

Manufacturers Suggested Retail Price (MSRP)

This concept is something that anyone who has purchased a car before is familiar with. The number that is on the window of the car when you look on the lot is list price. The first number the dealer brings up lower then sticker is MSRP. Depending on the popular of the product, the competition in that particular space, and the negotiating power of the customer you will either pay that price, or some percentage below. For example if you are buying one new car you may have the negotiating power to get the price to drop 5% off of list. If you are buying 200 new cars (say a fleet) you have significantly higher negotiating power, and you may be able to drop the price by 15% of of list price.

In Networking Sales MSRP is significantly less then list price. A good exercise to see what this number is, is to find a device, say a WS-C3560E-12SD-E (3560 with 12 Gig SFP ports and 2 10 Gig ports) in the Global Price List. You have access to this at any partner level at www.cisco.com/dprg . (my point here is that this is no big secret). As of Friday June 12 2009 the LIST price for this product is $19,995.

Now take that same part number - WS-C3560E-12SD-E and pop it into your google search window. Within the top four links I found this product for $12,434.15 . This price is for  pure fulfillment, with no value added consulting or design work from you local Cisco partner.

If you do the quick math, this price difference is equal to 38% off of list price. Come to your own conclusions, but it would be safe to say that this could be considered MSRP for Cisco products.

Purchase / Buy Price

Buy price is just that, the price at which the customer purchases (buys) the product. This is can be at MSRP, or if the customer is buying significant amounts of hardware at a time, or if there is a “special” (programs and incentives) going on the number could be slightly lower then MSRP.

Percent off of list differences between legacy server vendors and networking vendors

This is where the biggest confusion is coming from. Legacy server manufacturers  have set their list prices much closer to MSRP then networking vendors (remember, MSRP is the price where most customers purchase at).

Why is this? In the networking space, vendors have historically created their own processors, ASICS and boards. This means that the sales discussions are feature to feature. It also meant that you had to have a conversation with the networking vendor or networking partner to properly size your network devices and get a quote – which is around MSRP, not List price.

In the legacy server space, especially the majority of the x86 server space, the market has been essentially commoditized. E.G. – You can buy an intel based server with X amount of memory and hard drives that will perform roughly equally from any of the main manufacturers. That made it much easier for a sever admin to just pull a price off of the web and compare. So what the server vendors ended up doing is setting their list price  only slightly above MSRP.

What this translates to is the list price, between legacy compute vendors and Cisco will be drastically unequal. What is equal is MSRP, or the generally accepted purchase price by common customers.

Why did Cisco set the list price of UCS higher then the legacy server manufacturers?

For the vast majority of its sales, Cisco relies on what is called the channel model. This means that Cisco partners with local Value Added Resellers (VAR’s) who sell Cisco’s products and then provide consultative services to design and implement them in customer networks. Most customers who purchase any regular amount of Cisco product either have a general expectation that they will buy Cisco product at a certain percentage discount off of list and sometimes the partner and customer have entered into purchasing contracts which require that all Cisco product is provided at a specific discount off of list price.

If Cisco decided to set the List price at a small percentage lift over MSRP, this would cause a problem for the entire channel. This would be especially hard for any customer who had a contract to buy product at a specific discount. What would happen is contracts would have to be renegotiated, which generally takes months and is about as fun as pulling teeth.

The second reason for setting list price the for compute the same as list for network is quoting. Right now, if you buy hundreds of different Cisco devices through a reseller it is very likely that the discount is going to be the same across all products. This makes the mechanics of sales much simpler, because you don’t have a lot of math in the quote (this can cause errors). On the customer side, having one set discount makes it much easier to compare quotes and to ensure that they are getting the best deal possible. In short, sticking with Cisco’s current list pricing structure benefits both the customer and the partner.

Now that we have set the record straight on list price, MSRP, and Buy price, lets take a deeper dive into what components make up a blade system powered data center. And then we will compare the price structures of both.

Components of all Blade Systems

Blade Server – The compute blade where commodity silicon elements such as the CPU and RAM are housed. As of writing this article, the latest high performance blades from all major server manufactures support two xeon 5500 processors (Nehalem) and DDR3 memory.

Mezzanine cards – These cards take the place of PCI-e cards in a rack form factor server. In a blade system these provide data network and storage network connectivity. They attach to the blade itself via proprietary connectors that implement either PCI-e 8 or 16 lane connectivity at the time of writing. In some cases other functions such as IO accelerators can also be attached in the mezzanine card form factor.

Blade Enclosure – This is functionally a tin can where eight to sixteen blades are placed. It also is used to provide a centralized power distribution fabric, as wells as slots for interconnections of data and storage network devices.

Data Network Modules – These are effectively ethernet switches that have been miniaturized to fit into the tight confines of a blade enclosure. Classically they have provided 1 gig connectivity to the servers, and 10 gig to the distribution layer, however with Nehalem processors and VMware there is a move towards presenting 10 gig connections to the server, and multiple 10 Gig connections into the distribution layer.

Storage Network Modules – The local disk in a blade server is classically anemic. To provide higher IOPS (input outputs per second) to disk, Fibre Channel connectivity is extended by taking SAN fabric switches and miniaturizing them to fit into the blade enclosure.

Data Network Distribution – If you have multiple blade enclosures there is a need to connect them together at a reasonably high bandwidth. To serve that need a variety of 10 Gig distribution switches are provided from all server manufactures at varying cost and performance levels.

Storage Network Distribution – Along the same lines of the data network distribution, SAN fabric switches have to aggregate up to a SAN distribution layer, or if the installation is reasonably large a “director” class SAN switch. This allows all the blade enclosures to see the same storage network, as well as providing for deterministic storage network performance as you scale out.

Management Infrastructure – All manufactures have a need to manage and monitor all of the devices that comprise their blade system. Many manufactures have multiple management modules per blade enclosure.

Comparison of Costs – Cisco vs Legacy Server Manufacturers

The funny thing, is that many people have assumed that Cisco’s Unified Computing System will be priced higher then legacy server manufactures products. In my mind this is because they associate higher quality with higher price (basically the Mercedes vs Kia discussion). Here is something that will shock you - it costs less to buy an entire blade system through Cisco then to buy from the legacy server manufacturers.

When people hear this, they are puzzled. How can two server manufacturers, who buy their CPU’s from the same company (Intel) and their memory from the same fabs end up with different prices? The answer is elegance in engineering. Lets go through each of the elements of a blade system infrastructure and find out where the costs are. More importantly lets look at where Cisco has innovated to provide higher performance at a lower cost.

Dollars and Cents – How much is the cost difference

I worked up two quotes recently. These quotes included all elements required to build an end to end blade system using both legacy server manufactures devices, and using Cisco’s Unified Computing System. I have broken out two scenarios.

8 blade servers - Cisco wins with a savings of 11%

In this scenario the cost of servers and enclosures were fairly equal. The cost savings started racking up as storage and data networking devices were included, as well as base management software was taken into consideration.

320 blade servers - Cisco wins with a savings of 31%

With 32o blade servers the same cost savings seen in the 8 server scenario were amplified. Economies of scale translated into significantly less devices being required to support the individual compute blades. This resulted in 31% savings compared to the legacy server manufacturers.

Summing it up

Cisco has entered into a highly competitive server market by taking an elegant approach to its blade systems. This approach lowers the purchase price of the UCS through reducing the amount of components compared to legacy server manufacturers. I know that there is a lot of misinformation flying around, and I hope this helps to set the record straight on the pricing of Cisco’s Unified Computing System.Similar Posts:

• Cisco Nexus 4000 Blade Switch
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Cisco introduces the C-Series Rack Servers
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Cisco’s Cloud Computing Offering

--Colin McNamara

Confusion about Cisco UCS pricing – Setting the Record Straight

Why do I say that? It is simple, every server that moves from a corporate data center into a cloud provider is a switchport and fibre channel port (and now server) that is not purchased from Cisco. More so, each system that is moved into the cloud hurts secondary sales of security and content switching products.

The promise of enterprise cloud computing

The ability to dynamically scale enterprise compute workloads while only running a “right sized” private infrastructure is top of every CIO’s mind. This is the promise of cloud computing in the enterprise space. However, right now most cloud offerings are too new, and lack the critical integrations with VMware or XenSource (the two most common enterprise virtualization platforms) to make a serious dent in Cisco’s revenue stream. But fast forward 12 to 16 months and the kinks will be worked out. Projects that would previously have required new capital infrastructure will be restructured to use cloud providers as an operational expense. This will present a real threat to Cisco’s revenue moving forward.

John Chambers and his team of technologist are not new to this game, this is not the first threat to Cisco’s sales model. And I am sure that it won’t be the last. So if I was in their shoes, what would I do? (and more specifically, what do I think they are doing)

Create a compute platform that can power the cloud at a much lower cost that my competitors

Cisco publicly announced their computing offering, the Unified Computing System in March of this year. The promise of the UCS is to minimize power, cooling, capital costs and management overhead of data center compute. Looking at this new product line from an enterprise sales perspective it makes sence. For Cisco to continue with their growth plans they had to choose to enter the Compute or Storage markets, with the compute (server) market being the logical step.

While the Unified Computing System is well placed as an enterprise computing platform, I think there is a larger goal in mind. The large goal is to make a platform that can be shared by Cisco’s largest enterprise clients in their emerging private clouds, as well as by Cisco itself for it’s own cloud offering. By producing their own servers, with technology that Cisco alone has access too (memory expansion / hypervisor bypass) Cisco sets themselves up to have both lower hardware costs in their own cloud, as well as lower operational costs (power/cooling). This will provide Cisco with higher margin at the same price point as their competitors.

Distribute application aware network devices at customer locations

Cisco already has a significant edge over any competitive cloud offering. A vast majority of enterprise customers already run Cisco routers, switches and firewalls. If Cisco decided to say, port the TCP optimization code from their WAN acceleration platform into IOS, and configure it to work with their own cloud offerings this would give them an immediate leg up on the competition. Combine this with the existing WAAS auto discovery and Cisco could conceivably automatically integrate a cloud based caching offering with a customer’s onsite devices.

Create an application centric cloud security model that can be integrated with virtualization platforms

Last year Cisco announced a new approach to security called Cisco TrustSec. This technology includes a change from layer 4 based acl’s to an application focused role based implementation. This is applicable in the cloud environment because it provides a standard integration for controlling the access to and mobility of applications as they travel between public and private clouds.

An interesting side bar, is the fact that when integrating public and private clouds, there will always be applications that you want to keep on your internal cloud. The easiest way to do this is to put some sort of meta information on the virtual server containing a flag that this server should only run on the private cloud. With VMware there are fields that are used for DRS that can house just such data. I would not be surprised that with all the work that Cisco and VMware have been doing together if this was not implemented with vSphere (Virtual Infrastructure 4).

Learn as an organization how to profit from a SaaS model

I think this last piece of the puzzle has been overlooked by many people. Cisco already has in house experience dealing with a massive Software as a Service (SaaS) offering – Cisco WebEx. In acquiring WebEx Cisco also acquired the talent and technology behind the worlds largest collaboration platform. Cisco should be able to take the lessons learned from running and improving this platform, and apply them to their upcoming cloud offering.

Summary

Cisco has to go to market with a Cloud offering to maintain long term viability as a company. When they do they will have the benefit of lower cost of building and operating the grids that their cloud offering will run on. They will be able to leverage millions of Cisco network devices in their current install base as well as provide application centric security integrated with these same devices. And most importantly they will be able to use the lessons learned from running WebEx to ensure flawless delivery of an upcoming cloud computing offering.Similar Posts:

• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• VMworld 2009 Schedule
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Cisco releases Nexus 1000V virtual switch for VMware
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

--Colin McNamara

Cisco’s Cloud Computing Offering

Brian Schwarz (SAVBU) and I were guests, and had the chance to talk about Cisco’s Unified Computing offering. It was my first time doing a podcast, and I have to say it was a blast.

If you can spare the time, I recommend you check out it -

Cisco Interactive Network – Unified Computing PodcastSimilar Posts:

• It’s on like Donkey Kong – CCDE practical registration is open
• Cisco’s Cloud Computing Offering
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• CCDE Practical – Beta candidate deadline August 1 2008
• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• VMworld 2009 Schedule

--Colin McNamara

Unified Computing Podcast with Cisco Interactive Network

This March, Cisco formally announced its entry into the enterprise computing market with a new product line called the Cisco Unified Computing System (UCS). I say formally announced, becase the existence of the Unified Computing System might just be the worst kept secret in history. In the months prior, to launch Cisco has been openly talking about the system, as well as many news agencies. The only people who haven’t been talking about the system were those of us who got briefed in early and were under strict NDA. (yours truly being one of them). But now, the multiple layers of NDA have been removed and I am free to talk. So, let me tell you about a new concept called Unified Computing.

The Unified Computing System answers a simple question – “what would you do if you could build a system with no preconceptions”. That same question has been asked over the years by Cisco. The results have given us the Catalyst 6500 line of switches, the Cisco MDS storage line, as well as the Nexus 7000/5000/2000/1000V family of switches.

Nuova – a history of innovation

A couple of years ago, Cisco funded a startup called Nuova (meaning “New” in Italian). The founders of this startup were the same innovators who led the Catalyst, MDS and Nexus products. Not only did they have a track record of successful products, but they also had experience in both types of Data Center networking – Ethernet, and Fibre Channel. This startup took the lessons they learned creating a non blocking, low latency, highly available fabric for the MDS line of SAN switches and created a new line of Data Center switches, the Nexus 5000 that is able to transport Fibre Channel and Ethernet traffic at the same time, through a single adapter.

In April of 2008 Cisco formally acquired Nuova (which in effect was just hiring prior innovators back). Nuova was renamed the Server and Virtualization Business Unit (SAVBU), and the Nexus 5000 was released to market, making Cisco the first vendor to deliver a solution based around the upcoming Fibre Channel over Ethernet standard (FCOE). This pattern of innovation continued, as SAVBU released a virtual switch for VMware (the nexus 1000V) and this January released a remote line card technology called Fabric Extension. Those of us on the inside got the hint that the choice of names for this new business unit (SERVER and VIRTUALIZATION Business Unit) was a foreshadow of things to come.

Unified Computing System enclosure w/ redundant 6120 Fabric Interconnects

Here are some spec’s to get you started –

• Single point of management for all devices in the fabric.
• Virtual machine enabled networks adapters. (VNtag capable)
• Up to 320 B-series compute blades in one fabric.
• Up to 384 Gigabytes of memory per blade (full width blade)
• Server Profiles – virtualize server identities (UUID, WWN, MAC)
• Hardware Assisted Virtualization using Intel’s next generation Xeon (code named Nehalem-EP) processors
• Redundant 10 Gigabit connections between servers.
• Fibre Channel SAN access available to every blade.
• Capital costs up to 20% less
• Operational costs up to 30% less

The UCS does all of this while using 1/3 less components then the competition. What does using less components give you? Less components means less things to buy (lower capital expense). It also means less things to power and cool (lower operational expens). And finally it means less items to manage (lower management burden) How does Cisco do all of this while using drastically fewer components? I think it is necessary to talk about the major components that they system is built from to answer that question.

Cisco UCS 6100 Fabric Interconnect (Nexus 5000 on steroids)

.

The primary building block of the system is the Cisco UCS 6100 Fabric Interconnect. Cisco took the non blocking, low latency, lossless fabric from the MDS that was used on the Nexus 5000 and used it as a building block for the 6120 and 6140 Fabric Interconnects. These fabrics support 20 and 40 (6120/6140)10gig Data Center Ethernet (combined Fibre Channel and Ethernet support).

• 6120 – 20 Fixed 10 Gig Data Center Ethernet ports along with an expansion module that supports native Fibre Channel, or additional Data Center Ethernet interfaces
• 6120 – 40 Fixed 10 Gig Data Center Ethernet ports along with two expansion module that supports native Fibre Channel, or additional Data Center Ethernet interfaces

The 6100 series Fabric Interconnect unifies Storage and Ethernet network, as well as providing supervisory functions for its remote line cards, the 2100 series fabric extenders that are inserted the compute chassis. The other thing the 6100 Fabric Interconnect does is house the Unified Computing System Manager (UCSM).

Cisco Unified Computing System Manager (UCSM)

The UCSM runs on the fabric switches, providing a single point of management for all components in the fabric -

• I/O Fabric
• Chassis and Services
• Adapters and Virtual I/O

This interface can be accessed through either a web based Gui or CLI. It also supports a full API for programatic integration and management of the system. The biggest thing that they SAM gives you is the ability to dynamically provision server attributes down to the compute blades. Attributes that can be pushed down dynamically include CPU UUID, SAN PWWN, Ethernet MAC address, and many more. These items are pushed down through as Service Profiles.One key component of the service profile is the Port Profile.

.

These Port Profiles are dynamically created in the SAM, and most importantly enable you to create virtual network interfaces (vNics) that show up to your server administrators as normal network interfaces. This allows your server administrators to follow their application vendors recommended interconnection topologies.

A great example would be VMware’s recommended topology for ESX. In this topology there are four network intefaces defined. Each for a specific function. This logical topology can be implemented, with all the relevant speed, QOS, VLAN, and security attributes all “pre-configured” for the virtulization administrators, simplifying their virtualization cluster deployments.

Cisco UCS 2100 Fabric Extender (Next Generation FEX)

 The next component in this architecture is the UCS 2100 Series Fabric Extender (FEX). There are two of these in each bladechassis. Each FEX has the capability of up-linking to 6100 Fabric Interconnects with four 10 Gig ports. for a total of 80 Gigabit persecond out of each blade chassis (supporting 8 half width blades, or 4 full width blades). Each FEX is managed as a “remote line card” connected off of the Fabric Interconnects.

Logically, think of your 6500 series switch in your Data Center right now. You have three logical functions, supervisory functions (sup module), a bus for the switching fabric (traces in the chassis) and line cards (6748 for example). You will notice that you only manage the sup module itself. You don’t shell into to each line card to set up backplane interconnects, or to update microcode. You update software and configurations on the sup module, and the intelligence that Cisco builds into its software manages this for you.

The 6100 and the 2100 interact in the exact same way. In this case the 6100′s are the sup modules, the 2100′s are the line cards, and we are running 10 Gig connections to build the switching fabric. What this gives you is a simplified network architecture, which takes elements that would in the past be individually managed, maintained, upgraded, etc and consolidates that into one highly available, high bandwidth consolidated SAN and Ethernet for your Data Center compute needs.

Putting Network Intelligence on the Compute Blade

Cisco will be giving customers three mezzanine card network adapters options for the UCS.

The first card (based is based on a the Palo chipset. This chipset was developed internally at Cisco and performs Network, Storage and Virtual Machine networking functions all on one 10 Gig capable chip. This chipset effectively extends network intelligence into the blade itself allowing for some pretty interesting integrations with Virtual Machine Hypevervisors (VMware, Hyper-V, Etc).

The second option is based on the Menlo chipset in conjunction with either a Qlogic or Emulex Fibre Channel adapter chipset that gives you 10 Gig network access, while retaining strict compatability with applications that require either of these classic HBA’s.

The third option is an “economy” option based on an Intel chipset. This will give the compute blade 10 Gig access.

What will drive your choice of network adapter? I think the biggest driver will be a technology called “VNtag”. This technology is currently in use on the FEX to encapsulate traffic that enters the fabric with a little shim header that communicates the identity of the incoming port (VLAN, QOS, Security info, etc). This shim header passes from the FEX up to the 6100 where the shim header is removed and the frame is processed.

This alone is very cool, however Cisco has taken things one step further and put the ability to impose VNtag’s in the Palo chipset (on the blade) itself. What this gives you the ability to logically attach a virtual machine DIRECTLY to the network. We no longer need a vSwitch, or even a 1000V to give full network functionality to a virtual machine. Effectively, Cisco is giving the network adapter inside of the blade many of the functions of a network switchport. Logically what this does is reduce your network tiers from four tiers in a competitive system, to two tiers in a Unified Computing System.

Cisco UCS B-Series Blades

The compute blades themselves are available in either 1/2 width of full width form factors. As stated in the formal announcement the maximum memory for a full width blade will be 384 Gigabytes per blade. Why is this important?

At release, there will be two option for compute blades.

Half width blade

• 2 quad core Intel Xeon 5500 processors
• 96 Gigabytes of memory
• Two small form factor SAS drives (raid 0 and 1)
• Single Converged Network Adapter slot (connected to redundant fabrics)

Full width blade

• 2 quad core Intel Xeon 5500 processors
• 384 Gigabytes of memory
• Two small form factor SAS drives (raid 0 and 1)
• Dual Converged Network Adapter slots

As CPU’s become more powerful, the host compression ratio’s (the amount of virtual machines you run on one physical blade) increases. The one thing is, you can only over commit your memory to a point. Once you are at that point you have to physically add more memory. Your only option with legacy server architectures is to add another physical server, pay for additional licenses and add more memory to it. Then add that server to the cluster and allow your virtual machines to utilize its additional resources in the pool.

The problem with this is a simple one, Cost. The problem was that the virtual machines needed more memory, but since there was a limitation on how much memory the legacy server could handle, it forces costs skyward to meet those memory needs. In this scenario if the customer had been using a full width Unified Computing blade the customer could have avoided purchasing an additional server and avoided the additional licensing and management cost associated with that additional server.

What does this balance out to in real costs? By allowing for higher host compression ratios there is the capability to avoid a significant amount of cost (50%+) in your compute layer. 

Intel Xeon 5500 ( Code Named Nehalem-EP)

Intel’s next generation Xeon CPU (code named Nehalem) will be driving the Compute Blades. The Xeon 5500 signals Intel’s move away from the legacy “Front Side Bus” (FSB) architecture into what is called “Intel Quick Path Interconnect” (QPI). Intel’s Xeon 5500 with Quick Path Interconnect changes a couple key things from previous generation Xeon procesors.

• high bandwidth, full mesh, routed interconnect between CPU’s instead of a low bandwidth bus
• DDR3 vs DDR2 memory and moves the memory bank adjacent to the CPU’s for higher performance
• I/O Hubs are now dedicated for network and storage interconnects

First and Second Gen Xeon vs Xeon 5500 (Nehalem)

.\

If you look at the picture above, where the last generation processors are to the left, and the processor that will be in the UCS is on the right, you will notice a couple key items. I think the most important items to point out is that the bandwidth in pretty much all directions is superior in this architecture. If you think about this as a network on your server itself, you can see how moving from a bus based network, to a full mesh routed network has significant performance advantages. Stay tuned for later posts where I will talk about some of these advantages in detail.

Hardware Assisted Virtualization

One of most important new features that Intel is bringing to the table is the notion of hardware assisted virtualization. Intel has created a couple key technologies to address the following problems faced in virtual environments.

Processor Virtualization

Currently to virtualize an operating system, we rely on the hypervisor (ESX, for example) to accomplish two key tasks – Ring Depriveleging and Context Switching. When we are talking about CPU overhead in virtual environments this is what we are talking about.

Memory Virtualization

Again, in current environments the hypervisor is used to abstract memory. ESX uses a technique called page table shadowing to virtualize the physical memory. This again however adds inefficiency to the process and shows up as virtualization overhead.

I/O Device Virtualization

As our virtual machines go to disk, we are again presented with a situation where the hypervisor has to abstract disks and networks presented tot he physical server, and connect these to the relevant virtual machines. Again, this shows up as virtualization overhead, lowering the efficiency of our virtual servers as well as introducing I/O sprawl.

Solving virtualization overhead problems

With the Xeon 5500 on Cisco’s Unified Compute System blade you will be able to use processor features to solve the problems listed above. VT-X is used to extend CPU virtualization down to the physical CPU, Extended Page Tables, Cisco Memory Expansion and DMA remapping is used to speed remove memory access overhead from virtual network and I/O interfaces, and I/O devices sharing through VT-C is used to integrate the physical network with the virtual network transparently and with much less overhead.

Summary

Cisco’s entry into the computing space is not a “me too” entry into a commodity x86 market. It is a well thought out strategic move unifying storage, network and compute functions in a unique way that will differentiate the Unified Computing System from other compute offerings. I expect some fierce debate of the upcoming months as competitors release products to compete. I feel that Cisco’s value proposition of a unified compute / network layer in the Data Center uniquely solves problems that most customers face. And at the end of the day, the vendor that can solve the customers problems is the vendor that will succede.
Similar Posts:

• None Found

--Colin McNamara

Cisco’s Unified Computing System – It’s not just a blade center

It is truly my pleasure and honor to join Arista Networks as the President and CEO of the company. I have known founders, Andy Bechtolsheim and David Cheriton for a decade and worked with them closely during their Granite/Cisco days. To me, Arista is a symbol of an exciting, innovative silicon valley start-up company at the brink of pioneering new models for cloud networking.

After corporate life and managing multibillion dollars of business at Cisco Systems, you might ask why I chose Arista Networks? Three things really drew me here:

1. A top-notch and talented team, with focus on best of breed products and innovative EOS™ (Extensible Operating System) technology.
2. Working with Andy Bechtolsheim, and our long-standing 20+ year professional kinship.
3. A truly unique opportunity in our industry to build and grow into a great company!

Inside of Cisco Jayshree proved herself as a leader that could get results. There was even speculation that she was tagged to be John Chambers eventual replacement. I have a feeling the same qualities that drove her success inside of Cisco will follow her as she explores this new opportunity.Similar Posts:

• Thanks and farewell to Jayshree Ullal
• Arista Networks – Their approach to cloud networking
• Heading home – Cisco Live 2008
• About Colin McNamara
• What does it take to pass the CCIE exam?
• Challenges integrating VMware into Cisco networks

--Colin McNamara

Jayshree Ullal takes the helm of Arista Networks

First and foremost it is important to discuss how many networks are right now (click for larger images)

Figure 1.1

This is a pretty common setup, with 80% of so of systems still physically attached to a mix of 100 and 1000 Megabit access layer switches. The other 20% of virtual systems are attached as through blade center switches with 10 gig to distribution or larger (8-16 core) systems with bundled uplinks to the distribution switches. Service aggregation such as firewall, load balancing, and wan acceleration , image deployment, monitoring / management and other key Data Center services generally provisioned off 1000 Megabit ports in the distribution.

Last but not least, a shadow storage network runs connected to a small percentage of physical servers, and connected to all of the virtual servers via Fiber Channel, iSCSI, or NFS presentation. One thing to keep in mind that all of these elements may not be configured in the “optimal” SRND setup, but it is something that you can expect to see in a real life network today.

What’s so special about Vi4 Application vServices ?

Figure 1.2

The one new feature that is going to throw your network on its heels is vApp. Imagine if any application could be installed on any server in any location of your network at any time. What vApp enables you to do is create a portable application, similar to a Java application that installs in your web browser. But this application can be dynamically deployed to any virtual system in your Data Center as needed in response to a new application request, or the need to dynamically scale an application. What this means to us as network engineers is that any corner of our networks where virtualization is present can become a hot spot for critical application flows. This introduces a new dynamism to our fabrics which wasn’t there before, and frankly many networks are not equiped to handle it successfully.

Currently, to provide virtual machine redundancy we have VMware HA, where we both monitor the availability of a virtual machine. If there is a problem we can restart that virtual machine on another ESX host. With Application vServices there are many new elements and traffic flows. The two most important ones are vLockstop and vCenter Data Recovery. VMware is taking high availability to the next level by keeping a hot standby VM running on a second physical ESX server. If you think about it, you now are adding both additional latency sensitive heartbeat traffic as well as creating a situation where your storage traffic flows can be highly volatile. Additionally vData Center Recovery will be throwing traffic in new and interesting ways across your links.

Adjusting your network designs to deal with the cloud

Figure 1.3

First and foremost application virtualization needs a front end, in network engineering circles we have been handling this successfully for a long time with content switches (load balancers). These provide the logical rallying point for dynamic cloud applications. Since more and more systems will be utilizing these services it is important to ensure that your current content switches have headroom to grow, and if you don’t have any content switching capabilities, it is probably time to take a look at adding them to your data center.

Since applications can exist in any corner of the network, dynamic provisioning of storage and network connections has become critical. Maintaining “shadow” storage networks can provide some dynamic access to storage, however it is now becoming advantageous to virtualize your storage fabric along with your systems and network devices. Fibre Channel Over Ethernet (FCOE) provides just that.

If you look at figure 1.3 above, you will notice a new color introduced into the diagram, as well as the “shadow” storage network removed. This is possible because all the orange links run Data Center Ethernet (DCE) which provides a lossless path for FCOE to follow. The ESX servers now only connect into the Nexus 5000 switches. As you can see, we also have removed the shadow storage network, as it is now consolidated onto our new data center fabric. There may be use cases where we need to extend classic fibre channel connectivity out to certain hosts, and we can do that of the Nexus 5000. However if all possible it is advantageous to utlize FCOE to gain storage mobility and higher bandwidth for your hosts.

You may notice that our uplink counts have doubled. Since we are moving both storage and data traffic over the same links, as well as supporting vMotion and other bandwidth intensive network applications it is time to make the push to port channeled 10 gig adapters. Luckily prices have dropped considerably, where it is not cheaper to use 10 Gig then to bundle 8 1 Gig adapters together.

Last but not least you will notice the core switching is a different color. This is because the Nexus 7000 has found its home in the data center. I wont go to deep into the nexus as that is an article in and of itself. What I will say is that it is the best platform to use to aggregate the amount of 10 Gig links that are populating the data center in a highly available fashion. If you want to learn more about the 7000 I recommend reading these previous articles here here and here, as well as Cisco’s Data Center Switching page.

Your network once Virtual Infrastructure 4 (Vi4) and the Nexus 1000V are released

Figure 1.3

Fast forward to early summer 2008 VMware Virtual Infrastructure 4 (Vi4) and Cisco’s Nexus 1000V are released. Of course a new major version of VMware running your compute cloud, application vServices are in effect, vLockstep is running, and many other features that go with the platform such as the Nexus 1000V.

The Nexus 1000V brings a long missing feature to the Data Center, a defined network edge. Since VMware has taken hold in the Data Center, the boundary layer between the virtual machine and the network has devolved to a dumb bridge running in the memory of a ESX server. Installing the Nexus 1000V in your ESX servers creates a virtual switch with interface counters, pvlans, access controls, QOS and many other features that are critical to operating a Data Center. (check out a previous article about the Nexus 1000V).

Enjoying the fruits of our labors

Cloud computing in general, and specifically Virtual Infrastructure 4 have specific benefits that will drive efficiency and agility in IT as a whole. The mechanisms for these benefits will put increasing load on the storage and data networks in your Data Center. It is our responsibility as network architects to take a proactive stance and provision a network with the immediate future in mind. Luckily planning and preparing for these changes in advance have both benefits for our current infrastructure, as well as allowing us to enjoy the fruits of our labors as Cloud Computing changes from a buzz word to a reality.Similar Posts:

• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Cisco releases Nexus 1000V virtual switch for VMware
• VMworld 2009 Schedule
• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco’s Cloud Computing Offering
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

--Colin McNamara

Is your network ready for Cloud Computing with Virtual Infrastructure 4?

The boundary between server team and network team responsibilities has become “fuzzy”

Cisco address’s this issue by putting a switch that can be managed via the same methods common to other network devices inside the ESX cluster. This switch runs the same code that has become standard on Cisco’s Nexus series of Data Center switches – NX-OS.

Prior to adoption of virtualization, when there was a connectivity problem with a host it was quite common for the network team to verify functionality down to the switch port. The server team would do the same. This allowed for each team to focus on areas that met their core competancy. Once we moved from a real switch port, to a dumb bridge inside ESX, lots of finger pointing resulted.

Now, with a Nexus 1000V sitting virtually inside the ESX clusters, the boundary between network and systems teams has been re-estabilished. Now when there is a problem with a host inside an ESX cluster, the network team can use the same day to day troubleshooting tools available to them in other portions of the network to resolve issues faster, and with less finger pointing.

Security controls have been moved further away from the hosts then we would like

A best practice for applying security policy is to apply controls as close to the source as possible. Think of this analogy – Your kids are blasting Radio Disney from their computer. Which of the following do you do?

A. Turn down the speakers at the source

B. Distribute earplugs to all members or the household

Of course, the obvious action is to go to the source, and apply a control (turn down the volume, and tell the kids to clean their rooms). The same principle is valid on the networking side. The best practice is to apply security policies such as VLAN ACL’s and TrustSec policies directly to the switchports that host your switches. Before the Nexus 1000V this was impossible to do in ESX, and forced many environments to move security controls further up into the distribution layer. The side effect of this was that now the security stance from host to host inside ESX clusters was diminished.

The Nexus 1000V brings something called port policies to the table to address this. What these are is pre-configured application security descriptions that are available to you systems administrators to apply in a point and click fashion. Once these policies are applied to the virtualized host, they follow the host where ever it is moved in your virtual cluster.

Provisioning and integrating the networks of VMware ESX clusters with classic networks for most is challenging at best

I wrote an article in march about this specific issue in my post – Challenges integrating VMware into Cisco networks . The core of this issue is that in general that the network integration portions of VMware ESX clusters is not really designed to address server teams , or network teams. In fact, you need to be pretty savy with both portions to successfully integrate VMware clusters into your network. In the real world, you generally find people that are good at one or the other, not both.

By putting a Nexus 1000V in your VMware clusters, you know give the networking teams something they can understand without having to learn Linux, and how it handles bridges (key to understanding ESX networking). With a Cisco switch running virtually inside your clusters, network teams can follow standard core / distribution / access models with the access layer now residing inside the ESX clusters. The network teams can also leverage their existing LAN switching skills for integrating the virtual switches in the clusters with the existing Data Center switching fabrics.

With these roadblocks addressed, Cisco is moving to further the DC 3.0 vision

To realize the DC 3.0 vision, the network inside of VMware clusters had to be under control, and follow the same architectural guidelines that the rest of our network is subject to. With the Nexus 1000V this is now a reality. The next steps withing the DC 3.0 vision to are to extend virtualization and mobility throughout our storage fabrics, and to continue to extend virtualization to the network as a whole, as well as focusing on application virtualization and acceleration to truly realize the vision of cloud computing in the data center.

On the storage virtualization side, Cisco will be using a technology called FlexAttach to enable virtual and physical hosts to change locations in the datacenter without storage team intervention (more on this in a near future post). And on the application virtulization and acceleration side, expect Cisco to continue to enhance it’s existing Application Control Engine (ACE) and Wide Area Application Services (WAAS), and further integrate these into their virtualization offerings.

Want to learn more ?

Introduction to VN-Link network services – Cisco.com

Nexus 1000V overview – Cisco.com

VMware distributed vNetwork switch demo – VMware.com

Challenges integrating VMware into Cisco networks – colinmcnamara.com

Douglas Gourley speaking about how Cisco and VMware will drive Cloud Computing in the Data CenterSimilar Posts:

• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?

--Colin McNamara

Cisco releases Nexus 1000V virtual switch for VMware