You may notice that I am using a tool called Netformx. You may ask – since this is a Cisco product, why aren’t we using the Dynamic Configuration Tool or the MultiLine Configurator? The answer that was given to me was that UCS requires a bottom up configuration. It requires that you start with the type and number of systems, and then work your way up to the amount of Chassis, FEX’s, links and Fabric Interconnects that are required. All of the workflows in Cisco’s existing tools require a top down approach. Until the new configuration workspace is released from Cisco, Netformx is your only option to quote any UCS product.

.

Thank you for taking the time to watch this walk through. If you found this helpful, please feel free to leave a comment or ping me on twitter.Similar Posts:

• Twitter Updates for 2008-06-05
• Cisco Nexus 5020 and 5010 FCOE video ordering guide
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Cisco Nexus 4000 Blade Switch

--Colin McNamara

Cisco Unified Computing System Quoting and Configuration with Netformx

“it looks like one of the key features not on the list of components for the California boxes is going to be a red discount pen”

Timothy references sources who have obtained a price list and shared it with “El Reg” . I wish Timothy would have contacted an actual Cisco Unified Computing System Advanced Technology Partner, because any partner that is involved in the launch could have explained to him the concepts of List price (List), Manufacturers Suggested Retail Price (MSRP), and Purchase or Buy price.

In this article I want to dispel the myths of server and network manufacturer pricing, demonstrate the true cost of building a data center with blade systems, and at the end provide a cost comparison between legacy server vendors options and Cisco’s Unified Compute System.

First, lets go over some the basic concepts of vendor pricing. At the end of this you should understand the difference between list price, manufacturers suggested retail price, and purchase price.

List Price

List price is a high level number that Cisco publishes weekly in its global price list. The purpose of this list price is to provide a uniform price list across all product sets that Cisco offers. The most important thing about list price is NOBODY EVER PAYS LIST PRICE. Let me repeat that again NOBODY EVER PAYS LIST PRICE. Are we clear? This is similar to list price on a car on the car lot. All list price provides is a starting point where a Cisco partner and a customer can negotiate a common discount and end up with something close to (generally at or below depending on technology type and yearly spend) MSRP.

Manufacturers Suggested Retail Price (MSRP)

This concept is something that anyone who has purchased a car before is familiar with. The number that is on the window of the car when you look on the lot is list price. The first number the dealer brings up lower then sticker is MSRP. Depending on the popular of the product, the competition in that particular space, and the negotiating power of the customer you will either pay that price, or some percentage below. For example if you are buying one new car you may have the negotiating power to get the price to drop 5% off of list. If you are buying 200 new cars (say a fleet) you have significantly higher negotiating power, and you may be able to drop the price by 15% of of list price.

In Networking Sales MSRP is significantly less then list price. A good exercise to see what this number is, is to find a device, say a WS-C3560E-12SD-E (3560 with 12 Gig SFP ports and 2 10 Gig ports) in the Global Price List. You have access to this at any partner level at www.cisco.com/dprg . (my point here is that this is no big secret). As of Friday June 12 2009 the LIST price for this product is $19,995.

Now take that same part number - WS-C3560E-12SD-E and pop it into your google search window. Within the top four links I found this product for $12,434.15 . This price is for  pure fulfillment, with no value added consulting or design work from you local Cisco partner.

If you do the quick math, this price difference is equal to 38% off of list price. Come to your own conclusions, but it would be safe to say that this could be considered MSRP for Cisco products.

Purchase / Buy Price

Buy price is just that, the price at which the customer purchases (buys) the product. This is can be at MSRP, or if the customer is buying significant amounts of hardware at a time, or if there is a “special” (programs and incentives) going on the number could be slightly lower then MSRP.

Percent off of list differences between legacy server vendors and networking vendors

This is where the biggest confusion is coming from. Legacy server manufacturers  have set their list prices much closer to MSRP then networking vendors (remember, MSRP is the price where most customers purchase at).

Why is this? In the networking space, vendors have historically created their own processors, ASICS and boards. This means that the sales discussions are feature to feature. It also meant that you had to have a conversation with the networking vendor or networking partner to properly size your network devices and get a quote – which is around MSRP, not List price.

In the legacy server space, especially the majority of the x86 server space, the market has been essentially commoditized. E.G. – You can buy an intel based server with X amount of memory and hard drives that will perform roughly equally from any of the main manufacturers. That made it much easier for a sever admin to just pull a price off of the web and compare. So what the server vendors ended up doing is setting their list price  only slightly above MSRP.

What this translates to is the list price, between legacy compute vendors and Cisco will be drastically unequal. What is equal is MSRP, or the generally accepted purchase price by common customers.

Why did Cisco set the list price of UCS higher then the legacy server manufacturers?

For the vast majority of its sales, Cisco relies on what is called the channel model. This means that Cisco partners with local Value Added Resellers (VAR’s) who sell Cisco’s products and then provide consultative services to design and implement them in customer networks. Most customers who purchase any regular amount of Cisco product either have a general expectation that they will buy Cisco product at a certain percentage discount off of list and sometimes the partner and customer have entered into purchasing contracts which require that all Cisco product is provided at a specific discount off of list price.

If Cisco decided to set the List price at a small percentage lift over MSRP, this would cause a problem for the entire channel. This would be especially hard for any customer who had a contract to buy product at a specific discount. What would happen is contracts would have to be renegotiated, which generally takes months and is about as fun as pulling teeth.

The second reason for setting list price the for compute the same as list for network is quoting. Right now, if you buy hundreds of different Cisco devices through a reseller it is very likely that the discount is going to be the same across all products. This makes the mechanics of sales much simpler, because you don’t have a lot of math in the quote (this can cause errors). On the customer side, having one set discount makes it much easier to compare quotes and to ensure that they are getting the best deal possible. In short, sticking with Cisco’s current list pricing structure benefits both the customer and the partner.

Now that we have set the record straight on list price, MSRP, and Buy price, lets take a deeper dive into what components make up a blade system powered data center. And then we will compare the price structures of both.

Components of all Blade Systems

Blade Server – The compute blade where commodity silicon elements such as the CPU and RAM are housed. As of writing this article, the latest high performance blades from all major server manufactures support two xeon 5500 processors (Nehalem) and DDR3 memory.

Mezzanine cards – These cards take the place of PCI-e cards in a rack form factor server. In a blade system these provide data network and storage network connectivity. They attach to the blade itself via proprietary connectors that implement either PCI-e 8 or 16 lane connectivity at the time of writing. In some cases other functions such as IO accelerators can also be attached in the mezzanine card form factor.

Blade Enclosure – This is functionally a tin can where eight to sixteen blades are placed. It also is used to provide a centralized power distribution fabric, as wells as slots for interconnections of data and storage network devices.

Data Network Modules – These are effectively ethernet switches that have been miniaturized to fit into the tight confines of a blade enclosure. Classically they have provided 1 gig connectivity to the servers, and 10 gig to the distribution layer, however with Nehalem processors and VMware there is a move towards presenting 10 gig connections to the server, and multiple 10 Gig connections into the distribution layer.

Storage Network Modules – The local disk in a blade server is classically anemic. To provide higher IOPS (input outputs per second) to disk, Fibre Channel connectivity is extended by taking SAN fabric switches and miniaturizing them to fit into the blade enclosure.

Data Network Distribution – If you have multiple blade enclosures there is a need to connect them together at a reasonably high bandwidth. To serve that need a variety of 10 Gig distribution switches are provided from all server manufactures at varying cost and performance levels.

Storage Network Distribution – Along the same lines of the data network distribution, SAN fabric switches have to aggregate up to a SAN distribution layer, or if the installation is reasonably large a “director” class SAN switch. This allows all the blade enclosures to see the same storage network, as well as providing for deterministic storage network performance as you scale out.

Management Infrastructure – All manufactures have a need to manage and monitor all of the devices that comprise their blade system. Many manufactures have multiple management modules per blade enclosure.

Comparison of Costs – Cisco vs Legacy Server Manufacturers

The funny thing, is that many people have assumed that Cisco’s Unified Computing System will be priced higher then legacy server manufactures products. In my mind this is because they associate higher quality with higher price (basically the Mercedes vs Kia discussion). Here is something that will shock you - it costs less to buy an entire blade system through Cisco then to buy from the legacy server manufacturers.

When people hear this, they are puzzled. How can two server manufacturers, who buy their CPU’s from the same company (Intel) and their memory from the same fabs end up with different prices? The answer is elegance in engineering. Lets go through each of the elements of a blade system infrastructure and find out where the costs are. More importantly lets look at where Cisco has innovated to provide higher performance at a lower cost.

Dollars and Cents – How much is the cost difference

I worked up two quotes recently. These quotes included all elements required to build an end to end blade system using both legacy server manufactures devices, and using Cisco’s Unified Computing System. I have broken out two scenarios.

8 blade servers - Cisco wins with a savings of 11%

In this scenario the cost of servers and enclosures were fairly equal. The cost savings started racking up as storage and data networking devices were included, as well as base management software was taken into consideration.

320 blade servers - Cisco wins with a savings of 31%

With 32o blade servers the same cost savings seen in the 8 server scenario were amplified. Economies of scale translated into significantly less devices being required to support the individual compute blades. This resulted in 31% savings compared to the legacy server manufacturers.

Summing it up

Cisco has entered into a highly competitive server market by taking an elegant approach to its blade systems. This approach lowers the purchase price of the UCS through reducing the amount of components compared to legacy server manufacturers. I know that there is a lot of misinformation flying around, and I hope this helps to set the record straight on the pricing of Cisco’s Unified Computing System.Similar Posts:

• Cisco Nexus 4000 Blade Switch
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Cisco introduces the C-Series Rack Servers
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Cisco’s Cloud Computing Offering

--Colin McNamara

Confusion about Cisco UCS pricing – Setting the Record Straight

This March, Cisco formally announced its entry into the enterprise computing market with a new product line called the Cisco Unified Computing System (UCS). I say formally announced, becase the existence of the Unified Computing System might just be the worst kept secret in history. In the months prior, to launch Cisco has been openly talking about the system, as well as many news agencies. The only people who haven’t been talking about the system were those of us who got briefed in early and were under strict NDA. (yours truly being one of them). But now, the multiple layers of NDA have been removed and I am free to talk. So, let me tell you about a new concept called Unified Computing.

The Unified Computing System answers a simple question – “what would you do if you could build a system with no preconceptions”. That same question has been asked over the years by Cisco. The results have given us the Catalyst 6500 line of switches, the Cisco MDS storage line, as well as the Nexus 7000/5000/2000/1000V family of switches.

Nuova – a history of innovation

A couple of years ago, Cisco funded a startup called Nuova (meaning “New” in Italian). The founders of this startup were the same innovators who led the Catalyst, MDS and Nexus products. Not only did they have a track record of successful products, but they also had experience in both types of Data Center networking – Ethernet, and Fibre Channel. This startup took the lessons they learned creating a non blocking, low latency, highly available fabric for the MDS line of SAN switches and created a new line of Data Center switches, the Nexus 5000 that is able to transport Fibre Channel and Ethernet traffic at the same time, through a single adapter.

In April of 2008 Cisco formally acquired Nuova (which in effect was just hiring prior innovators back). Nuova was renamed the Server and Virtualization Business Unit (SAVBU), and the Nexus 5000 was released to market, making Cisco the first vendor to deliver a solution based around the upcoming Fibre Channel over Ethernet standard (FCOE). This pattern of innovation continued, as SAVBU released a virtual switch for VMware (the nexus 1000V) and this January released a remote line card technology called Fabric Extension. Those of us on the inside got the hint that the choice of names for this new business unit (SERVER and VIRTUALIZATION Business Unit) was a foreshadow of things to come.

Unified Computing System enclosure w/ redundant 6120 Fabric Interconnects

Here are some spec’s to get you started –

• Single point of management for all devices in the fabric.
• Virtual machine enabled networks adapters. (VNtag capable)
• Up to 320 B-series compute blades in one fabric.
• Up to 384 Gigabytes of memory per blade (full width blade)
• Server Profiles – virtualize server identities (UUID, WWN, MAC)
• Hardware Assisted Virtualization using Intel’s next generation Xeon (code named Nehalem-EP) processors
• Redundant 10 Gigabit connections between servers.
• Fibre Channel SAN access available to every blade.
• Capital costs up to 20% less
• Operational costs up to 30% less

The UCS does all of this while using 1/3 less components then the competition. What does using less components give you? Less components means less things to buy (lower capital expense). It also means less things to power and cool (lower operational expens). And finally it means less items to manage (lower management burden) How does Cisco do all of this while using drastically fewer components? I think it is necessary to talk about the major components that they system is built from to answer that question.

Cisco UCS 6100 Fabric Interconnect (Nexus 5000 on steroids)

.

The primary building block of the system is the Cisco UCS 6100 Fabric Interconnect. Cisco took the non blocking, low latency, lossless fabric from the MDS that was used on the Nexus 5000 and used it as a building block for the 6120 and 6140 Fabric Interconnects. These fabrics support 20 and 40 (6120/6140)10gig Data Center Ethernet (combined Fibre Channel and Ethernet support).

• 6120 – 20 Fixed 10 Gig Data Center Ethernet ports along with an expansion module that supports native Fibre Channel, or additional Data Center Ethernet interfaces
• 6120 – 40 Fixed 10 Gig Data Center Ethernet ports along with two expansion module that supports native Fibre Channel, or additional Data Center Ethernet interfaces

The 6100 series Fabric Interconnect unifies Storage and Ethernet network, as well as providing supervisory functions for its remote line cards, the 2100 series fabric extenders that are inserted the compute chassis. The other thing the 6100 Fabric Interconnect does is house the Unified Computing System Manager (UCSM).

Cisco Unified Computing System Manager (UCSM)

The UCSM runs on the fabric switches, providing a single point of management for all components in the fabric -

• I/O Fabric
• Chassis and Services
• Adapters and Virtual I/O

This interface can be accessed through either a web based Gui or CLI. It also supports a full API for programatic integration and management of the system. The biggest thing that they SAM gives you is the ability to dynamically provision server attributes down to the compute blades. Attributes that can be pushed down dynamically include CPU UUID, SAN PWWN, Ethernet MAC address, and many more. These items are pushed down through as Service Profiles.One key component of the service profile is the Port Profile.

.

These Port Profiles are dynamically created in the SAM, and most importantly enable you to create virtual network interfaces (vNics) that show up to your server administrators as normal network interfaces. This allows your server administrators to follow their application vendors recommended interconnection topologies.

A great example would be VMware’s recommended topology for ESX. In this topology there are four network intefaces defined. Each for a specific function. This logical topology can be implemented, with all the relevant speed, QOS, VLAN, and security attributes all “pre-configured” for the virtulization administrators, simplifying their virtualization cluster deployments.

Cisco UCS 2100 Fabric Extender (Next Generation FEX)

 The next component in this architecture is the UCS 2100 Series Fabric Extender (FEX). There are two of these in each bladechassis. Each FEX has the capability of up-linking to 6100 Fabric Interconnects with four 10 Gig ports. for a total of 80 Gigabit persecond out of each blade chassis (supporting 8 half width blades, or 4 full width blades). Each FEX is managed as a “remote line card” connected off of the Fabric Interconnects.

Logically, think of your 6500 series switch in your Data Center right now. You have three logical functions, supervisory functions (sup module), a bus for the switching fabric (traces in the chassis) and line cards (6748 for example). You will notice that you only manage the sup module itself. You don’t shell into to each line card to set up backplane interconnects, or to update microcode. You update software and configurations on the sup module, and the intelligence that Cisco builds into its software manages this for you.

The 6100 and the 2100 interact in the exact same way. In this case the 6100′s are the sup modules, the 2100′s are the line cards, and we are running 10 Gig connections to build the switching fabric. What this gives you is a simplified network architecture, which takes elements that would in the past be individually managed, maintained, upgraded, etc and consolidates that into one highly available, high bandwidth consolidated SAN and Ethernet for your Data Center compute needs.

Putting Network Intelligence on the Compute Blade

Cisco will be giving customers three mezzanine card network adapters options for the UCS.

The first card (based is based on a the Palo chipset. This chipset was developed internally at Cisco and performs Network, Storage and Virtual Machine networking functions all on one 10 Gig capable chip. This chipset effectively extends network intelligence into the blade itself allowing for some pretty interesting integrations with Virtual Machine Hypevervisors (VMware, Hyper-V, Etc).

The second option is based on the Menlo chipset in conjunction with either a Qlogic or Emulex Fibre Channel adapter chipset that gives you 10 Gig network access, while retaining strict compatability with applications that require either of these classic HBA’s.

The third option is an “economy” option based on an Intel chipset. This will give the compute blade 10 Gig access.

What will drive your choice of network adapter? I think the biggest driver will be a technology called “VNtag”. This technology is currently in use on the FEX to encapsulate traffic that enters the fabric with a little shim header that communicates the identity of the incoming port (VLAN, QOS, Security info, etc). This shim header passes from the FEX up to the 6100 where the shim header is removed and the frame is processed.

This alone is very cool, however Cisco has taken things one step further and put the ability to impose VNtag’s in the Palo chipset (on the blade) itself. What this gives you the ability to logically attach a virtual machine DIRECTLY to the network. We no longer need a vSwitch, or even a 1000V to give full network functionality to a virtual machine. Effectively, Cisco is giving the network adapter inside of the blade many of the functions of a network switchport. Logically what this does is reduce your network tiers from four tiers in a competitive system, to two tiers in a Unified Computing System.

Cisco UCS B-Series Blades

The compute blades themselves are available in either 1/2 width of full width form factors. As stated in the formal announcement the maximum memory for a full width blade will be 384 Gigabytes per blade. Why is this important?

At release, there will be two option for compute blades.

Half width blade

• 2 quad core Intel Xeon 5500 processors
• 96 Gigabytes of memory
• Two small form factor SAS drives (raid 0 and 1)
• Single Converged Network Adapter slot (connected to redundant fabrics)

Full width blade

• 2 quad core Intel Xeon 5500 processors
• 384 Gigabytes of memory
• Two small form factor SAS drives (raid 0 and 1)
• Dual Converged Network Adapter slots

As CPU’s become more powerful, the host compression ratio’s (the amount of virtual machines you run on one physical blade) increases. The one thing is, you can only over commit your memory to a point. Once you are at that point you have to physically add more memory. Your only option with legacy server architectures is to add another physical server, pay for additional licenses and add more memory to it. Then add that server to the cluster and allow your virtual machines to utilize its additional resources in the pool.

The problem with this is a simple one, Cost. The problem was that the virtual machines needed more memory, but since there was a limitation on how much memory the legacy server could handle, it forces costs skyward to meet those memory needs. In this scenario if the customer had been using a full width Unified Computing blade the customer could have avoided purchasing an additional server and avoided the additional licensing and management cost associated with that additional server.

What does this balance out to in real costs? By allowing for higher host compression ratios there is the capability to avoid a significant amount of cost (50%+) in your compute layer. 

Intel Xeon 5500 ( Code Named Nehalem-EP)

Intel’s next generation Xeon CPU (code named Nehalem) will be driving the Compute Blades. The Xeon 5500 signals Intel’s move away from the legacy “Front Side Bus” (FSB) architecture into what is called “Intel Quick Path Interconnect” (QPI). Intel’s Xeon 5500 with Quick Path Interconnect changes a couple key things from previous generation Xeon procesors.

• high bandwidth, full mesh, routed interconnect between CPU’s instead of a low bandwidth bus
• DDR3 vs DDR2 memory and moves the memory bank adjacent to the CPU’s for higher performance
• I/O Hubs are now dedicated for network and storage interconnects

First and Second Gen Xeon vs Xeon 5500 (Nehalem)

.\

If you look at the picture above, where the last generation processors are to the left, and the processor that will be in the UCS is on the right, you will notice a couple key items. I think the most important items to point out is that the bandwidth in pretty much all directions is superior in this architecture. If you think about this as a network on your server itself, you can see how moving from a bus based network, to a full mesh routed network has significant performance advantages. Stay tuned for later posts where I will talk about some of these advantages in detail.

Hardware Assisted Virtualization

One of most important new features that Intel is bringing to the table is the notion of hardware assisted virtualization. Intel has created a couple key technologies to address the following problems faced in virtual environments.

Processor Virtualization

Currently to virtualize an operating system, we rely on the hypervisor (ESX, for example) to accomplish two key tasks – Ring Depriveleging and Context Switching. When we are talking about CPU overhead in virtual environments this is what we are talking about.

Memory Virtualization

Again, in current environments the hypervisor is used to abstract memory. ESX uses a technique called page table shadowing to virtualize the physical memory. This again however adds inefficiency to the process and shows up as virtualization overhead.

I/O Device Virtualization

As our virtual machines go to disk, we are again presented with a situation where the hypervisor has to abstract disks and networks presented tot he physical server, and connect these to the relevant virtual machines. Again, this shows up as virtualization overhead, lowering the efficiency of our virtual servers as well as introducing I/O sprawl.

Solving virtualization overhead problems

With the Xeon 5500 on Cisco’s Unified Compute System blade you will be able to use processor features to solve the problems listed above. VT-X is used to extend CPU virtualization down to the physical CPU, Extended Page Tables, Cisco Memory Expansion and DMA remapping is used to speed remove memory access overhead from virtual network and I/O interfaces, and I/O devices sharing through VT-C is used to integrate the physical network with the virtual network transparently and with much less overhead.

Summary

Cisco’s entry into the computing space is not a “me too” entry into a commodity x86 market. It is a well thought out strategic move unifying storage, network and compute functions in a unique way that will differentiate the Unified Computing System from other compute offerings. I expect some fierce debate of the upcoming months as competitors release products to compete. I feel that Cisco’s value proposition of a unified compute / network layer in the Data Center uniquely solves problems that most customers face. And at the end of the day, the vendor that can solve the customers problems is the vendor that will succede.
Similar Posts:

• None Found

--Colin McNamara

Cisco’s Unified Computing System – It’s not just a blade center