ccie, vcp ccvp, rhce, giac, gcih, cisco, netscreen, netscaler, juniper, f5, security, virtualization, vmware

OpenStack Nerd, CCIE, DevOps Junkie

Changing the world, one person at a time

OpenStack Nerd, CCIE, DevOps Junkie header image 2

Simplifying scale out DataCenter design with UCS Manager 2.1

November 1st, 2012 · 27 Comments · CISCO, Cisco UCS, Cisco Unified Computing System, Cisco Unified Computing System Managerm UCSM

I’ve been designing and deploying UCS since the product was released a couple years ago (technically I was involved in the pre-release so we will say since UCSM v 0.8). From the start I was constantly pushing up against scalability and design constraints of UCS. The benefits of the system outweighed the challenge, but these design constraints created some challenges in creating external systems to meet the needs of large UCS customers.

Don’t get me wrong, out of any server platform I prefer UCS. That being said there are a few area’s that have really caused headaches for me over the years.

Headaches solved with the release of UCSM 2.1


Headache #1 – Once I scale past a certain number of servers, I have to establish a new UCS domain

This has been a huge challenge for both large single data center instances, as well as multi data center instances (such as DR). In both these cases I would have to utilize tricks like placing mac address, wwn pools and other “unique” identifiers into a CMDB (Configuration Management Database) outside of UCS. And even when utilizing external CMDBs, there was a still a bit of design necessary to lay out UCS domains in a fashion that would support eventual integration in the future without overlapping configuration elements.

All of this work was done to ensure that if two servers were instantiated in two different UCS domains that they wouldn’t have conflicts if they wound up on the same segment. Handling this logically by bit swapping the UCS domain ID in certain resource pools wasn’t terribly complicated, but in my opinion unnecessary (though integration with CMDB’s can be very complicated).

This got even more complicated If you wanted to have a DR site. Making something simple happen like having a server that boots from SAN boot of the DR site SAN in an outage involved using external tools or scripts. In my opinion this is something that should be handled by UCSM or a manager of UCSM.

Headache solved – UCS Central Manager of Managers


For those in the now, this has been in the works for a VERY long time. In fact the early install (1000+ servers) that I mentioned above where we had to use external CMDB’s to glue UCS domains together in the first year of UCS generated this feature request. 

UCS Central is in a sense a manager of managers. This allows you to aggregate pools and policies of multiple independent UCS domains into one central management platform.

It solves the problems -

  • resource conflicts across pools,
  • mobility of service profiles between UCS domains as well as
  • centralizing access logs
  • centralizing access to console servers

Headache #2 – Even when Cisco released code to manage c-series 19″ rack mounts under UCSM it still required a bunch of extra cables and equipment to make it work.

70% of the worlds x86 servers are in a 19″ rack mount form factor. Recently Cisco enabled them to be managed under UCSM and to have a data path that exits through the fabric interconnects. This allowed a couple key things to happen. First, it allowed a unified view of systems for an administrative staff for a data center. Second, it allowed a clean data path from, say a storage caching engine run on a a b-250 blade, to a compute node housed on a c240 rack mount. All of this communication would be contained within the fabric interconnects, and not have to exit northbound as it had in the past.

I was happy with this release. It allowed the c-series servers to be managed under UCSM with the same tools techniques and API’s that we manage the blades with. However the code was not updated to allow all that magic to happen over a single wire.

You would end up with beautiful cabling on the backs of your blade centers, and a giant mess of cables coming out of your rack mounts since you needed separate cables to support data path vs management plane. Call me a neatnick, but I like my racks to be pretty and clean (and not have to buy extra switches, cables and adapters).

Headache solved – Single wire management for ALL UCS servers


With the 2.1 release now all you need is a single Cisco Virtual Interface Card in your UCS 19″ rack mount (two if you want redundancy) to allow the full feature set that you have available on a UCS blade. For me this not only simplify’s my designs, but also allows flexibility in things like designing Hadoop and OpenStack Swift Object Storage clusters where redundancy is done at the application level and dual 10 gig interfaces are not needed.

Headache #3 – For certain topologies I want storage locally attached to my pods using whatever protocol I want

Here is a dirty little secret. Even though you can abstract a bunch of storage functions into UCS, most server guys are still a bit impatient with their peers on the storage teams. There are many times when the server guys want to consolidate a bunch of boot disks into an array and connect them directly to the fabric interconnects.

Over time Cisco has been releasing support for additional protocols connected in this way, however it was not ubiquitous. This created problems because you could not create a standard topology that supported flexible protocol consumption in your network. You would end up with two to three variants of supported topologies. In my opinion this creates issues with operational procedures and leads to extension of outages and generally inefficient designs.

Headache solved – Flexible and consistent storage topology options no matter what protocol is being used.

ucs fcoe multi-hop cisco

With UCS 2.1 now, no matter what protocol floats your boat you can implement them in a consistent manner. This may include directly connecting Fibre Channel storage to your fabric interconnects and zoning them. Or it may include utilizing multi-hop FCoE (I’ll leave the argument to whether you SHOULD use this till later).

Either way, the most important thing to me is that no matter what the design requirements are. Now you have the tools available to meet them in a consistent fashion without changing your entire network and systems topology.

Colin’s Thoughts

Quite often there is lots of glitz and glamor when a new product is released. Press conferences are held where everybody looks at the shiny blinky things and oohs and aweess. However when new software makes things you already use every day work better, or allows them to do new things comes out nobody notices.

In this case the 2.1 release of UCSM takes a product that many people already have (Unified Computing System) and makes it do more. There aren’t going to be press conferences about this, but it is worth taking a closer look at. It will make my life easier, and I hope it does the same for you.




Tags: ···············

27 responses so far ↓

  • 1 Cisco Announces Updates to UCS and Nexus Lines // Nov 1, 2012 at 6:02 pm

    [...] us to do Fiber Channel zoning directly on the Fabric Interconnects as well as multi-hop FCoE.  Here’s a post by Colin McNamara that talks about both UCS Central and UCSM [...]

  • 2 Will Hgan // Nov 2, 2012 at 5:03 am

    Great post on UCS.

    Somehow this reminds me of when Cisco came out with that Call Manager manager of managers (I cant remember the product name).

    Really hope you follow up on FCOE comment:
    “I’ll leave the argument to whether you SHOULD use this till later”

  • 3 Cisco UCS Central :Scaling Is a Necessity. Adding Cost and Complexity Is Not. - WKSB Solutions // Nov 6, 2012 at 8:37 am

    [...] McNamara about the 3 major “headaches” solved by the introduction of UCS Central and UCSM 2.1 [Simplifying Scale Out DataCenter Design with UCSM 2.1]“The 2.1 release of UCSM takes a product that many people already have (Unified Computing System) [...]

  • 4 party planning ideas // Jun 16, 2013 at 9:38 am

    Nice post. I was checking constantly this blog and I am impressed!
    Very useful info particularly the ultimate section
    :) I maintain such info a lot. I used to be looking for this certain
    info for a very long time. Thank you and good luck.

    Look at my web page; party planning ideas

  • 5 Lindsay // Jun 17, 2013 at 7:25 am

    Thought you might want to learn about our new single fiber X2 transceiver. Might be of interest to your readers.
    Let me know what you think:

  • 6 elephant birthday party Supplies // Jun 19, 2013 at 2:37 pm

    I have been browsing on-line greater than 3 hours nowadays,
    but I never found any attention-grabbing article like yours.
    It is beautiful value enough for me. In my opinion,
    if all site owners and bloggers made just right content material as you did, the
    web will likely be a lot more useful than ever before.

    My blog post: elephant birthday party Supplies

  • 7 land financing // Jun 23, 2013 at 1:31 pm

    I have been browsing online more than 4 hours today, yet I never found any interesting article like yours.
    It’s pretty worth enough for me. In my opinion, if all site owners and bloggers made good content as you did, the internet will be much more useful than ever before.

    Here is my blog – land financing

  • 8 personalized bridal shower // Jun 26, 2013 at 2:53 pm

    I think the admin of this website is really working hard for his site, since
    here every information is quality based information.

  • 9 // Aug 4, 2013 at 3:33 am

    It’s awesome to pay a visit this web page and reading the views of all mates about this piece of writing, while I am also keen of getting experience.

  • 10 commander viagra acheter // Mar 28, 2014 at 5:07 pm

    I am sure this article has touched all the internet visitors, its really really nice paragraph on building up new web

  • 11 Learn Additional // Apr 2, 2014 at 8:04 pm

    Sounds simple but we aren’t generally doing it in the United States.
    Me: How do you feel about petfinder like a service to help dogs find forever homes.

    Page numbers ought to get incorporated inside upper left half of the header.
    , it could possibly confuse an individual, once you actually mean a similar
    link Learn Additional You will likely be notified as you might be typing the ‘allow links’ in your article.

    It can be less stressful to write by hand in case you have poor typing skills or you never actually enjoy using a

  • 12 // Apr 10, 2014 at 11:05 am

    It is really a great approach to get lots of exposure and post ads several times in various ways.
    We feature latest and detailed info on Barrington homes for sale that can help you to make your decision to get a home easier.

  • 13 games in mobile // Apr 14, 2014 at 3:37 am

    Hey There. I found your blog using msn. That is a very neatly written article.
    I will be sure to bookmark it and come back to read more of your
    helpful information. Thank you for the post.
    I’ll certainly comeback.

  • 14 Mary W. Herrington // Apr 29, 2014 at 5:44 am

    Genuinely motivating points you’ll have mentioned, thank you so much for writing.

  • 15 Michael K. Erdman // Apr 29, 2014 at 5:44 am

    I just need to reveal to you in which I am certain inexperienced to having a blog and utterly admired your work. Very possible I am going to bookmark your article post . You really have wonderful article text. Like it for giving out with us your web post.

  • 16 Margaret C. Crowe // Apr 29, 2014 at 5:44 am

    Very beneficial points that you have remarked, thank you for putting up.

  • 17 Johnd505 // Apr 30, 2014 at 8:03 pm

    The data are part of an autumn of just about 16 targets in Lib Dem provide because of the 2010 commander selection, before Huhne scooped 46.5 with all the electionHermes kelly felix bags didn can far in order to move benefit ddfaedfkaged

  • 18 Timothy S. Lester // May 8, 2014 at 5:41 pm

    Was looking throughout . I appreciate your concepts. Do write up much more of this theme ok. Thank you

  • 19 Michael T. Anderson // May 8, 2014 at 10:26 pm

    I observe there are several junk mail on this blog site. Are you thinking of intending to clean it up. Perhaps I can assist to complete?

  • 20 Kevin R. Harvey // May 8, 2014 at 10:44 pm

    We merely have to tell you that I am certain beginner to putting up a blog and really valued your webpage. Likely I am most likely to store your blog post . You literally have fabulous article content. Acknowledge it for share-out with us your internet site write-up.

  • 21 clumsy ninja hack.rar // May 26, 2014 at 6:51 am

    Aw, this was an exceptionally good post. Finding the time and actual
    effort to generate a good article… but what
    can I say… I hesitate a lot and don’t manage to get anything done.

    My blog :: clumsy ninja hack.rar

  • 22 7 zip free download // Jul 14, 2014 at 5:50 pm

    Incredible points. Outstanding arguments. Keep up the amazing spirit.

    Feel free to surf to my web-site – 7 zip free download

  • 23 mpomponieres vaptisis // Jul 20, 2014 at 4:21 am

    mpomponieres vaptisis I’ve very little expertise in computer programming however I had been hoping to start my own blog in the near future. Anyway, if you have any ideas or techniques for new blog owners please share. I know this is off subject but…

    I’ve very little expertise in computer programming however I had been hoping to start my own blog in the near future. Anyway, if you have any ideas or techniques for new blog owners please share. I know this is off subject but I just had to ask. Thank…

  • 24 norton internet security 2014 serial key // Aug 2, 2014 at 8:06 am

    Spot on with this write-up, I absolutely feel this website needs far more attention. I’ll probably be returning to
    read more, thanks for the info!

    Look into my webpage; norton internet security 2014 serial key

  • 25 Jc // Aug 3, 2014 at 11:32 am

    It is a journey that must be made for personal happiness but I have
    never met a transgender who hadn’t suffered greatly from the journey.

    The individual can first of all think about shaving.
    In 2010, Abi had her FFS in Marbella, Spain, which has
    significantly improved her ability to pass in public and therefore her self-esteem.

  • 26 leadpages system Review // Aug 14, 2014 at 6:11 pm

    My spouse and I absolutely love your blog and find a lot of your post’s to be
    exactly what I’m looking for. Does one offer guest writers to
    write content for yourself? I wouldn’t mind producing a post
    or elaborating on most of the subjects you write about here.
    Again, awesome site!

  • 27 medzilla Job Search // Aug 14, 2014 at 8:48 pm

    I know this if off topic but I’m looking into starting
    my own blog and was wondering what all is needed to get set up?
    I’m assuming having a blog like yours would cost a pretty penny?

    I’m not very internet savvy so I’m not 100%
    sure. Any tips or advice would be greatly appreciated.

    Appreciate it

    my weblog: medzilla Job Search

Leave a Comment