What is a Fabric Extender?

If you think of a large switching chassis such as a Catalyst 6500 or Nexus 7000 you have the notion of a Supervisory module, a switching fabric forming the backplane, and line cards connected on the back into the fabric, and connecting to servers and other networking devices in the front. In the case of these switching chassis all of the items we just talked about are connected on the inside of the same physical box. Now suppose we took each of those components, the Sup, Fabric, and Line Card and installed them in different parts of your data center. The Sup module in this case would be a Nexus 5020 or Nexus 5010. The fabric would be four 10 Gig port bundles from the Nexus to the Fabric Extender, and the line cards themselves would be replaced by the Nexus 2000 Fabric Extender.

Why use a Nexus Fabric Extension in your Data Center designs?

1. Simplify network management – By moving from individualy managed standalone edge switches to an extended fabric you minimize the amount of devices in your datacenter that you have to manage. Say you have 10 racks full of pizza box servers, (That makes 42 servers per rack) you may throw 3750′s or 4948′s top of rack to provide connectivity. In this case your access layer would consist of at minimum 10 unique devices to manage, upgrade, etc. in your server access layer. If instead you put a Fabric Extender at the top of each rack, you still get 48 ports of coppert gigabit ethernet, but logically you would only have 1 device to manage.

2. Simplify your network topology – In the scenarios mentioned above with classic switching, you would have to manage a spanning tree topology accross your entire access layer. Half of your uplinks will be blocked by spanning tree so your network does not loop. This is a perfectly valid topology, but it does leave you open to the pains of spanning tree in your data center Now replace these switches with a Nexus 2000 Fabric Extender. Instead of spanning tree to integrating individual switches, you now how a very simple U topology forming your edge.

3. Lower your costs for Data Center class switching – Before the Nexus 2000 if you wanted to properly deploy a Data Center access layer, you either had to do buy 6748 blade on a 6500, a 48 port gig blade on a Nexus 7k (M148GT) , ora 3750 or 4948 switch. These all are not the cheapest switches and blades, but they provide each gig switch port with the proper buffers and throughput that are needed for true Data Center applications. Now if we chose to use the Nexus 2000 Fabric Extender in conjunction with the Nexus 5000 we can provide Data Center class 1 Gig Ethernet services at roughly half the cost of using the older switches.

4. Integrate 10 gig and 1 Gig services in a clean way – I have seen many many clients try to cleanly integrate 10 Gig services (normally from blade centers w/ ESX clusters) into an existing 1 gig data center edge. Normally what happens is that the “best” integration case is to put some form of oversubscribed 10 Gig card into their existing distribution layer, or in a worst case some customers are only able to provide 10 Gig of a collapsed core. This is obviously not an ideal setup. By integrating the Nexus 5000 40 or 20 port 10 Gig switches with the Nexus 2000 Fabric extender it provides us with a very clean way to integrate both 10 Gig and 1 Gig services in our Data Center access layer.

Why am I excited about this product?

I am lucky enough to work with some forwarding thinking customers with very large Data Center requirements. We have been working with Cisco over the past couple months to incorporate this end to end Nexus infrastructure into multiple customer designs. Before the Nexus  5000 / 2000 combination, designs to provide large layer 2 adjacent mixed 10 and 1 Gig services were well…. not as elegant as we would have liked to see. Now with Fabric Extension, we  are able to answer this mixed 10 / 1 Gig Data Center access layer design scenario in an elegant and scalable way.

Want to learn more?

Nexus 2148T Product Page - Cisco.com
Similar Posts:

• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco Nexus 5010 released
• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Cisco introduces the C-Series Rack Servers

--Colin McNamara

Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)

NX-OS / DCNM XML API Fundamental Technology

Netconf – platform for secure communication of XML data between DCNM / NX-OS and mgmt station.

XML, using clear text tags, creates a self describing API. This is necessary because of the confusion and unneccessary complexity inherent in the current SNMP MIB based structure commonly in place today.

Webservices API = SOAP/XML over HTTPS.

Netconf/XML on NX-OS capabilities.

• Must be run over SSH
• Maximum of eight concurrent NetConf Sessions per nexus virtual device context.
• Every single CLI capabilities is assigned a XML tag

JMS is supported in DCNM as wel as NETCONF

• Native integration with IBM Message Queue

Similar Posts:

• Arista Networks – Their approach to cloud networking
• Challenges integrating VMware into Cisco networks
• Interesting TechWise TV episode on virtualization
• Identity aware networking using Cisco TrustSec
• Why was Storage Networking my first CCIE? And What did I do to prepare?
• Cool new features in 12.4(15)T

--Colin McNamara

Cisco Nexus NX-OS and DCNM XML API’s – Cisco Live 2008

1. Layer 2 Tunneling Protcol (V3) static and hairpin configuration example - my buddy Rick was nerding it out in the lab and sent a great configuration doc for L2TPv3 my way. L2TP(V3) is used to create a layer 2 psuedowire across layer 3 routed links. This is a great service provider tool that you can use in your own network, no MPLS needed .

2. SNIA Education – Fiber Channel Over Ethernet – There is a lot of buzz going around right now about Fiber Channel Over Ethernet (FCOE). There is also a lot of misunderstanding about the fundamentals of this architecture. This Storage Networking Industry Association (SNIA) does an outstanding job of covering FCOE at both at an architectural level, as well as going over low level messaging structures.

3. Trill (Rbridge) architecture – IETF internet draft – I think the last time I was this interested in an internet draft was when iSCSI was first being proposed in the IP Storage working group. Trill, in my opinion is basically a light weight version of MPLS / VPLS. It has as far as I can tell most of the advantages of this architecture, without some of the configuration and hardware requirement drawbacks. Fair warning, reading this document started a doc hunt that killed my Saturday.

4. Cisco’s Security Response to Sebastian Muniz’s IOS rootkit – Security is a very important aspect of network design. Sebastian’s IOS rootkit demonstration is going to force some customers who in the past have been “OK” with having older, possibly vulnerable IOS versions floating around to update their operational practices and start keeping their routers and switches operating systems as often as they do their servers. Thankfully, Cisco has been embracing technologies such as kernel virtual machines, in service software upgrades and more to lesson or remove the impacts of software upgrades.

5. Turning Wounded Warriors into Network Ninja’s – As a former Marine (well, always a Marine, formerly employed by the USMC) this program goes straight to the heart. Cisco is partnering with Naval Medical Center San Diego (NMCSD, or Balboa Naval Hospital for us locals) to provide technical training to Marines and Sailors who have recieved service ending wounds in Afghanastan and Iraq.Similar Posts:

• Fibre Channel over Ethernet is taking off
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Zone based IOS firewalls
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers
• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0

--Colin McNamara

Link Round Up – L2TPv3 FCOE Trill Wounded Warriors

For new readers to this site, please you might enjoy the following articles -

Who is Colin McNamara ?

Usability features on the Nexus 7000

Overview of NX-OS 4.0 (the operating system for the Nexus 7000)

Why did I choose to get a CCIE in storage networking?

What does it take to pass the CCIE exam?Similar Posts:

• And it begins again – On the road to my CCIE in Storage
• What does it take to pass the CCIE exam?
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Cisco Nexus 4000 Blade Switch
• created a new map of ashley’s site
• Certguards inflamitory post about Ethan Banks removed from Network World Magazine

--Colin McNamara

Featured on Network World – 20 useful sites for Cisco networking professionals

What useability enhancements do you feel are the most beneficial?

1. A separate, IP enabled, Management Interface. This has been a long time coming. The out of band management interface is very similar to a Ilo card in the HP world. it is effectively a supercharged console server that happens to site on the backplane of the sup engine. I am sure whoever pushed this feature through is going to get flowers one day from a Tech who DIDN’T lock himself out because the management interface was effectively a separate system.
2. Finally, a functionally USB Interface that I can transfer IOS (well, now NX-OS) images through. Everyone has a USB key nowadays, even my Grandmother has one, it will make life so much easier when I can have a 4 gig key with me that has most IOS / NX-OS  versions and my common configs and just pop them right in.
3. The integrated Cabling system is CLEAN. I love that it forces you to reserve the appropriate space for cabling, and that there finally is the possibility to avoid the flying spaghetti train wreck we see so often in Data Centers.
4. Front to back Cooling. The cooling design is well thought out. I liked the fact that it draws from directly above the front floor and exits rear top.. This should help out in raised floor data centers that have a large temperature gradient as you move to the top of the rack. It also negates problem of having multiple 6500 chassis side to side and having warm air blowing from the exhaust of one 6500 to the intake of another 6500.
5. Fan Slots are now placed where it is IMPOSSIBLE to cover with cables. I would say 7 out of 10 times when I walk into a new customers Data Center I find that there are cables run directly over the fan tray with no slack. That is not a failure in design per say, but it could have been avoided. With the Nexus 7000 fan trays in the back the problem is solved before it is created.
6. Power supplies are in the back . FAR away from the data cabling. It never fails that 20 amp circuits get uncomfortably close to copper cabling. By moving the power supplies to the back side of the chassis, this becomes a mute point and we remove any shadow of a doubt about EM interference causing craziness in our cabling.
7. This one sounds really mundane, but a quick heads up grouping of status lights. In the past these were normally in a position where you had to squat down to see them, or they are obscured by cables. Buy putting them on the front of the cable tray assembly it ensures these will always be visible.

What can we focus on now to make it a better platform?

1. One thing that worried me a little was the placement of the compact flash cards in the supervisory module. For those how haven’t it up close look at this picture of the chassis  and look for the Grey cover midway up the sup modules in the center slots. Behind them are two flash cards, one for system partition extension, and one to dump log files into. Having these cards available are great features however I could see an operational process of security rotating out the log partitions, or more likely and engineer pulling the flash card after dumping some data for analysis to it, and then pulling the wrong card by accident. Having a simple strap (like the screw downs for power supply plugs) or something similar would go along way towards mitigating that risk.
2. Continue with the spirit of innovation that has defined Cisco over the years. Cisco has consistently came out with or acquired and integrated many great products that directly address the needs of the market place into the product line (MARS, ASA, AireSpace, TelePresence, MDS, ACE, Etc) but frankly the last GAME CHANGING product that set the industry on its heals and forced everyone to rethink how we utilize technology to accelerate business as a whole was the acquisition of Selsius and the introduction of VOIP as an enterprise class product to the world. I remember having the hair stand up on my arms from the excitement of going up against Avaya and Nortel back then and fighting that uphill battle, educating customers and peers about this “new thing called VOIP and how CallManager (now Unified Communications Manager) is your ticket towards productivity.

   When we talk about the Virtual DataCenter, I/O Virtualization (FCOE) and VFrame Automation it is not just another incremental improvement of existing technology. It is a paradigm shift, a leap ahead, a GAME CHANGER. I get the same chills that I did when VOIP was new because I know that those are technologies that will force us to rethink how we approach computing and data systems. These technologies are to the Data Center what IP telephony was to the PBX, and Cisco is the only company with technologies and engineering know how in all the verticals necessary to pull this off.

Similar Posts:

• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Moving towards a Green Data Center – Truth behind the hype
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Usability features in Cisco’s Nexus 7000

As with any vendor specific event, there is a mix of for public knowledge and for private consumption content, so I generally choose not to talk or write about subjects that may have been covered in the VT unless I can find some public documentation on that subject. So don’t expect to find any juicy pre-release information or gossip here. I don’t want to have the NDA police knocking at my door, and its just not cool to let stuff slip. So, I will generally avoid the subject.

What I can tell you is this – There is a lot of buzz about the Nexus 7000. It is a rocking platform, and we spent the majority of a day going over it. I can’t share much more then I did the night before the VT just yet (will wait till I get lab access to one) but I can share this.

Yes, most people are proud of their shots with Tom Cruise, or Oprah, Richard Stephens or BSD Girl. But I can Top that.. I have a picture of me and the Nexus 7000.

The DC Channels team was nice enough to take us down to the DataCenter and Network Applications (DNA) lab. Where the Nexus 7000 has taken its new throne. They allowed to ooh and ahh and poke and prod it. Weirdly enough, the one thing that struck most was the attention to detail that went into the physical design of this chassis. It is not only good looking, but has some super usability enhancements that really impressed me.Similar Posts:

• Darrel Hinshaw – New Triple CCIE [Storage]!!!!!!!
• Where was Colin in 2010?
• Cisco releases Nexus 1000V virtual switch for VMware
• Humor inside the Nexus 5000 switch fabric architecture
• BIG Cisco – VMware announcement – 1:30 Pacific time
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

--Colin McNamara

Me and the Nexus 7000 last week at the Data Center VT

• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Thanks and farewell to Jayshree Ullal
• Jayshree Ullal takes the helm of Arista Networks
• John McCool chosen as Jayshree Ullal’s replacement to lead Cisco’s Data Center Switching and Services Group (DSSG)
• The emergence of MDS features in Cisco’s datacenter networking equipment
• Cisco Nexus 5010 released

--Colin McNamara

DataCenter SVP Jayshree Ullal interviewed about the Nexus 7000

One feature that is new, and frankly extremely exciting is Virtual Device Contexts. Each virtual device runs with its own process, vs the use of tagged differentiators in technologies such as VRF-Lite. This provides for paravirtualized management instances, and clear lines of delineation for both software and hardware for a resource that can be shared between different groups within an enterprise.

Chassis that run NX-OS will support In Service Software Upgrades (NSSU) to allow operations groups to upgrade operating systems with zero downtime. This is accomplished through a combination of modular software architecture, and the decoupling for the control and forwarding planes.

One of my favorite features in SAN-OS is the embedded is fabric analyser. This is a tool that can sniff management traffic without having to plug in a sniffer, or provision a span port. You can dump in real time to a tcpdump like interface in the command line, output to a local file, or map to the ip of a wireshark instance that layer 3 access to the management port. Cisco again has taken the best of SAN-OS and bundled it with NX-OS. You will be able to remotely span management traffic without having to set up rspan, or trudge down to the datacenter to set up a sniffer.

Now, your router can call home right now so that is not a totally new feature. Smart Call Home was released recently into IOS. But that still doesn’t stop it from being a great feature. This allows you to configure NX-OS powered devices to mail an xml formatted troubleshooting email to TAC, and / or your support staff. This has been proven to drop the average time to resolution from 16-30 hours to 6 hours.

Now the drum roll…… All IP routing features are VRF aware. This has been a point of contention with me for a while. As Cisco and the market in general has embraced virtualization as an answer to pressing business concerns of leveraging shared infrastructure, while retaining security controls segregating disparate environments technologies such as MPLS and VRF within the datacenter have become more and more prevalent. That is great, however it never fails that the feature you need at that moment always seems to be coming out in the NEXT IOS release. With Cisco NX-OS 4.0 this is no longer a question.

Now, if I was a CIO and I was reading about all these new technologies that Cisco was pushing with NX-OS, I would frankly be cautious, and rightfully so. The thing is, most of these features are not new, they have been in use, and in production under the most stringent uptime conditions in the world – storage networking. They have been tried and tested on Cisco’s MDS line of storage networking switches. So get comfortable, get educated, but most importantly get on board for DataCenter 3.0.Similar Posts:

• The emergence of MDS features in Cisco’s datacenter networking equipment
• Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers
• New features in VMware 3.1
• Link Round Up – L2TPv3 FCOE Trill Wounded Warriors
• Arista Networks – Their approach to cloud networking
• About Colin McNamara

--Colin McNamara

Cisco NX-OS 4.0 | Next Generation Internet Operating System