What is a Fabric Extender?

If you think of a large switching chassis such as a Catalyst 6500 or Nexus 7000 you have the notion of a Supervisory module, a switching fabric forming the backplane, and line cards connected on the back into the fabric, and connecting to servers and other networking devices in the front. In the case of these switching chassis all of the items we just talked about are connected on the inside of the same physical box. Now suppose we took each of those components, the Sup, Fabric, and Line Card and installed them in different parts of your data center. The Sup module in this case would be a Nexus 5020 or Nexus 5010. The fabric would be four 10 Gig port bundles from the Nexus to the Fabric Extender, and the line cards themselves would be replaced by the Nexus 2000 Fabric Extender.

Why use a Nexus Fabric Extension in your Data Center designs?

1. Simplify network management – By moving from individualy managed standalone edge switches to an extended fabric you minimize the amount of devices in your datacenter that you have to manage. Say you have 10 racks full of pizza box servers, (That makes 42 servers per rack) you may throw 3750′s or 4948′s top of rack to provide connectivity. In this case your access layer would consist of at minimum 10 unique devices to manage, upgrade, etc. in your server access layer. If instead you put a Fabric Extender at the top of each rack, you still get 48 ports of coppert gigabit ethernet, but logically you would only have 1 device to manage.

2. Simplify your network topology – In the scenarios mentioned above with classic switching, you would have to manage a spanning tree topology accross your entire access layer. Half of your uplinks will be blocked by spanning tree so your network does not loop. This is a perfectly valid topology, but it does leave you open to the pains of spanning tree in your data center Now replace these switches with a Nexus 2000 Fabric Extender. Instead of spanning tree to integrating individual switches, you now how a very simple U topology forming your edge.

3. Lower your costs for Data Center class switching – Before the Nexus 2000 if you wanted to properly deploy a Data Center access layer, you either had to do buy 6748 blade on a 6500, a 48 port gig blade on a Nexus 7k (M148GT) , ora 3750 or 4948 switch. These all are not the cheapest switches and blades, but they provide each gig switch port with the proper buffers and throughput that are needed for true Data Center applications. Now if we chose to use the Nexus 2000 Fabric Extender in conjunction with the Nexus 5000 we can provide Data Center class 1 Gig Ethernet services at roughly half the cost of using the older switches.

4. Integrate 10 gig and 1 Gig services in a clean way – I have seen many many clients try to cleanly integrate 10 Gig services (normally from blade centers w/ ESX clusters) into an existing 1 gig data center edge. Normally what happens is that the “best” integration case is to put some form of oversubscribed 10 Gig card into their existing distribution layer, or in a worst case some customers are only able to provide 10 Gig of a collapsed core. This is obviously not an ideal setup. By integrating the Nexus 5000 40 or 20 port 10 Gig switches with the Nexus 2000 Fabric extender it provides us with a very clean way to integrate both 10 Gig and 1 Gig services in our Data Center access layer.

Why am I excited about this product?

I am lucky enough to work with some forwarding thinking customers with very large Data Center requirements. We have been working with Cisco over the past couple months to incorporate this end to end Nexus infrastructure into multiple customer designs. Before the Nexus  5000 / 2000 combination, designs to provide large layer 2 adjacent mixed 10 and 1 Gig services were well…. not as elegant as we would have liked to see. Now with Fabric Extension, we  are able to answer this mixed 10 / 1 Gig Data Center access layer design scenario in an elegant and scalable way.

Want to learn more?

Nexus 2148T Product Page - Cisco.com
Similar Posts:

• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco Nexus 5010 released
• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Cisco introduces the C-Series Rack Servers

--Colin McNamara

Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)

The boundary between server team and network team responsibilities has become “fuzzy”

Cisco address’s this issue by putting a switch that can be managed via the same methods common to other network devices inside the ESX cluster. This switch runs the same code that has become standard on Cisco’s Nexus series of Data Center switches – NX-OS.

Prior to adoption of virtualization, when there was a connectivity problem with a host it was quite common for the network team to verify functionality down to the switch port. The server team would do the same. This allowed for each team to focus on areas that met their core competancy. Once we moved from a real switch port, to a dumb bridge inside ESX, lots of finger pointing resulted.

Now, with a Nexus 1000V sitting virtually inside the ESX clusters, the boundary between network and systems teams has been re-estabilished. Now when there is a problem with a host inside an ESX cluster, the network team can use the same day to day troubleshooting tools available to them in other portions of the network to resolve issues faster, and with less finger pointing.

Security controls have been moved further away from the hosts then we would like

A best practice for applying security policy is to apply controls as close to the source as possible. Think of this analogy – Your kids are blasting Radio Disney from their computer. Which of the following do you do?

A. Turn down the speakers at the source

B. Distribute earplugs to all members or the household

Of course, the obvious action is to go to the source, and apply a control (turn down the volume, and tell the kids to clean their rooms). The same principle is valid on the networking side. The best practice is to apply security policies such as VLAN ACL’s and TrustSec policies directly to the switchports that host your switches. Before the Nexus 1000V this was impossible to do in ESX, and forced many environments to move security controls further up into the distribution layer. The side effect of this was that now the security stance from host to host inside ESX clusters was diminished.

The Nexus 1000V brings something called port policies to the table to address this. What these are is pre-configured application security descriptions that are available to you systems administrators to apply in a point and click fashion. Once these policies are applied to the virtualized host, they follow the host where ever it is moved in your virtual cluster.

Provisioning and integrating the networks of VMware ESX clusters with classic networks for most is challenging at best

I wrote an article in march about this specific issue in my post – Challenges integrating VMware into Cisco networks . The core of this issue is that in general that the network integration portions of VMware ESX clusters is not really designed to address server teams , or network teams. In fact, you need to be pretty savy with both portions to successfully integrate VMware clusters into your network. In the real world, you generally find people that are good at one or the other, not both.

By putting a Nexus 1000V in your VMware clusters, you know give the networking teams something they can understand without having to learn Linux, and how it handles bridges (key to understanding ESX networking). With a Cisco switch running virtually inside your clusters, network teams can follow standard core / distribution / access models with the access layer now residing inside the ESX clusters. The network teams can also leverage their existing LAN switching skills for integrating the virtual switches in the clusters with the existing Data Center switching fabrics.

With these roadblocks addressed, Cisco is moving to further the DC 3.0 vision

To realize the DC 3.0 vision, the network inside of VMware clusters had to be under control, and follow the same architectural guidelines that the rest of our network is subject to. With the Nexus 1000V this is now a reality. The next steps withing the DC 3.0 vision to are to extend virtualization and mobility throughout our storage fabrics, and to continue to extend virtualization to the network as a whole, as well as focusing on application virtualization and acceleration to truly realize the vision of cloud computing in the data center.

On the storage virtualization side, Cisco will be using a technology called FlexAttach to enable virtual and physical hosts to change locations in the datacenter without storage team intervention (more on this in a near future post). And on the application virtulization and acceleration side, expect Cisco to continue to enhance it’s existing Application Control Engine (ACE) and Wide Area Application Services (WAAS), and further integrate these into their virtualization offerings.

Want to learn more ?

Introduction to VN-Link network services – Cisco.com

Nexus 1000V overview – Cisco.com

VMware distributed vNetwork switch demo – VMware.com

Challenges integrating VMware into Cisco networks – colinmcnamara.com

Douglas Gourley speaking about how Cisco and VMware will drive Cloud Computing in the Data CenterSimilar Posts:

• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?

--Colin McNamara

Cisco releases Nexus 1000V virtual switch for VMware

1. Layer 2 Tunneling Protcol (V3) static and hairpin configuration example - my buddy Rick was nerding it out in the lab and sent a great configuration doc for L2TPv3 my way. L2TP(V3) is used to create a layer 2 psuedowire across layer 3 routed links. This is a great service provider tool that you can use in your own network, no MPLS needed .

2. SNIA Education – Fiber Channel Over Ethernet – There is a lot of buzz going around right now about Fiber Channel Over Ethernet (FCOE). There is also a lot of misunderstanding about the fundamentals of this architecture. This Storage Networking Industry Association (SNIA) does an outstanding job of covering FCOE at both at an architectural level, as well as going over low level messaging structures.

3. Trill (Rbridge) architecture – IETF internet draft – I think the last time I was this interested in an internet draft was when iSCSI was first being proposed in the IP Storage working group. Trill, in my opinion is basically a light weight version of MPLS / VPLS. It has as far as I can tell most of the advantages of this architecture, without some of the configuration and hardware requirement drawbacks. Fair warning, reading this document started a doc hunt that killed my Saturday.

4. Cisco’s Security Response to Sebastian Muniz’s IOS rootkit – Security is a very important aspect of network design. Sebastian’s IOS rootkit demonstration is going to force some customers who in the past have been “OK” with having older, possibly vulnerable IOS versions floating around to update their operational practices and start keeping their routers and switches operating systems as often as they do their servers. Thankfully, Cisco has been embracing technologies such as kernel virtual machines, in service software upgrades and more to lesson or remove the impacts of software upgrades.

5. Turning Wounded Warriors into Network Ninja’s – As a former Marine (well, always a Marine, formerly employed by the USMC) this program goes straight to the heart. Cisco is partnering with Naval Medical Center San Diego (NMCSD, or Balboa Naval Hospital for us locals) to provide technical training to Marines and Sailors who have recieved service ending wounds in Afghanastan and Iraq.Similar Posts:

• Fibre Channel over Ethernet is taking off
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Zone based IOS firewalls
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers
• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0

--Colin McNamara

Link Round Up – L2TPv3 FCOE Trill Wounded Warriors

For new readers to this site, please you might enjoy the following articles -

Who is Colin McNamara ?

Usability features on the Nexus 7000

Overview of NX-OS 4.0 (the operating system for the Nexus 7000)

Why did I choose to get a CCIE in storage networking?

What does it take to pass the CCIE exam?Similar Posts:

• And it begins again – On the road to my CCIE in Storage
• What does it take to pass the CCIE exam?
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Cisco Nexus 4000 Blade Switch
• created a new map of ashley’s site
• Certguards inflamitory post about Ethan Banks removed from Network World Magazine

--Colin McNamara

Featured on Network World – 20 useful sites for Cisco networking professionals

What useability enhancements do you feel are the most beneficial?

1. A separate, IP enabled, Management Interface. This has been a long time coming. The out of band management interface is very similar to a Ilo card in the HP world. it is effectively a supercharged console server that happens to site on the backplane of the sup engine. I am sure whoever pushed this feature through is going to get flowers one day from a Tech who DIDN’T lock himself out because the management interface was effectively a separate system.
2. Finally, a functionally USB Interface that I can transfer IOS (well, now NX-OS) images through. Everyone has a USB key nowadays, even my Grandmother has one, it will make life so much easier when I can have a 4 gig key with me that has most IOS / NX-OS  versions and my common configs and just pop them right in.
3. The integrated Cabling system is CLEAN. I love that it forces you to reserve the appropriate space for cabling, and that there finally is the possibility to avoid the flying spaghetti train wreck we see so often in Data Centers.
4. Front to back Cooling. The cooling design is well thought out. I liked the fact that it draws from directly above the front floor and exits rear top.. This should help out in raised floor data centers that have a large temperature gradient as you move to the top of the rack. It also negates problem of having multiple 6500 chassis side to side and having warm air blowing from the exhaust of one 6500 to the intake of another 6500.
5. Fan Slots are now placed where it is IMPOSSIBLE to cover with cables. I would say 7 out of 10 times when I walk into a new customers Data Center I find that there are cables run directly over the fan tray with no slack. That is not a failure in design per say, but it could have been avoided. With the Nexus 7000 fan trays in the back the problem is solved before it is created.
6. Power supplies are in the back . FAR away from the data cabling. It never fails that 20 amp circuits get uncomfortably close to copper cabling. By moving the power supplies to the back side of the chassis, this becomes a mute point and we remove any shadow of a doubt about EM interference causing craziness in our cabling.
7. This one sounds really mundane, but a quick heads up grouping of status lights. In the past these were normally in a position where you had to squat down to see them, or they are obscured by cables. Buy putting them on the front of the cable tray assembly it ensures these will always be visible.

What can we focus on now to make it a better platform?

1. One thing that worried me a little was the placement of the compact flash cards in the supervisory module. For those how haven’t it up close look at this picture of the chassis  and look for the Grey cover midway up the sup modules in the center slots. Behind them are two flash cards, one for system partition extension, and one to dump log files into. Having these cards available are great features however I could see an operational process of security rotating out the log partitions, or more likely and engineer pulling the flash card after dumping some data for analysis to it, and then pulling the wrong card by accident. Having a simple strap (like the screw downs for power supply plugs) or something similar would go along way towards mitigating that risk.
2. Continue with the spirit of innovation that has defined Cisco over the years. Cisco has consistently came out with or acquired and integrated many great products that directly address the needs of the market place into the product line (MARS, ASA, AireSpace, TelePresence, MDS, ACE, Etc) but frankly the last GAME CHANGING product that set the industry on its heals and forced everyone to rethink how we utilize technology to accelerate business as a whole was the acquisition of Selsius and the introduction of VOIP as an enterprise class product to the world. I remember having the hair stand up on my arms from the excitement of going up against Avaya and Nortel back then and fighting that uphill battle, educating customers and peers about this “new thing called VOIP and how CallManager (now Unified Communications Manager) is your ticket towards productivity.

   When we talk about the Virtual DataCenter, I/O Virtualization (FCOE) and VFrame Automation it is not just another incremental improvement of existing technology. It is a paradigm shift, a leap ahead, a GAME CHANGER. I get the same chills that I did when VOIP was new because I know that those are technologies that will force us to rethink how we approach computing and data systems. These technologies are to the Data Center what IP telephony was to the PBX, and Cisco is the only company with technologies and engineering know how in all the verticals necessary to pull this off.

Similar Posts:

• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Moving towards a Green Data Center – Truth behind the hype
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Usability features in Cisco’s Nexus 7000

As with any vendor specific event, there is a mix of for public knowledge and for private consumption content, so I generally choose not to talk or write about subjects that may have been covered in the VT unless I can find some public documentation on that subject. So don’t expect to find any juicy pre-release information or gossip here. I don’t want to have the NDA police knocking at my door, and its just not cool to let stuff slip. So, I will generally avoid the subject.

What I can tell you is this – There is a lot of buzz about the Nexus 7000. It is a rocking platform, and we spent the majority of a day going over it. I can’t share much more then I did the night before the VT just yet (will wait till I get lab access to one) but I can share this.

Yes, most people are proud of their shots with Tom Cruise, or Oprah, Richard Stephens or BSD Girl. But I can Top that.. I have a picture of me and the Nexus 7000.

The DC Channels team was nice enough to take us down to the DataCenter and Network Applications (DNA) lab. Where the Nexus 7000 has taken its new throne. They allowed to ooh and ahh and poke and prod it. Weirdly enough, the one thing that struck most was the attention to detail that went into the physical design of this chassis. It is not only good looking, but has some super usability enhancements that really impressed me.Similar Posts:

• Darrel Hinshaw – New Triple CCIE [Storage]!!!!!!!
• Where was Colin in 2010?
• Cisco releases Nexus 1000V virtual switch for VMware
• Humor inside the Nexus 5000 switch fabric architecture
• BIG Cisco – VMware announcement – 1:30 Pacific time
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

--Colin McNamara

Me and the Nexus 7000 last week at the Data Center VT

• Cisco Nexus 7000 DataCenter switch released – Welcome to DataCenter 3.0
• Thanks and farewell to Jayshree Ullal
• Jayshree Ullal takes the helm of Arista Networks
• John McCool chosen as Jayshree Ullal’s replacement to lead Cisco’s Data Center Switching and Services Group (DSSG)
• The emergence of MDS features in Cisco’s datacenter networking equipment
• Cisco Nexus 5010 released

--Colin McNamara

DataCenter SVP Jayshree Ullal interviewed about the Nexus 7000