He writes about “the VLAN leaking myth” and how it encourages clients to utilize physically separate network infrastructure in the DMZ’s. Now first things first, I wouldn’t call VLAN leaking a myth. At one time it was a very real and serious vulnerability that was exploited by overflowing the capacity of the switch you were attacking, and causing it to “downgrade” from switch to a hub. Once this happened you now had access to previously protected devices, as well as having the ability to sniff data as it passed through the shared hub backplane.

As he mentions though, this is 8 years ago. Most switches have evolved to the point where backplanes far exceed the traffic that could ever be injected into their switchports. Even beyond backplane enhancements there are many ways to further firm up your security stance – Virtual Device Contexts, not using Layer 3 SVI’s on a DMZ VLAN, utilizing PVLANs, using port security, virtual routing instances, and many more. Of course, there are still many other attack vectors that still remain, but can be mitigated by utilizing features built into the majority of enterprise switches available today.

I think the real question is not “are VLANs safe in a DMZ”. The important question is have you mitigated the probability of compromise (the actual threat) to levels that are acceptable to your business. This question remains whether you have a standalone switch or not. So many times we hear about risk risk and more risk. But risk alone is meaningless in a business context. What is important is combining risk with likelihood. For that I like to use a simple table to come up with the true threat.

For example, as I drive to Fry’s there is the risk of me dying due to a car crash. The impact of me dying is very high (risk) however the likelihood of an accident is low, and furthermore I reduce (mitigate) the latent risk (threat) by wearing my seat belt. So all in all the threat of me dying on my way to Fry’s is pretty darn low.

In a business context this may be that I have public facing web servers and network devices in my DMZ. The impact of them being compromised is that my public image may be tarnished for a short time, and my end users may lose productivity if they are not able to VPN into work, or access the Internet while on premise. I mitigate this risk by using firewalls and both host and network based Intrusion Prevention Systems as well as implementing best security practices on my network and systems devices. The latent risk (threat) remaining is at a level that is acceptable to the business leaders, so the system is allowed.

One question that I have seen coming up more often as we move towards fully virtualized data centers is centered around commingling of virtual infrastructure. There are some hard questions which challenge some practices that we have held true over the years.

• Should you allow sharing of physical memory on a host virtual machine between an internal and DMZ server?
• Should you allow virtual infrastructure from multiple security zones to share a storage array or cluster of arrays?
• Should you allow multiple virtual switches in different security zones commingling on the same ESX or Hyper-V cluster?
• Should you allow virtual firewall and load balancing instances protecting internal and external zones to reside on the same hardware?
• Should you allow virtual routing instances from multiple zones to share a physical infrastructure?

In the past world of standalone systems, the additional cost of providing a wholly separate infrastructure for DMZ environments was relatively low. Each system generally had internal disk, or at most direct attached storage. Network devices themselves were scaled down to support one chassis one function. This fit quite neatly into the Enterprise Composite Network model that was quite common from 1999-2003.

Now, many data centers have moved to the Service Oriented Network Architecture (SONA). In this model the cost of a virtualized data center is primarily focused on foundation elements such as the virtual storage and virtual fabrics, virtualized network, and virtual systems elements. The cost of providing additional virtualized services off these elements is low, however the cost of duplicating the physical infrastructure is quite high on both the capital and operational levels. This is forcing the technical and executive leadership at many companies to take a long hard look at the true threats they are facing in previously physically separate security zones such as DMZ’s, Financial and other secure zones. In the end, they are having to decide whether the threat remaining after their security controls is worth duplicating hundreds of thousands of dollars worth of infrastructure or not.

These are hard questions, with really no single good answer. My gut feel is that over the next few years we will continue the move towards the fully virtualized data center where components such as memory, PCI-X buses, storage and network devices are even further decentralized. This will make the cost of duplicating the infrastructure more and more significant, causing consolidated data center (or compute) fabrics to be the norm. At this point the discussion will move away from securing zones by creating separate infrastructure, to providing end to end security, starting integrated application level security, maybe with TrustSec or a dirivative, all the way down to securing the data at rest on disk. For the time being however, the best we can do is sit down and do an honest appraisel of our security stances, mitigate what we can, and do our best to design data center architectures that provide the flexibility of implementing whatever choice the technical and business leaders agree on.Similar Posts:

• Moving towards a Green Data Center – Truth behind the hype
• Cisco’s Cloud Computing Offering
• About Colin McNamara
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

I think Rick (5x CCIE) put it best in a company wide email earlier this afternoon  -

Please join me in congratulating Darrel in obtaining his Storage CCIE. There are only 24 double CCIES (RS/Storage) in the world, so he is probably about 1 of 15 or less in the world to hold all three.

What is everyone’s vote what is next in his career?  And, NO, you don’t have a say, Darrel

A) Service Provider

B) VOICE

C) CCDE

Thanks,

Rick Davis

ePlus, Senior Network Engineer

CCIE – Storage, Voice, Security, Service Provider, Routing and Switching (#5672)

Great job Darrel, everyone is really proud of you. and our apologies in advance to the wife for stealing you for choices A, B or C. (My vote is for C)

–Colin

Similar Posts:

• Updated CCIE numbers
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Passed CCDE written and Recertified my CCIE – Killed two birds with one stone
• About Colin McNamara
• CCIE Party 2008 Recap – Cisco Live Networkers 2008
• Are you a kick ass engineer looking to grow?

--Colin McNamara

Darrel Hinshaw – New Triple CCIE [Storage]!!!!!!!

This is the first step towards sitting for a practical examination that will hopefully be released this October. And, in response to Micheal Morris’s blog post – Yes Mike, I can pass the CCDE written.

The second big item on my todo list was to recertify my CCIE. Thankfully, Cisco counts the CCDE written towards my recertification requirements. I am waiting for the system to update, but I think this will set me up to be certified till June of 2011.

There is one interesting side effect of passing my CCDE written the week before networkers. I had originally planned on using the free exam at networkers as a “safety” exam in case I struck out on this attempt. Now of course, that is not necessary. Since I am already a CCIE, there isn’t much incentive to take a professional level exam.

So, after weighing my options, I have decided to take the CCIE Service Provider written. It covers much of the MPLS / IP Next Generation Networks material that I have been studying. I only have 7 days to prepare, so the odds are against me. However, with the heavy service provider focus in the CCDE blueprint, I have a feeling after I battle the CCDE practical I will be set to roll right into the CCIE Service Provider lab.Similar Posts:

• It’s on like Donkey Kong – CCDE practical registration is open
• CCDE Practical – Beta candidate deadline August 1 2008
• Cisco Certified Architect – Board examination above the CCIE and CCDE
• My experience taking the CCDE Practical Beta
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• What does it take to pass the CCIE exam?

--Colin McNamara

Passed CCDE written and Recertified my CCIE – Killed two birds with one stone

In the past couple years, VMware has changed from a product hidden in development and testing environments to a full fledged enterprise computing platform. It brings many benefits to the companies that implement it, however with those benefits come changes to the access layer of your data center. Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers – Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco – Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware – Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe – Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Arista Networks – Their approach to cloud networking
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Resume – Colin McNamara, CCIE #18233
• New features in VMware 3.1

--Colin McNamara

Challenges integrating VMware into Cisco networks

For those who haven’t heard yet, I passed my CCIE Lab on June 14th. Now I can officially put CCIE #18233 after my name.

Anyone who has been down the same path understands how long of a road this is.
I started down this path in the spring of 1999 as the 432nd student ever in Cisco’s Networking Academy (which I promptly dropped out of to move to the bay area). I grew my networking skills all the way to passing my CCIE Written in 2001. I attempted my Route Switch lab in 2002, however I got called away to war for a year. Being away from it all for a year really cramped my style technically. In that time many of my certs expired, and I lost much of the momentum I had built of the past half decade.

Since I got back to the real world I had focused on work to the detriment of my certifications. I really did some great things, however I really neglected my resume.
In July of 2006 I had worked 2038 hours that year (for those not mathematically inclined, that is a full work year, in 1/2 a year) . At that point I decided it was time to stop neglecting my certifications. At that point I dedicated 8 hours each saturday, along with two nights a week to studies. By December of that year I had Certified on a few of the technologies that I had tons of experience in.

I managed to get my RHCE, Cisco Storage Support and Design Specialist, update my old IP Telephony specialist certifications to CCVP, get my Netapp Certified Administrator, and pass my CCIE Written for storage networking all by the end of the year. This spring I finished my Netapp Certified Expert and scheduled my Storage lab for early summer.

Anyone who knows me well knows how closely I track my time. That time tracking extends to my training. I tracked my training (reading, lab practice, testing, etc) just like any other part of my professional life. I spent around 150 hours studying for my Design, Support specialist certs, along with reading the recommended books of the CCIE reading list, and around 300 hours preparing for my lab exam. That is 300 hours configuring every possible combination and permutation of technology that could be setup, and then refining my speed in configuring those technologies until i got to the point where speed as well as brains would be an advantage in the lab.

So now that I have my CCIE, whats next? Well, oddly enough.. I am thinking of getting my second CCIE cert. In my office I will be the Jr guy by only having a single CCIE (on of our guys has all five). I also need to take my VMware certified professional cert, and probably get my HP Master Accredited Storage Engineer. I guess I am just a glutton for punishment.

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”Similar Posts:

• Where was Colin in 2010?
• About Colin McNamara
• Updated CCIE numbers
• Why was Storage Networking my first CCIE? And What did I do to prepare?
• And it begins again – On the road to my CCIE in Storage
• Update on the Cisco / Nuova connection

--Colin McNamara

What does it take to pass the CCIE exam?

The first lesson was that knowledge alone is not suffient to pass the lab.
The second was that the lab is not the real world, and the procotors do not intend to replicate real world problems.
The third lesson was that speed is as important as brains.
The fourth lesson is that you can’t make it alone.

This time I am taking a different tack. I am studying with a partner. My focus will not be real world implementations, it will be on all possible hellish combinations a proctor could come up with. . I will be spending many late nights working on my speed.

I have passed All of Cisco’s storage exams, including the Written qualification for the CCIE Storage Networking exam. At first I thought the biggest challenge would be getting ahold of lab gear. It is not like this stuff is cheap. However I have been finding more gear out there then I had though I would. This is a good sign.

I will be posting my experiences throughout this entire process. I am not under the illusion that it will be easy. Luckily I have the lessons learned from my attempts before the war, and a new found enthusiam.

–Colin
Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• It’s on like Donkey Kong – CCDE practical registration is open
• What does it take to pass the CCIE exam?
• Why was Storage Networking my first CCIE? And What did I do to prepare?
• 3 minute management course
• My CCIE Storage Shopping List
• Updated CCIE numbers

--Colin McNamara

And it begins again – On the road to my CCIE in Storage

This is great information to have, however when you have a multi tiered application it becomes harder to get a true status of your application stack. Luckily most new applications nowadays incorporate a HTTP interfaces.
However, this interface responding to a http get request generally does not indicate full application functionality. To get that level of information out you usually need to present some authentication to the web app, execute a click through, etc.

Now this has always presented a problem for me. I can check the availability of each level of the application stack. What I can’t do is verify that the application is truly working properly. This puts a major blind spot in my monitoring strategies. Luckily I just came across a very cool application. This sweet application is webinject – http://www.webinject.org/
It can be configured to graph application performance to mrtg, or output to nagios network monitor. Below is some information about this project.

What is WebInject?
WebInject is a free tool for automated testing of web applications and services. It can be used to test individual system components that have HTTP interfaces (JSP, ASP, CGI, PHP, Servlets, HTML Forms, etc), and can be used as a test harness to create a suite of [HTTP level] automated functional, acceptance, and regression tests. A test harness, also referred to as a test driver or a test framework, allows you to run many test cases and collect/report your results. WebInject offers real-time results display and may also be used for monitoring system response times.
WebInject can be used as a complete test framework that is controlled by the WebInject User Interface (GUI). Optionally, it can be used as a standalone test runner (text/console application) which can be integrated and called from other test frameworks or applications.

Programming Language and Platforms
WebInject uses an XML API (interface) for defining and loading test cases. You can use WebInject without ever seeing it’s internal implementation (no scripting or programming necessary to use it).
WebInject is written in Perl and can run on any platform that a Perl interpreter can be installed on (MS Windows, GNU/Linux, BSD, Solaris, MAC OS, and many more). Currently, binary executables of WebInject are only available for MS Windows. If you would like to run on other platforms, you must have a Perl interpreter and run it from the Perl source code.

Test Cases
Test cases are written in XML files, using XML elements and attributes, and passed to the WebInject engine for execution against the application/service under test. This abstracts the internals of WebInject’s implementation away from the non-technical tester, while using an open architecture [written in Perl] for those that require more customization or modifications.

Results/Reporting
Result reports are generated in HTML (for viewing) and XML (for tranformation by external programs). These detailed results include pass/fail status, errors, response times, etc. Results are also displayed in a window on the User Interface if you are running the WebInject GUI, and are sent to the STDOUT channel if you are running the WebInject Engine as a standalone (console) application.

Service-Level Monitoring
HTTP response times can be collected and monitored in real-time during test execution. Timer statistics are calculated and displayed in a monitor window during runtime. When used along with gnuplot (a plotting utility), a response time graph is generated and updated in real-time as the test runs. This is used to verify responses from the web application or web service under test are within an acceptable range (to meet your SLA or quality of service criteria). This also enables WebInject to be run as a performance probe for application/service monitoring.
WebInject can also be integrated as a plugin for external monitoring systems. In this case, it is used in console mode as an intelligent test agent that returns status and response times to your external program.
For real-time monitoring of your web applications or web services, WebInject is able to run in a mode that makes it compatible with Nagios. Nagios is an open source host, service, and network monitoring program.
For graphical trending of web service-levels over a long period of time, WebInject is able to run in a mode that makes it compatible with MRTG. MRTG (Multi Router Traffic Grapher) is an open source tool for collecting, storing, and graphing time-series data.

–Colin
Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• What should I do this thanksgiving break?
• Application Extension API notes – Cisco Live 2008
• Why GoDaddy Linux Virtual Dedicated Hosting Sucks & How to Fix It
• New Buzzword Enterprise Service Bus
• RSS feeds – an intranet aggregation solution?
• PHP / MySQL / FLASH website integration

--Colin McNamara

User experience testing – enhanced