The boundary between server team and network team responsibilities has become “fuzzy”

Cisco address’s this issue by putting a switch that can be managed via the same methods common to other network devices inside the ESX cluster. This switch runs the same code that has become standard on Cisco’s Nexus series of Data Center switches – NX-OS.

Prior to adoption of virtualization, when there was a connectivity problem with a host it was quite common for the network team to verify functionality down to the switch port. The server team would do the same. This allowed for each team to focus on areas that met their core competancy. Once we moved from a real switch port, to a dumb bridge inside ESX, lots of finger pointing resulted.

Now, with a Nexus 1000V sitting virtually inside the ESX clusters, the boundary between network and systems teams has been re-estabilished. Now when there is a problem with a host inside an ESX cluster, the network team can use the same day to day troubleshooting tools available to them in other portions of the network to resolve issues faster, and with less finger pointing.

Security controls have been moved further away from the hosts then we would like

A best practice for applying security policy is to apply controls as close to the source as possible. Think of this analogy – Your kids are blasting Radio Disney from their computer. Which of the following do you do?

A. Turn down the speakers at the source

B. Distribute earplugs to all members or the household

Of course, the obvious action is to go to the source, and apply a control (turn down the volume, and tell the kids to clean their rooms). The same principle is valid on the networking side. The best practice is to apply security policies such as VLAN ACL’s and TrustSec policies directly to the switchports that host your switches. Before the Nexus 1000V this was impossible to do in ESX, and forced many environments to move security controls further up into the distribution layer. The side effect of this was that now the security stance from host to host inside ESX clusters was diminished.

The Nexus 1000V brings something called port policies to the table to address this. What these are is pre-configured application security descriptions that are available to you systems administrators to apply in a point and click fashion. Once these policies are applied to the virtualized host, they follow the host where ever it is moved in your virtual cluster.

Provisioning and integrating the networks of VMware ESX clusters with classic networks for most is challenging at best

I wrote an article in march about this specific issue in my post – Challenges integrating VMware into Cisco networks . The core of this issue is that in general that the network integration portions of VMware ESX clusters is not really designed to address server teams , or network teams. In fact, you need to be pretty savy with both portions to successfully integrate VMware clusters into your network. In the real world, you generally find people that are good at one or the other, not both.

By putting a Nexus 1000V in your VMware clusters, you know give the networking teams something they can understand without having to learn Linux, and how it handles bridges (key to understanding ESX networking). With a Cisco switch running virtually inside your clusters, network teams can follow standard core / distribution / access models with the access layer now residing inside the ESX clusters. The network teams can also leverage their existing LAN switching skills for integrating the virtual switches in the clusters with the existing Data Center switching fabrics.

With these roadblocks addressed, Cisco is moving to further the DC 3.0 vision

To realize the DC 3.0 vision, the network inside of VMware clusters had to be under control, and follow the same architectural guidelines that the rest of our network is subject to. With the Nexus 1000V this is now a reality. The next steps withing the DC 3.0 vision to are to extend virtualization and mobility throughout our storage fabrics, and to continue to extend virtualization to the network as a whole, as well as focusing on application virtualization and acceleration to truly realize the vision of cloud computing in the data center.

On the storage virtualization side, Cisco will be using a technology called FlexAttach to enable virtual and physical hosts to change locations in the datacenter without storage team intervention (more on this in a near future post). And on the application virtulization and acceleration side, expect Cisco to continue to enhance it’s existing Application Control Engine (ACE) and Wide Area Application Services (WAAS), and further integrate these into their virtualization offerings.

Want to learn more ?

Introduction to VN-Link network services – Cisco.com

Nexus 1000V overview – Cisco.com

VMware distributed vNetwork switch demo – VMware.com

Challenges integrating VMware into Cisco networks – colinmcnamara.com

Douglas Gourley speaking about how Cisco and VMware will drive Cloud Computing in the Data CenterSimilar Posts:

• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware
• Cisco Nexus 4000 Blade Switch
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Where is Colin ? Passing the VCP exam (VMware Certified Professional)
• Simplifying your Data Center with Cisco’s Nexus 2000 Fabric Extender (FEX)
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?

--Colin McNamara

Cisco releases Nexus 1000V virtual switch for VMware

He writes about “the VLAN leaking myth” and how it encourages clients to utilize physically separate network infrastructure in the DMZ’s. Now first things first, I wouldn’t call VLAN leaking a myth. At one time it was a very real and serious vulnerability that was exploited by overflowing the capacity of the switch you were attacking, and causing it to “downgrade” from switch to a hub. Once this happened you now had access to previously protected devices, as well as having the ability to sniff data as it passed through the shared hub backplane.

As he mentions though, this is 8 years ago. Most switches have evolved to the point where backplanes far exceed the traffic that could ever be injected into their switchports. Even beyond backplane enhancements there are many ways to further firm up your security stance – Virtual Device Contexts, not using Layer 3 SVI’s on a DMZ VLAN, utilizing PVLANs, using port security, virtual routing instances, and many more. Of course, there are still many other attack vectors that still remain, but can be mitigated by utilizing features built into the majority of enterprise switches available today.

I think the real question is not “are VLANs safe in a DMZ”. The important question is have you mitigated the probability of compromise (the actual threat) to levels that are acceptable to your business. This question remains whether you have a standalone switch or not. So many times we hear about risk risk and more risk. But risk alone is meaningless in a business context. What is important is combining risk with likelihood. For that I like to use a simple table to come up with the true threat.

For example, as I drive to Fry’s there is the risk of me dying due to a car crash. The impact of me dying is very high (risk) however the likelihood of an accident is low, and furthermore I reduce (mitigate) the latent risk (threat) by wearing my seat belt. So all in all the threat of me dying on my way to Fry’s is pretty darn low.

In a business context this may be that I have public facing web servers and network devices in my DMZ. The impact of them being compromised is that my public image may be tarnished for a short time, and my end users may lose productivity if they are not able to VPN into work, or access the Internet while on premise. I mitigate this risk by using firewalls and both host and network based Intrusion Prevention Systems as well as implementing best security practices on my network and systems devices. The latent risk (threat) remaining is at a level that is acceptable to the business leaders, so the system is allowed.

One question that I have seen coming up more often as we move towards fully virtualized data centers is centered around commingling of virtual infrastructure. There are some hard questions which challenge some practices that we have held true over the years.

• Should you allow sharing of physical memory on a host virtual machine between an internal and DMZ server?
• Should you allow virtual infrastructure from multiple security zones to share a storage array or cluster of arrays?
• Should you allow multiple virtual switches in different security zones commingling on the same ESX or Hyper-V cluster?
• Should you allow virtual firewall and load balancing instances protecting internal and external zones to reside on the same hardware?
• Should you allow virtual routing instances from multiple zones to share a physical infrastructure?

In the past world of standalone systems, the additional cost of providing a wholly separate infrastructure for DMZ environments was relatively low. Each system generally had internal disk, or at most direct attached storage. Network devices themselves were scaled down to support one chassis one function. This fit quite neatly into the Enterprise Composite Network model that was quite common from 1999-2003.

Now, many data centers have moved to the Service Oriented Network Architecture (SONA). In this model the cost of a virtualized data center is primarily focused on foundation elements such as the virtual storage and virtual fabrics, virtualized network, and virtual systems elements. The cost of providing additional virtualized services off these elements is low, however the cost of duplicating the physical infrastructure is quite high on both the capital and operational levels. This is forcing the technical and executive leadership at many companies to take a long hard look at the true threats they are facing in previously physically separate security zones such as DMZ’s, Financial and other secure zones. In the end, they are having to decide whether the threat remaining after their security controls is worth duplicating hundreds of thousands of dollars worth of infrastructure or not.

These are hard questions, with really no single good answer. My gut feel is that over the next few years we will continue the move towards the fully virtualized data center where components such as memory, PCI-X buses, storage and network devices are even further decentralized. This will make the cost of duplicating the infrastructure more and more significant, causing consolidated data center (or compute) fabrics to be the norm. At this point the discussion will move away from securing zones by creating separate infrastructure, to providing end to end security, starting integrated application level security, maybe with TrustSec or a dirivative, all the way down to securing the data at rest on disk. For the time being however, the best we can do is sit down and do an honest appraisel of our security stances, mitigate what we can, and do our best to design data center architectures that provide the flexibility of implementing whatever choice the technical and business leaders agree on.Similar Posts:

• Moving towards a Green Data Center – Truth behind the hype
• Cisco’s Cloud Computing Offering
• About Colin McNamara
• Vote for my VMworld presentation – #3221 Built to fail (shameless pandering)
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Interesting TechWise TV episode on virtualization

--Colin McNamara

Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments

Where would you use these ?

You want  your custom application to be able to react, and act on network specific information. Integrating both at a very close level. Fundamentally your application can dynamically reconfigure your router in reaction to network events.

You want to remove common services such as AAA, Syslog, DHCP, etc, IVR apps, Unified communication apps all at the branch office in the ISR. If there is a failure, your router can dynamically reconfigure around that.

AXP architecture

Base Cisco Linux os, IOS CLI, Virtual Instances, C++, Perl, Java, OSGI, Bash. Fundementally this is very similar to a fedora core 4 systems doing paravirtualization.

API Fun – What can it do

1. You can query and change both the router and the network module
2. Leverage Embedded Event Manager (EEM) to trigger events on changes, and react to network events.
3. Network Packet monitoring .. Sniff, Sniff, Sniff

My Questions –

1. How do I automate network updates, similar to YUM?
2. Is Cisco using KVM for paravirtualization?

Similar Posts:

• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Cool new features in 12.4(15)T
• Will Cisco succede where Sun has failed?
• User experience testing – enhanced
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Routers can email you when they go down

--Colin McNamara

Application Extension API notes – Cisco Live 2008

When I work with data center infrastructure I expect the following – clean, remotely manageable, secure devices that runs on the same power and similar cabling, and everything can have a 24x7x4 support contract for hardware replacement. For the most part, you get this when dealing with Cisco, HP, Sun and similar manufacturers.

More often then not (with a few very cool exceptions), when I run into video surveillance infrastructure the video management infrastructure runs on some random third tier manufactured server. It never fails that the video management software is on Windows (normally XP or win2k). I have even seen some systems where the vendor requires you to have a session open to run the software.

And then when you get to the encoders themselves, it never fails. You have two choices.

1. The Uber package that can run a Casino, Identify and track dust mites , and if you point it at space, determine if there is life on mars.
2. Individual dinky encoders that run one or two camera’s each. They have limited encoding choices, limited camera control, no remote management, and normally run on 110 volt system that require different power distribution then the 220 that is common in systems today.

Cisco’s answer to this mess

Cisco has released both a video management solution, as well as a video encoding solution in a network module form factor for the Integrated Services Router (ISR).

The first part of this system, the Video Management and Storage System (VMSS) module fills the following roles -

• Management of multiple video streams from one interface, including IP cameras, 3rd party encoders, and streams from Cisco’s video encoding module
• Streaming of live and archived footage through a web browser interface
• This one is pretty cool – The module can mount external storage via iSCSI. So, in addition to its 160 gig internal drive, you can mount a filer and utilize external storage to scale the system.
• “fast forward” to events, as well as notify security and other personnel through SMS and email

The second part of the system (the module on the left in the picture above) is the Analog Video Gateway Network Module (EV-IPVS-16A). It has a couple functions -

• It can take up to 16 analogue video inputs and encode them with MJPEG or MPEG4 codecs
• You can use the first two ports to output video to a external monitors
• If you are using MPEG4, it can be used as a motion detector (handy for fast forwarding to important events, or triggering alerts)
• It can control pan and tilt cameras. This is good for pointing the camera at the janitor unplugging your servers each night to vacuum
• You can configure analogue contacts as an alarm. This can be bound to a door switch, or even temperature and water level monitors in a remote data center. This one will be very handy.

The third part of this solution is Cisco’s Video Surveillance Operations Manager. It manages, archives, displays and distributes the content that was created and collected on the two previous modules. You would use this if you had many branches to aggregate, or needed to staff a video wall (e.g. casino gaming commission operations). Now, you can run each of these components individually. Buy run together as a whole, Cisco has an enterprise class security solution.

Want to learn more ?

Branch office security page on cisco.com http://www.cisco.com/en/US/products/ps9671/prod_module_series_home.html

Cisco’s product page for the Video Managment Module – http://www.cisco.com/en/US/prod/collateral/modules/ps9671/data_sheet_c78_462225.htmlSimilar Posts:

• Interesting TechWise TV episode on virtualization
• About Colin McNamara
• Measuring and mitigating risk involved with sharing virtual infrastructure between DMZ and Internal environments
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Cisco Nexus 5020 and 5010 FCOE video ordering guide
• Cisco NX-OS 4.0 | Next Generation Internet Operating System

--Colin McNamara

Simplifying remote site security with Cisco’s new video surveillance modules on the ISR

In the past couple years, VMware has changed from a product hidden in development and testing environments to a full fledged enterprise computing platform. It brings many benefits to the companies that implement it, however with those benefits come changes to the access layer of your data center. Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers – Who is responsible for the configuration and maintenance of this Vswitch? At first glance most groups reference the port on the last Cisco switch as the division of responsibility between network operations and systems operations. This has worked well in the past for a three main reasons.

First, it divided responsibilities based on technical skillset. For example a network engineer understands spanning tree, trunking, routing protocols, firewalling. While a systems engineer understands file systems, databases and Linux and Windows operating systems.

Second, it provided for a interconnection point where standardized configurations could be applied by an operational group, versus complicated configurations that could impact overall network designs and require an architectural board review.

Third it provided for a clean hand off for troubleshooting. Both network and systems operations could agree on layer 2-4 functionality in an area that provided for detailed debugging on both sides.

Lack of a defined access layer

VMware ESX throws a wrench in this model. We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration.

The need for collaborative design sessions

The single most powerful element in a successful VMware integration is the creation of strong design documents. These are created by holding planning sessions where both your systems and networking leads hash out a strong design that takes both short and long term virtualization and network goals into account. Also, many times when people hear the word design, they think it is a high level Visio and a bill of materials. That is a just a fraction of the effort required. A proper design should cover everything from a 10,000 foot overview Visio down to protocol flow diagrams and configuration examples. By created a detailed design like this it is likely to bring up common issues such as 10 gig aggregation, trunking, VMotion security, layer two adjacency and layer 7 network service delivery on a white board instead of a production environment.

To create this detailed design, both your Network and Systems leads have to understand this product. VMware recognizes this is critical to successful implementation (and to further sales of their product) an offers the VMware Certified Professional certification. If you have the resources, I would recommend sending both your network and systems leads to this training at the same time. Having them attend training together allows them to leverage each others strengths and bring up questions specific to their network and their goals.

A real world example of this is the company I work for, Eplus. Last April forty of us, all senior engineers attended VMware Certified Professional training at the same time. The class was mixed up so there was an even distribution of CCIE’s, Systems Experts, and Storage Experts. Needless to say this presented our instructors with some extremely challenging questions, but more importantly it set the stage and created a venue for collaboration between these different practices within our own company.

Real world benefits

A great example of this model’s success this occurred last month. Rick and I were sitting in the engineering side of our Sunnyvale office, catching up on email after giving presentations at Cisco that morning and afternoon. In the bullpen behind us, one of the Microsoft architects was engrossed in a troubleshooting call with a large customer on the other line. It turns out a large systems vendor (who shall remain nameless) had been trying for a week to integrate the first ESX cluster into this network and just could not get the networking portion to work correctly. Our account manager received the call from a the customer, and asked the technical teams to step in to see if we could help out in any way.

The systems engineers were able to isolate the problem down to the network interconnections, but needed to bring in networking resources to resolve the problem. Rick and I were waved over and were given an overview of the problem and introduced us to the customer the far side of the call. We asked a few questions about the physical and logical architecture of their network and created a diagram of their network on the whiteboard. With this we were able to ask them to execute commands continuously isolating the problem domain until we found and resolved the issue.

Seven minutes had passed from the point Rick and I were waved over to the point the customer had a working installation. This allowed the customer to focus on moving their business forward instead of fixing a failed implementation. Three of us on the call had attended VMware Certified Professional training together. We had spent at a minimum 50 hours each creating a baseline of understanding in class, as well as many discussions in engineering meetings. The solution came in seven minutes not because of any one teams individual strengths, but because of collaboration. The systems engineers were able to isolate the problem domain very specifically. And as network engineers trained on VMware were able to quickly understand and digest the issues, and tie it together with our larger understanding of networks as a whole. Only at that point, when the team was able to leverage each others strengths were we able to address the problem so quickly.

There will come a point in the next few years where this fuzzy boundary between the “network” and the “server” is established again. My call is that this will coincide with Cisco finishing development of their Vswitch that will reside inside the ESX server. This switch will require both Cisco and VMware improve their design and integration guides for ESX which are both frankly lacking substance. Until those detailed architecture, integration and troubleshooting guides exist the key to successful ESX cluster implementation will be a strong cross trained systems and network teams that are collaborating on the next level of virtual network design in your enterprise.

Want to learn more?

Cisco – Integrating Virtual Machines Into Cisco Data Center Architecture

This is Cisco’s main design guide regarding the integration of virtual machines. You can use it as a decent high level overview if you are a network engineer who is curious how VMware ESX, or Xen servers for that matter will fit into your network.

VMware – Virtual networking Concepts

This VMware document goes between high level overviews and detailed descriptions. It is a decent resource for a network engineer, and provides an overview of ESX network features, however it misses the target for providing configuration examples.

Blog of Scott Lowe – Technical Lead for Virtualization at Eplus Technology

Scott is an engineer that works with me at Eplus Technology. He is based out of the east coast and covers servers, storage and virtualization. His blog is chock full of good of information. A recent post of interest was how to enable Cisco Discovery Protocol (CDP) on VMware ESX server network interface cards.Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Arista Networks – Their approach to cloud networking
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Resume – Colin McNamara, CCIE #18233
• New features in VMware 3.1

--Colin McNamara

Challenges integrating VMware into Cisco networks

IOS-XE is takes the best elements out of Internet Operating System (IOS) which has its roots in a closet at Stanford, and combines them with the most successful open source technology ever – Linux. Cisco is leveraging Linux virtualization technologies such as Kernel Based Virtual Machines to protect against operating system failures as well as to allow for In Service Software Upgrades (ISSU).

To really appreciate this, we first have to dive down into the overall architectural changes of the ASR1000. The largest change that Cisco has made was to implement separate forwarding and control planes. In the past, Cisco routers would have the processes responsible for forwarding traffic, and the processes responsible for configuring the router running on the same root operating system. The side effect of this is that if you want to upgrade the root operating system of your router, you are going to have interrupt the traffic flowing through it to do so, or have a physically separate route processor to take over while you rebooted. This is a big headache operationally, and effectively forced engineers to design in separate physical chassis to meet high uptime requirements.

What Cisco has done to address this, was to mirror changes made in their storage and carrier routing portfolios. Both of those product lines utilize the operating system to push commands into advanced processors that exist on the line cards themselves. The ASICS on the line cards are designed to work in a distributed fashion, so that production traffic never goes into up into the router processor (or sup engine). This in effect ensures that the control and forwarding planes can exist as independent elements.

If you look at the graphic above, you will notice 3 main zones. The upper zone is what we would normally describe as the control plane. This is where the higher level functions such as your routing processes, ssh daemons, snmp daemons, and shells live. In short, if you you configure or read something, you are going to do it here. The only time traffic flows through this plane is when you are doing a thing called process switching. keep in mind this is a rare occurrence and usually occurs because of an oversight in your network designs.

By separating the control and forwarding planes, this allows Cisco to basically run a management station on the router, that programs chip sets in the line cards on the fly. This in my opinion is where the true power of this architecture comes through. By separating the two functions the software engineers are free to utilize powerful open source technologies such as Kernel-based Virtual Machines, and the Linux kernel, while letting the integrated circuit engineers design blazing fast chips which allow full functionality at line rate.

What benefits should we receive from a virtualized control plane? First, in larger routing and switching chassis (including the top end of the ASR1000 line) you normally have physically redundant route processors (RP)/ supervisory engines(SUP). The operating systems on these RP’s synchronize many things, including configuration, process state, routing tables, security associations and much more. The primary reason for this, is if you have a failure in the active RP, you can failover to the standby RP without interrupting traffic flows.They also can be used to streamline the software upgrade process by only upgrading one RP at a time, and then gracefully transferring traffic to it. Once proper operation is verified, the backup RP can be brought up to the same code revision.In any production environment this is highly desirable, and helps immensely in the battle for five nines.

The ASR1000 takes the redundant RP concept seen in high end chassis, and allows you to implement redundant upgrades, as well as protection against software failure, with only one physical route processor. This is done by utilizing Linux kernel virtualization. Instead of running the control plane directly on the production hardware, a small kernel is inserted. Booting from that are two copies of IOS-XE. These run independently, and synchronize state and configurations just as if you had two physically separate route processors. What this means in operational English, is that where in the past, you would have to either have two devices, or a larger device with redundant RP’s to upgrade without disruption, you can now have that same ease of maintenance, in a much smaller (and at the end of the day, less total cost) package.

Below this is the forwarding plane.It plugs into to a high speed interconnected fabric which all line cards and RP’s are redundantly connected to. In the diagram above, this is the bottom level. Items in this plane include buffer memory, Cisco Express Forwarding (CEF) ASICS, and now the new QuantumFlow processor. This is normally where you would find your DCEF enabled line cards, fibre channel and Nexus7000 line cards, as well as the modules for the ASR1000 routers. When properly utilized, traffic should be relatively isolated to this tier, and function independently from the control plane.

The shining star of the ASR1000′s forwarding plane is a group of chips that is referred to as QuantumFlow. The QuantumFlow architecture itself merges Cisco’s strength in integrated circuit design, with its strengths in IOS software design. In the past, Cisco would design ASICS’s for specific functions, and then write commands down into them. This has worked very well, until they point that a new feature came out that couldn’t leverage the fixed configuration of an older ASIC. Your choice at that point was generally to process switch for that feature (which is slower, and honestly bad form), or upgrade your cards to the newer ASIC design. The QuantumFlow chipset approaches this problem from a new angle. The first chip in the set (Popeye) is designed to be field programmable in C, as well as no fixed internal pipelines. This combined with utilizing 40 cores running between 900 and 1200 megahertz allows the programmers to utilize parallel processing techniques to utilize an immense amount of processing power in real time.

To put things into perspective, remember when you got your first multi core laptop or desktop. You were able to say watch a DVD, as well as compile code at this same time, while continuing to have a responsive workstation. Now imagine what you could do with a 40 core processor. This is the kind of power that we are talking about. Now imagine, that not only is your workstation immensely powerful, but you could also offload common jobs such as running daily builds, or encoding videos to another machine (or in this case processor.

In the ASR1000 this processor is called Spinach (yellow are in the graphic above). And of course just like the cartoon, Popeye’s potential really comes to light when combined with Spinach. Spinach is a separate chip, that is used a a traffic manager. This chip handles queueing and quality of service, ensuring that the proper packets arrive at the proper time, as well as interconnecting with cryptographic offload engines so it can equally apply services to encrypted flows.

At the end of the day, the most important question is not how fast something is, or how cool it is. The question is what can it do for me? By leveraging this new architecture the ASR1000 is now able to do line rate inspection of traffic using Network Based Application Recognition (NBAR), Support 128,000 queues for deep quality of service, secure and encrypt data using zone based firewalls and embedded crypto engines, segregate traffic using MPLS, integrate advanced voice and video functionality, as well as providing fulling Netflow v9 support for all of the above. It provides all of these services in an always on solution utilizing Linux virtualization, as well as leveraging an flexible chip set architecture that allows for field programmable improvements in the future.

My hope is that after reading this article that you are in a better to understand how Cisco is leveraging open source technology and integrated circuit designs to improve the foundation of the internet. In upcoming articles I will be discussing design scenarios utilizing this features in this product, as well as highlighting other areas where Cisco is embracing both open source technology, as well as open architectures that can properly leverage projects such as Linux, Ntop, Wireshark and more. If this article has you interested in learning more about some of the technologies mentioned today, then I encourage you to check out some of the links below, or shoot me and email to be highlighted in a future readers questions article.

Learn more about Linux Kernel-based Virtual Machines

Learn more about Cisco’s ASR1000

Learn more about Cisco QuantumFlowSimilar Posts:

• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Application Extension API notes – Cisco Live 2008
• Zone based IOS firewalls
• Arista Networks – Their approach to cloud networking
• Simplifying remote site security with Cisco’s new video surveillance modules on the ISR
• Altor Virtual Network Security Analyzer (VNSA) integrated with Cisco’s Nexus 1000v for VMware

--Colin McNamara

Cisco is using Linux virtualization and 40 core CPU’s for its next generation routers

GoDaddy, one of the nations largest registrars and hosting providers is distributing bloated and possibly insecure code that will cause the average user to more then double their hosting costs. In the pages below you will learn exactly what GoDaddy is doing to your server, how their support staff will try to upsell you, and the steps you need to take to ensure proper operation of your Virtual Dedicated Server.

Background

One of my new years resolutions this year was to consolidate hosting accounts into one virtual server (hosted). I had my domains, and my old hosting with Godaddy already so it was a no brainer to try out one of GoDaddys VDS (Virtual Dedicated Servers).

I went with their 29.99 a month package, with Centos5, unlimited domains, 10Gb disk, and 256 MB of memory. This should be perfectly fine for hosting a couple MySql driven sites, and a couple gallery instances. Let me emphasise this is only handling 4 active domains, two of which only have static HTML.

Provisioning was a breeze, from order to shell account only took 4 hours. I was provided with shell access, pre-configured yum repositories, and this web control panel – simple control panel, or TurboPanel (seems to have two different names). I was able to pop into GoDaddy’s control panel interface with a direct link from their server manager console, and was setting up domains in no time. (Let me throw this caveat out though, don’t buy this product for your mom’s hosting…. the documentation is horrible, and by horrible I mean non-existant).

So I get my server all set up a couple weeks ago, transfer all my files from 2 Cups Solutions and set up my new www.colinmcnamara.com site. Things go just fine, I changed over to wordpress as a CMS and am totally thrilled. My applications and email are working perfectly. Plus, I have a shell account at GoDaddy which is a very handy thing to have as a network engineer. Things are going so well, that I give my buddy Rick a Christmas present and get ricksdavis.com and el-cinco.net for him, and host it on my GoDaddy VDS.I go ahead and purchase the domain through godaddy’s domain manager. This is obviously GoDaddy’s core competency, and goes flawlessly as usual. Next step, I go into the Turbo Panel web interface. Let me give you a little background on TurboPanel. This is the “free” equivalent “to plesk or cpanel. It is actually ok for automating your domain provisioning, though if you are a normal skill level user, I would recommend paying the extra 9.99 a month for Cpanel.

Server Error

I open up my TurboPanel interface and go to provision ricksdavis.com into the domain manager. It comes back with the least descriptive error I have ever received.

Server Error
We are sorry, the system has encountered an error while processing your request.

Home

If you continue to receive this error, please contact your system administrator.

Your URL: /domain/edit.do

Error details:

CommandFailedException: Unable to get min/max uids
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUids:825
at c.g.t.f.systems.user.LinuxUserSubsystem.getMinUid:780
at c.g.t.f.systems.user.LinuxUserSubsystem.loadUserInfo:670
at c.g.t.f.systems.user.LinuxUserSubsystem.getUserInfo:646
at c.g.t.w.actions.domain.ActionDomainEdit.process:84
at c.g.t.w.actions.AbstractSpringAction.execute:118
…
at c.g.t.w.filters.AuthorizedResourceFilter.doFilter:38
…
at c.g.t.w.filters.RequestPopulationFilter.doFilter:117
…

This is the most descriptive error ever right? it tells you what is wrong, has a link to the support system, and gives you actionable information…. I would say a resounding NO. This is a classic example of why friends don’t let friends program in Java. When I got this error last night, I was scratching my head. As an engineer, the first thing I will look at is the last change to the system. Coincidentally I had installed awstats two nights before, and looking at my change logs, I saw that I had upgraded my perl version. So, with no fast response to the support email from GoDaddy I chose to put in a server re provision request (fully automated) and restore from my backups. That process took about an hour, but afterwards I was back online with no errors. Eureka! I found it (I thought). I provisioned Rick’s domain, wordpress, gallery2 etc and then went to bed.

GoDaddy Support Response

Fast forward to this morning, and I finally recieve an email reply from godaddy support. The email is pasted below-

(I have replaced the agents name with John Doe. Tech support is a hard job and I see no reason to highlight him specifically)

Before re-provisioning the server, I went ahead and tried the old three finger salute (reboot) the error still existed. So even if this email would have came to me on time, it would not have helped.

But that is besides the point. lets dig into the solution

The agent suggested the following fix -

Restart tomcat – which I was NOT using for any of my web applications (not an ejb guy)

/etc/init.d/tomcat55 restart

Restart TurboPanel (or simple control panel now – they haven’t updated their init scripts)

/etc/init.d/turbopanel restart

GoDaddy tried to upsell me, instead of fixing their code

He gives the standard, run less stuff on your server speech (remember, I only have 4 domains on this server)

Now here is the kicker – To remedy this issue long-term, you could either setup a server with 512mb RAM, or upgrade to a Dedicated server.

What the heck is with that? I should not need an upgrade with only 4 domains on a server. Especially when in the setup the default Cpanel implies support of 30 domains. Is this Tech Support or a Sales Call?

But, since my server was working fine I don’t pay much attention to the email and move on with my life.

Fast forward an hour, and I am show Rick how to access all the features of his new site, and I figure that I need to change an email account on his domain to forward to his old account. Fine, this should take two minutes. I log onto the TurboPanel interface to put the email forward in.. and there it, a big useless error screen. What the heck is with that?

So this time I actually read the email, and try the fix. Results = nothing. GoDaddy offers shell access so I log in, run top and filter for %memory used. Low and behold, there is only 8142 bytes of memory free, and a Java process owned by root is using 300Mb of virtual memory, and 132Mb of real memory, and Tomcat is using 115Mb of virtual memory and 86Mb of real memory.

Let me translate this into English – Godaddy’s control panel application was using 218 Megabytes of the 256 Megabytes of memory I had purchased. That left me with 34 Megabytes of memory . Let me clarify this, I had paid for a virtual server with 256Mb of ram, up to 1000Mb bursted (which I think is their code for swap). I am only running 4 domains on this server, and two pop3 email servers. This should not be a problem.So what is the cause of the problem? I can sum it up, crappy Java programming. Someone decided to write this program in Java (probably easier to outsource) instead of optimising it to run on lean systems. Their code effectively takes up all the available memory. And on top of that, they are are running a webserver process as root… yes as root. It is like asking for your server to get hacked.

Now, that I am done ranting, let me highlight how to fix this problem.

If you are on Windows use the following procedure to get shell access to your GoDaddy VDS -

• you will want to download a ssh client called putty – Download Here
• Copy this file to your desktop, double click putty.exe , and you should see something like this -

• In the host name field I have www.yourdomain.com . replace yourdomain with your domain name.
• Click on the open button on the bottom right, and a shell should pop up, along with a warning that looks like this (you can click OK to the warning)

• Now skip past the linux section

If you are running Linux or Unix start here -

• Open up a command line terminal
• ssh using your godaddy simple control panel username example -

ssh your-godaddy-control-panel-username@www.yourdomain.com

• If this is your first time connecting to this server, you will be prompted to accept an unkown ssh key into known_hosts, choose yes to accept

Both Windows, Linux and Unix Continue Here –

• Enter your the password you use to access your GoDaddy Simple Control Panel Interface
• You will be presented with what looks like a DOS window, this is called a secure shell terminal. Type in the username you use to access your Godaddy Simple Control Panel and then hit enter

• Now type in the password that you use to access your Simple Control Panel Interface and hit enter

• Congratulations, if you see the window below you are now shelled into your virtual dedicated server.

Change to the Root user

• Now that you are shelled into a Linux device you need to escalate your privileges to get administrator level access. In the Unix world this user is known as Root. You can change to this user, and get full system privileges by using the following command.

su – root

• You will be presented with a password prompt, enter in the same password that you have used to log into your Simple Control Panel Interface
• You are now root, be careful with what command you enter under this user, as you can do some damage if you are careless

Clean out your servers memory

• Most recent Redhat direvatives (including Centos) utilize a tool called yum to add and remove packages. This is also true with your linux servers at GoDaddy.
• you need to install a tool called memhog, it is part of a package called numactl that is normally used to assign specific process’s to specific cpu’s in a multi-core system. We will be using it today to fix GoDaddy’s memory hogging application

yum install numactl

• choose yes to all the prompts, and numactl will automatically be downloaded and installed on your server

Stop GoDaddy Simple Control Panel, and Tomcat, and tell them not to start automatically when your server restarts.

• Tomcat is a special type of webserver for Java based applications. Godaddy uses it to run their control panel interface. 99.99999 percent of users will not need to use Tomcat. If you do need to use Tomcat then you are a technical user and will know what to do.
• In Linux, server applications are called daemons. The are executed by init scripts. We will use these scripts to turn off these server applications
• Turn off the TurboPanel daemon ( this is the process that runs your simple control panel web interface

/etc/init.d/turbopanel stop

• Next we need to turn off Tomcat

/etc/init.d/tomcat55 stop

• Now that we have these services turned off, we need to make sure that they don’t come back when we reboot the server. We can do this by using the chkconfig command.
• Stop the Simple Control Panel Interface from starting automatically by executing the following command

chkconfig turbopanel off

• Stop the Tomcat server from starting automatically by executing the following command

chkconfig tomcat55 off

Clean the mess GoDaddy made of your servers memory

• A couple steps back we installed numactl. The executable we wanted out of this package is memhog. Issue the following command to take your memory back. This command will overwrite 200 megabytes of your memory, allowing the rest of your applications to get access to that memory.

memhog 200m

• This will Clear out the memory that GoDaddy’s application took over, and allow the rest of your daemons to run fine.

Great, my server is running better now. But I want to use my Simple Control Panel Interface. How do I do that?

• Easy, all you need to do is temporarily start the turbopanel daemon. When you are done making changes, you can turn it off again

/etc/init.d/tomcat55 start

/etc/init.d/turbopanel start

• When you are done, don’t forget to turn it off

/etc/init.d/turbopanel stop

/etc/init.d/tomcat55 stop

*** UPDATE *** *** UPDATE ***

I managed to stay with Godaddy and use their Virtual Dedicated Servers for 12 months. However I have moved to a new hosting provider (rimuhosting) in the spring of 2009. My last straw with GoDaddy occurred when I noticed that even with my memory utilization under control my applications were performing poorly. I shelled into verify and found that the virtual disk I/O was severly limited.

Performance issues –

Simple command to determine directory size on GoDaddy VPS

[root@ip-72-167-15-128 home]# time du -hs
1.3G    .
real    0m49.013s
user    0m0.011s
sys    0m0.062s

Simple command to determine directory size on Rimuhosting VPS
[root@colinmcnamara home]# time du -hs
1.3G    .
real    0m0.343s
user    0m0.050s
sys    0m0.290s

Translated into english, it took 49 seconds to execute a simple command on my GoDaddy server, and it too 0.34 seconds (under 1 second) to execute the same command on my virtual server at Rimuhosting.

On top of that, I was paying $32 dollars a month to GoDaddy for a server with 256 megs of ram as well as a FTP backup account. When I created my short list of hosts, I noticed that I was not getting the best deal by staying with GoDaddy.

Pricing at Rimuhosting

• Virtual Dedicated Server 190 Megabytes RAM – $19.95
• Virtual Dedicated Server 400 Megabytes RAM – $29.95
• Virtual Dedicated Server 900 Megabytes RAM – $39.95
• Virtual Dedicated Server 1150+ Megabytes RAM – $49.95

One thing you may notice, is that for the same price as the base GoDaddy VDS ($29.95) you get 400 Megabytes of RAM instead of 256. (and they won’t put on a memory sucking turbopanel app) On top of that, things that I had to pay extra for at GoDaddy such as backup space, backup mail relays, and DNS service came bundled for free.

I made the choice to move to Rimuhosting, and the service has been superior to GoDaddy in every single way. If you get sick of GoDaddy like I did, you might want to check Rimuhosting out, I think you’ll like them.Similar Posts:

• What should I do this thanksgiving break?
• Zone based IOS firewalls
• Routers can email you when they go down
• New features in VMware 3.1
• Cisco NX-OS 4.0 | Next Generation Internet Operating System
• Identity aware networking using Cisco TrustSec

--Colin McNamara

Why GoDaddy Linux Virtual Dedicated Hosting Sucks & How to Fix It

CERTIFICATIONS / ACCREDITATIONS HELD

• CCIE – Cisco Systems Internetwork Expert #18233
• VCP – VMware Certified Professional
• CDCUCSS – Cisco Data Center Unified Computing Support Specialist
• VSP – VMware Sales Professional
• VTSP – VMware Technical Sales Professional
• TSS – Cisco Technical Solutions Specialist, Data Center
• GCIH – GIAC Certified Incident Handler
• CCVP – Cisco Certified Voice Professional
• CSNSSS – Cisco Storage Networking Solutions Support Specialist
• CSNSDS – Cisco Storage Network Solutions Design Specialist
• CADCNSS – Cisco Advanced Data Center Networking Infrastructure Support Specialist
• CCIE Storage Networking
• RHCE v4/5 – Redhat Certified Engineer #804006368822511
• RHCT v4/5 – Redhat Certified Technician #804006368822511
• EMCPA – EMC Proven Professional Associate – Information Storage and Management
• NSCA – Netscaler Certified Administrator #2005072
• NACE – Network Appliance Certified Expert #12912
• NACP – Network Appliance Certified Professional #12017 – Data Protection
• NACP – Network Appliance Certified Professional #11985 – Storage Area Network
• NACP – Network Appliance Certified Professional #12911 – High Availability

Retired Certifications

• Cisco Qualified Specialist – IP Telephony Support
• Cisco Qualified Specialist – IP Telephony Design
• Cisco Qualified Specialist – IP Telephony Operations
• Cisco Wireless LAN Design Specialist
• Cisco Wireless LAN Support Specialist

PROTOCOL PROFICIENCY

EIGRP, OSPF, RIP, BGP, MPLS,  Spanning Tree, Rapid Spanning Tree, VPC, VSS, VDC, TRILL, Fabric Path, OTV ATM, RTP, SIP, H.323, LWAPP, RADIUS, TACACS+, Ethernet, Fibre Channel, iSCSI, NFS FCIP, FCP, FSPF, NDMP 802.11a, 802.11b, 802.11g, RBE, ISDN, SNMP

Virtualization , Parallel and High Performance Compute Platforms

VMware ESX, Kernel Virtual Machine, Xen, Platform LSF, Sun Grid Engine, Hadoop

VOICE and VOICE OVER IP

CallManager, Unity, ICS7750, PBX Trunking, SRST, Active Directory Integration, Extended Services, Call Detail Recording, Automated Attendant, Extension, Mobility, Asterisk, Callware and VSR VM.

HARDWARE

Cisco Unified Computing System (UCS) 6100, 2100, 5100, Nexus 7000, Nexus 5000, Nexus 2000 and Nexus 1000v switches, Catalyst 1900-6509 switches, 1600-7500 series routers, Cisco PIX firewalls, Cisco Load Balancers, Cisco

MDS , F5 Load Balancers, Netscreen / Juniper Firewalls, Cisco VPN3000 VPN concentrators, Cisco ASA Adaptive Security Appliances, Nortel Contivity VPN Concentrators,  Aironet Access Points and Bridges, Airespace LWAPP

concentrators. 3com TotalConnect racks, Ascend dial concentrators, Netscaler Load balancers, SSL accelerators, SSL VPN concentrators. Brocade Silkworm, HP Eva Storage

NETWORK MANAGEMENT

Nagios, Cacti, NTOP, IPswitch What’s Up Gold, BIG Brother, Spectrum Network Management, Kiwi Syslog,, MRTG , HP OpenView, Cisco Secure Intrusion Detection system,

Cisco Network Based Application Recognition, Snort IDS, Netscreen Firewall Manager, Unified Compute System Manager

OPERATING SYSTEMS

Redhat, Suse and Ubuntu Linux, Windows 2000, Windows 2003, Windows 2008, Windows XP, NT4.0, BSD, Solaris, OSX

BUSINESS ENVIRONMENTS

Consulting, Valued Added Reseller, Large Enterprise, Startup, Banking, Service Provider, Software Development, Manufacturing, Military

EMPLOYMENT

6/11 -  Present , Nexus IS

Director, Data Center Practice

Responsible for got to market strategy for Nexus IS, a national Cisco DVAR.

Accomplishments

• TBD

1/07 – 6/11, ePlus Technology

Consulting Systems Engineer – Data Center (10/08 – 6/11)

Transformed ePlus western region from a #3 and #2 ranked voice and campus partner to the #1 ranked Data Center partner in Northern California

Accomplishments

• Changed regional sales focus from technology silo’s to solutions based selling covering network, systems, storage and applications under one umbrella
• Developed and deployed go to market strategy for Cisco’s Unified Computing System resulting in significant competitive advantage in the western United States.
• Deployed the first Nexus 7000/5000/2000 architecture into production securing competitive advantage across multiple verticals.

• Increased Data Center revenues year over year in the worst economy in a century.
• Attracted and retained top industry talent.
• Leveraged unique technology positioning to win multiple key global clients.
• Partnered with business units inside of Cisco, resulting in key product enhancements as well as increased revenue for both ePlus and Cisco.
• Passed multiple certifications resulting in ePlus being able to sell and install EMC Vblock.

1/07 – 6/11, ePlus Technology

Senior Systems Engineer (1/07 – 10/08)

Accelerate Technical Sales, design and implement network, storage, voice and systems solutions for ePlus Southern California customers.

Accomplishments

• Changed regional sales focus from technology silo’s to solutions based selling covering network, systems, storage and applications under one umbrella.
• Established a trend of Advanced Technology account wins.
• Accelerated ePlus’s southern California sales by providing high-end engineering support.
• Integrated MPLS service provider designs into cutting edge Enterprise and Casino Gaming solutions.
• Filled PM and lead network engineer roles for large publicly traded company data center migrations.
• Created modular Cisco design / quote format and menu based hardware and services options to address rapidly changing customer needs.

9/05 – 1/07 ID Analytics

Lead Network Engineer

Lead team of four engineers, Define network and application integration architecture for large SaaS (financial cloud) analytics deployment , Leverage networking technology to increase security and availability, and decrease development and product deployment timelines

Accomplishments

• Led team of engineers responsible for all Production and Back Office systems in 2 offices and 3 datacenters
• Designed and Implemented ID Analytics Phase2 datacenter, processing 1.8 million financial transactions daily.
• Designed and Implemented Contents Switching and SSL offloading solution, enabled non-disruptive scaling of core products
• Integrated ID Analytics product with the largest card processors in the world – Equifax, Visa, TransUnion, etc.
• Designed and integrated centralized Fiber Channel and ISCSI SAN solution, increasing application speed and decreasing production database refresh times from 4 weeks to 1 week.
• Managed and maintained over 130 terabytes of storage
• Created lights out server imaging and deployment solution for remote datacenters
• Deployed and integrated monitoring solutions utilizing open source technology
• Created user emulation probes for real time application monitoring and trending of production systems
• Worked with development and Analytics to create structured Development and QA environments
• Spearheaded project to change Analytics / Informatics environment from “unix for workgroups” to high performance computing environment (HPC)
• Provide structured documentation to US Government and Corporate auditors
• Utilized project management skills for international rollouts

2/04 – 8/2005 Openwave Systems
Senior Network Engineer, Strategic Design and Integration Group
Provide technical leadership, Define network architecture, Establish standards and technical vision. Responsible for researching, developing, and architecting technical solutions to business needs.

Accomplishments

• Designed Openwave’s new Pacific Datacenter Networks, with 900 production, and 2000 development servers.
• Designed Openwave’s Pacific Shores Campus Networks, and Showcase Datacenter.
• Responsible for hardware acquisition budget of 1.7 million dollars
• Established ISCSI IP based SAN infrastructure with DR components in 4 major datacenters worldwide
• Promoted from the ranks, moving from running our VOIP phone systems, to Network team lead, to Senior Network Engineer in the Strategic Design and Integration team.
• Active and engaged member of multiple boards covering design review, change control, and security
• Negotiated with Cisco and SBC regarding datacenter purchases saving $906,000 off list price.
• Renegotiated Cisco support saving Openwave nearly $600,000 over our three year term
• Established improved data center controls, allowing Openwave to pass Sarbanes Oxley (SOX) audits
• Wrote and ran multiple RFP, RFQ, and RFI’s
• Utilized project management skills for international rollouts
• Managed, Piloted, and Installed new wireless systems for our Customer Briefing Center
• Responsible for 6 VOIP clusters around the world
• Recipient of multiple awards recognizing dedication and quality work.
• Attended continuing training for security management (CISSP)

2/03 – 1/04 USMC Reservist activated in support of Operation Enduring Freedom
Information Services Coordinator
Implement and maintain Tactical Data Networks, Provide consulting services to hosting units. Maintain Microsoft Exchange servers in both tactical and garrison environments. Perform security audits and remediation. Train support personnel.

Accomplishments

• Performed Disaster recovery of routed ATM LANE environment for Marine Corps Air Station Yuma enabling over 3000 users to resume work (awarded the Navy and Marine Corps Achievement Medal for that event)
• Performed security audit and created a security and performance remediation plan for MCAS Yuma
• Provided project management and security audit skills to 3^rd Marine Air Wing Yuma server support teams, managed server security audit, security remediation, and SMS rollout.
• Designed and implemented Nagios network monitoring system at Marine Corps Air Station Yuma.
• Implemented Norton Antivirus server for MWSS 473
• Provided training on to data teams from MWSS 473, MCAS Yuma Station IT, and 3^rd Marine Air Wing Yuma server teams.

12/02 – 2/04 2 Cups Solutions, Pleasanton , Ca
Principal Consultant
Founded 2 Cups Solutions to provide cutting edge Voice, Data, Wireless and Security services to clients in the San Francisco bay and Fresno areas.

Accomplishments

• Implemented WAN failover solution at two City of Hayward fire stations.
• Implemented email and web solution for Express Mobile Notary.
• Developed and implemented business plan focusing on State and Local Government contracts.

2/02 – 12/02 ExtraTeam, Pleasanton , Ca
Senior Systems Engineer
Design, Installation, Configuration and Maintenance of network systems consisting of Cisco CallManager, Unity, Cisco Secure ACS, LEAP secured wireless, Aironet, Cisco routers and switches, PIX firewalls, and VPN3000 concentrators. Integrating all systems with Active Directory. Performed VOIP feasibility studies. Managed the entire business cycle including sales, design, installation, training and maintenance.

Accomplishments

• Integrated CallManager voice system with Active Directory
• Recovered a failed CallManager implementation at Phase 2 Strategies (PR firm for Logitech). Implemented CallManager with up to date hardware and software, upgraded Unity up to reasonably current levels. Brought up remote office in Phoenix utilizing SRST.
• Implemented City wide wireless network integrated with active directory for the City of Hayward
• Implemented VPN Concentrators in conjunction with multiple levels of firewalls for City of Hayward and Hayward PD to meet CLETS requirements.
• Implemented network configuration management system responsible for the city of Hayward.
• Implemented new wan for Livermore Pleasanton Fire department moving fire stations from isdn to T1 and Gigabit fiber lines in conjunction with moving the location for the network core.
• Designed and implemented IPSEC based wan for Universal life resources, allowing nationwide secure remote office connectivity while minimizing wan connection costs.
• Designed CallManager based VOIP system for a 27 site school district
• Provided emergency support to Fire and Police agencies across the bay area
• Performed security remediation for a large bay area company
• Participated in large switched network cutover from 7500 to a 6509 with flex-wan modules for Stanislaus County.
• Achieved technical certifications for ExtraTeam to become certified under both the Wireless and IP Telephony revised specifications.

7/01 – 2/02 Infobond Inc. Burlingame , Ca
Network Engineer

Responsible for engineering duties in a leadership role. Integrated legacy PBX’s using VOIP technology. Used Quality of service to ensure VOIP service levels. Support legacy voice over IP and voice over Frame Relay technologies. Upgrade from legacy voice integrations to state of the art VOIP integrations. Create project plans and act on them.

Accomplishments

• Cut over evergreen lines shipping terminal from legacy 3com equipment to VOIP enabled Cisco routers and switches. Accomplished all work during Union stand downs.
• Contracted to Openwave, Inc. to run Remote Access while the engineer was on leave. Ran Remote Access for 5 weeks, resolving DSL RLAN issues and IPSec issues, while reducing trouble ticket backload to manageable levels. Assisted other engineers when needed.
• Implemented Cisco 6509’s to replace aging core network of a Benchmark Capital (bay area investment firm).
• Diagnosed and resolved VOIP issues that were stopping call center rollouts for Embarcadero Systems (a large bay area shipping company).

03/00 – 7/01 Knapp Publishing Corporation, San Ramon, Ca
Network Systems Administrator

Responsible for day-to-day operations of e-commerce data center, and wide area networks Performed DNS changes for both internal and external networks. Designed, piloted, and implemented network changes. Installation configuration and maintenance of NT, and Windows 2k file, print, and web servers

Accomplishments

• Improved service levels from 90% to 99.99%, enhanced security and increased bandwidth were benefits derived from implementing a state-of-the-art web hosting data center
• Implemented a network monitoring system to document, report, and notify of network status.
• Designed and implemented ISDN failover of Frame-Relay Network.
• Designed, piloted, and implemented network changes.
• Replaced NT servers with Linux based servers, integrated with the Windows network

01/98 – 03/00 DKA Computers Inc. Clovis, Ca
Manager Information Services (01/99 – 03/00 )

Ran day to day operations of a large valley ISP. Worked with systems manufacturing to bundle client software with all new PC’s. Partnered with local ISP’s to provide access numbers across the valley.

Accomplishments

• Managed web development, and professional services
• Moved web hosting from IIS on Windows NT to APACHE on Linux based servers, drastically increasing site availability
• Produced a forms based web application to configure custom systems online.
• Designed and implemented an IPSec based WAN connecting 3 stores point of sales systems.
• Managed corporate office and data center relocation project.

Senior PC Service Technician (01/98 – 01/99)

Provide on call service. Staff PC help desk. Provide direct customer systems support while maximizing company revenues. Configured all servers ordered from manufacturing.

Accomplishments

• Responsible for all day to day service activities for a 13 million dollar company. Management of 4 team members. Directly responsible for customer satisfaction
• Implemented hard drive imaging system, decreasing both warranty costs and turnaround time
• Installed and configured SCO Unix reservation system for National Park service, Kings Canyon
• Deploy Citrix Winframe Systems, Windows NT 4.0 Systems
• Designed, implemented inventory tracking database, reducing required stock on hand by $40,000

MILITARY

1996 – 2004 UNITED STATES MARINE CORPS RESERVE
Have held U.S. Government security clearance – Secret

EDUCATION

Ongoing professional education

Sans CISSP + Track

University of Oklahoma extension – Fire Science

Cisco Networking AcademySimilar Posts:

• What does it take to pass the CCIE exam?
• I’ll be at Cisco Live 2008 (networkers) in Orlando all week
• About Colin McNamara
• Cisco Certified Design Expert – CCDE – officially released by Cisco
• Challenges integrating VMware into Cisco networks
• Darrel Hinshaw – New Triple CCIE [Storage]!!!!!!!

--Colin McNamara

Resume – Colin McNamara, CCIE #18233

* Solid State Drive (SSD) boot support
As initially discovered last month, VMware will make available a special version of ESX Server (mentioned with terms like ESX Lite and Embedded ESX) for OEM vendors, to be installed into bootable Solid State storage devices (flash drives, etc.). This option will allow creation of ESX Server hardware appliances for easy jumpstart, granting smaller form-factors and improved reliability.
Dell, IBM and possibly other vendors will offer this option at announcement time in Q3 2007.
* DMotion
Unofficially introduced with ESX Server 3.0.1, in its first version DMotion is a special VMotion operation only capable of moving running virtual machines from an ESX Server 2.5.x host to a new ESX Server 3.x., without shared SAN LUN mandatory requirement.
In ESX Server 3.1 this capability will be extended, allowing hot migration of running virtual machines between ESX 3.1 hosts through the Ethernet cable.
* Patch management system for host and virtual machines (Update Manager 1.0)
ESX Server 3.1 will finally introduce an automated patch management system called Update Manager. This solution will be able to update both host itself and virtual machines (both Microsoft Windows and Red Hat Enterprise Linux).
Update Manager will look for available updates from Shavlik Technologies website (a possible acquisition after IPO), and will allow VI administrators to decide which patches to deliver to virtual machines.
Before applying them, Update Manager will take a snapshot and will even rollback automatically if something goes wrong.

(this product was originally codenamed VM Integrity and its developement started more than one year ago, when virtualization.info discovered it in June 2006)
* VMware Consolidate Backup (VCB) and VMware Converter 4.0 integration
VirtualCenter 2.1 will now allow restoring VCB images with an integrated version of VMware Converter, which reaches 4.0 release number.
* Server consolidation advisor
VirtualCenter 2.1 will expose a server consolidation assistant able to analyze which physical machines should be converted in virtual ones, and where to move existing VMs among available hosts.
(note that with this feature VMware is further extending competition with PlateSpin, covering both features with PowerRecon and PowerConvert)
* Guest OS disaster recovery capability
VirtualCenter 2.1 will be able to recognize a failure inside a virtual machine and restart it through VMware HA module.
* Support for VMware Server 2.0
VirtualCenter 2.1 will be finally able to seamless manage both ESX Server and VMware Server 2.0 hosts.
* Lockdown Mode
ESX Server 3.1 will expose a new security feature to completely disable local administrative account after a VirtualCenter 2.1 takes remote control.
* Power saving capability (Distributed Power Management)
VirtualCenter 2.1 will introduce a new resources utilization analysis feature, able to verify when a physical host can be powered off, VMotion-ing its virtual machines on other hosts without impacting performances.
* Support for Cisco Discovery Protocol (CDP)
VirtualCenter 2.1 will be able to recognize and use CDP to discover physical and virtual network topologies.
It stays unconfirmed if ESX Server 3.1 will already expose new virtual network architecture, allowing 3rd party virtual switches, as it will be announced by Cisco CEO at VMworld 2007.
* Support for 10Gbit Ethernet network cards
* Support for TCP/IP Offload Engine (TOE) network cards
* Support for network load balancing algorithms
* Support for 200 hosts and 2000 virtual machines
* Support for 128GB RAM per host and for 64GB RAM per virtual machine
* Support for SATA storage devices
* Support for N_Port ID Virtualization (NPIV)
* Support for VCB over iSCSI SANs
* Support for IPv6 in virtual networking
* Support for Para-virtualization guest OSes

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• Cisco releases Nexus 1000V virtual switch for VMware
• Is your network ready for Cloud Computing with Virtual Infrastructure 4?
• Cisco EMC and VMware partneship VCE VBlocks Acadia and the Partner Ecosystem
• Interesting TechWise TV episode on virtualization
• Cool new features in 12.2(33)SXH
• VMworld 2009 Schedule

--Colin McNamara

New features in VMware 3.1

My background is all over the place. It goes from running a small ISP when I was 18 to managing international CallManager clusters, to enterprise data center design and migrations. I swear that I have tech ADD. I see something new and I am like.. oooooh shiny, lets do a sniff and see how it works.

I actually attempted the Route Switch lab twice in 2002 (still had token ring and dlsw still) but got activated for the war before I could get my number. I can tell you.. that sucked horribly.

So, why did I choose to get my Storage CCIE first instead of finishing up my Route Switch first? Well, over the past couple years I have done a significant amount of IP storage (NFS, ISCSI, ATA over Ethernet, etc) including building my own IP storage heads based on linux. I had also done a little bit of fiber channel work. I felt that I had a significant advantage compared to most network engineers in the storage networking space. And, I think more importantly, the topics that I had to study were new and fresh. Whereas when I cracked open my Route Switch books.. I honestly wasn’t to excited about it at the time ( I think I was holding a grudge from my first attempts in 2002).

The partner e-learning central portion of Cisco has an Excellent lab access to labgear.net. It gives you 2 hour segments of time with 2 mds’s (both with IPS blades), 1 2 port jbod, and 2 2 port servers. They also have introductory tutorials for most major technology segments.

I did all of those, along with the every E-learning class that was on PEC. I think the major ones were the Design, Support, and CASSI classes.I took that, sat for the design and support specialist exams, and took my written last December.

After doing all that work, I was honestly 70% there. I took a little break for january and febuary (There was no open lab dates) and then started hitting the labs I think at the end of febuary.

For the majority of my practice I used the labs available through PEC. I also was able to weasel my way into 8 hour sessions every sunday from our channel SE. (I owe him plenty of drinks at networkers this year) For those 8 hour sessions I had labs created to summarize the major technology areas and to be as evil as possible to myself.

I had one attempt which didn’t work out as well as I would have liked. In RTP the lab starts at 7:15. This means if you are from the west coast like I am you will be getting up at 3:00 am in the morning for your lab. That royally kicked my but. I diverted from my attack plan and ended up
running out of time. It was a classic example of letting the lab run you, instead of you running the lab.

Luckily a data opened up just over a month later. I spent that time just working on my speed (speed is the secret sauce). I migrated to using Fabric Manager (gui interface) instead of command line. After 2 practice runs my time had dropped 25%. I also flew out to RTP 3 days before, and made sure to relax and get on east coast time.

For my final prep I got access to iementor’s lab. Their lab is excellent. I cannot reinforce enough how much it contributed to my confidence in that final week. Roman was really cool about working with an existing candidate to get me squeezed in. They are really cool guys, and they have the only workbook on the market right now.

So, I used their lab 2 days before mine, and then just chilled the day before. The day of the lab I had 70 points by lunch, and had completed configuration of the lab by 12:30. I spent the rest of the day reviewing my configurations (found 2 errors), fixing one bug, and generally harassing the proctor to make sure I didn’t misinterpret anything.

So.. if I had it all to do over again what would I do. I would still do the PEC stuff, and continue to primarily live in the Doc CD. Though I would probably pony up the cash and buy the iementor book and more lab time with them.

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233
“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• What does it take to pass the CCIE exam?
• Fibre Channel over Ethernet is taking off
• And it begins again – On the road to my CCIE in Storage
• Are you a kick ass engineer looking to grow?
• My CCIE Storage Shopping List
• Where was Colin in 2010?

--Colin McNamara

Why was Storage Networking my first CCIE? And What did I do to prepare?

YOUSERVER=`cat /var/lib/YaST2/you/config | grep LastServer= | sed ‘s/LastServer=”//g; s/;;”//g’`
YOU64=$YOUSERVER/$PLATFORM/update/SUSE-CORE/9/rpm/x86_64
YOU32=$YOUSERVER/$PLATFORM/update/SUSE-CORE/9/rpm/i586

if [[ `uname -i` = x86_64 ]]; then
echo $YOU64/kernel-source-$KERNELVER.x86_64.rpm

else
echo $YOU32/kernel-source-$KERNELVER.i586.rpm
fi

Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• I found a great tutorial on PHP + MySQL+Flash integration
• PHP / MySQL / FLASH website integration
• Why can’t I post from docs.google.com to blogger beta?
• Study on Perspective – Set 1
• Taking a closer look
• Gall stones update – Kaiser sucks

--Colin McNamara

Why does SUSE linux make it so rough?

This is great information to have, however when you have a multi tiered application it becomes harder to get a true status of your application stack. Luckily most new applications nowadays incorporate a HTTP interfaces.
However, this interface responding to a http get request generally does not indicate full application functionality. To get that level of information out you usually need to present some authentication to the web app, execute a click through, etc.

Now this has always presented a problem for me. I can check the availability of each level of the application stack. What I can’t do is verify that the application is truly working properly. This puts a major blind spot in my monitoring strategies. Luckily I just came across a very cool application. This sweet application is webinject – http://www.webinject.org/
It can be configured to graph application performance to mrtg, or output to nagios network monitor. Below is some information about this project.

What is WebInject?
WebInject is a free tool for automated testing of web applications and services. It can be used to test individual system components that have HTTP interfaces (JSP, ASP, CGI, PHP, Servlets, HTML Forms, etc), and can be used as a test harness to create a suite of [HTTP level] automated functional, acceptance, and regression tests. A test harness, also referred to as a test driver or a test framework, allows you to run many test cases and collect/report your results. WebInject offers real-time results display and may also be used for monitoring system response times.
WebInject can be used as a complete test framework that is controlled by the WebInject User Interface (GUI). Optionally, it can be used as a standalone test runner (text/console application) which can be integrated and called from other test frameworks or applications.

Programming Language and Platforms
WebInject uses an XML API (interface) for defining and loading test cases. You can use WebInject without ever seeing it’s internal implementation (no scripting or programming necessary to use it).
WebInject is written in Perl and can run on any platform that a Perl interpreter can be installed on (MS Windows, GNU/Linux, BSD, Solaris, MAC OS, and many more). Currently, binary executables of WebInject are only available for MS Windows. If you would like to run on other platforms, you must have a Perl interpreter and run it from the Perl source code.

Test Cases
Test cases are written in XML files, using XML elements and attributes, and passed to the WebInject engine for execution against the application/service under test. This abstracts the internals of WebInject’s implementation away from the non-technical tester, while using an open architecture [written in Perl] for those that require more customization or modifications.

Results/Reporting
Result reports are generated in HTML (for viewing) and XML (for tranformation by external programs). These detailed results include pass/fail status, errors, response times, etc. Results are also displayed in a window on the User Interface if you are running the WebInject GUI, and are sent to the STDOUT channel if you are running the WebInject Engine as a standalone (console) application.

Service-Level Monitoring
HTTP response times can be collected and monitored in real-time during test execution. Timer statistics are calculated and displayed in a monitor window during runtime. When used along with gnuplot (a plotting utility), a response time graph is generated and updated in real-time as the test runs. This is used to verify responses from the web application or web service under test are within an acceptable range (to meet your SLA or quality of service criteria). This also enables WebInject to be run as a performance probe for application/service monitoring.
WebInject can also be integrated as a plugin for external monitoring systems. In this case, it is used in console mode as an intelligent test agent that returns status and response times to your external program.
For real-time monitoring of your web applications or web services, WebInject is able to run in a mode that makes it compatible with Nagios. Nagios is an open source host, service, and network monitoring program.
For graphical trending of web service-levels over a long period of time, WebInject is able to run in a mode that makes it compatible with MRTG. MRTG (Multi Router Traffic Grapher) is an open source tool for collecting, storing, and graphing time-series data.

–Colin
Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• What should I do this thanksgiving break?
• Application Extension API notes – Cisco Live 2008
• Why GoDaddy Linux Virtual Dedicated Hosting Sucks & How to Fix It
• New Buzzword Enterprise Service Bus
• RSS feeds – an intranet aggregation solution?
• PHP / MySQL / FLASH website integration

--Colin McNamara

User experience testing – enhanced

Now this system work pretty good, except when we have to do usability testing. Its just not the same on a XP image, as it is from your laptop, on wireless, at home.. etc etc. So we decide we need some sort of VPN solution into our Exchange staging environment. After not alot of thought, I decide to set up a linux PPTP server. Its a protocol that pretty much everybody can use, pretty lightweight, and free .

I have set these boxes up before, and its not terribly hard. Until I had to do it on VMware. Let me set the stage for you – Redhat AS 3.3 / VMware Esx server / VMware tools installed / 1 nic . One NIC generally isn’t best for a VPN server, so I used Virtual Center to deploy a 2nd NIC to my PPTP box.
For anyone reasonably familiar with Redhat or Manadrake, we are used to installing well known hardware and seeing Kudzu add it on boot. Not this time. I rebooted, Kudzu came.. Kudzu went. I ran Kudzu manually no dice.

I know I know, auto anything never works… So I moved on to manually defining this network adapter. The first thing to check – is it actually plugged in. LSPCI reports the adapter as plugged in, and recognises it as a pcnet32 adapter. Normally I would take this as a positive sign, but not today. I issue IFCONFIG -a , network adapter is still not found. I was definately feeling a bit frustrated at this point. To be sure it wasn’t me, I bugged a friend and a co-worker (remind you of a game show?). Anson, who runs the VMware enviornment tried removing VMware tools, adding them…. still no dice. Shad tried to help through Yahoo, although to no avail.

By this time, I am frustrated, feeling like a moron, and ready for a break.
I went to get lunch from the Cafe, and settled down to eat at my desk.
Funny thing, how when you aren’t thinking about a problem is when you normally think of the solution. Let me give you a little background. Earlier when we were troubleshooting this problem, I googled this usenet post -
http://content.ix2.net/arc/t-4236.html . This fellow describes his headaches with debian, in which he had a very similar problem.

His problem, along with mine was that the proper modules werent loading for his network devices. He had tried, along with me to add the module listing to /etc/modules.conf. The one thing however that I hadn’t tried was manually loading pcnet32 using modprobe.

needless to say, modprobe pcnet32 now sits in my rc.local file. Its lame, but it works.

–Colin
Copyright ©2008 | Colin McNamara | CCIE 18233 | All Rights Reserved”

Colin McNamara
CCIE #18233

http://www.2cups.com

“The difficult we do immediately, the impossible just takes a little longer.”Similar Posts:

• New features in VMware 3.1
• Dragged to the hospital
• Cisco releases Nexus 1000V virtual switch for VMware
• Nexus 5020 – Consolidated 10 Gig Ethernet and 4 Gig Fibre Channel
• Challenges integrating VMware into Cisco networks
• My CCIE Storage Shopping List

--Colin McNamara

Redhat AS3.3 and VMware ESX network issues