Software Defined Networking - the future of the data center
I've been involved in networking computers since the middle 80s. Back then it was the early versions of a solution called Ethernet from a company called 3Com. Over the years I've watched the explosive growth of this industry and the introduction of first the Hub, then the router and then the switch. By 2006 I was Sr. Director of Networking and Telecommunications for Intuit running a large multi-site shop with over 2500 network devices and a staff of 125.
Today most data center networks are defined by the Access, Aggregation and Core concept introduced by Cisco. With this concept, individual servers connected to an Access or Top of Rack (TOR) switch. The TOR switches consolidated to an End of Row (EOR) Aggregation switch which then consolidated to a large redundant set of Core switches. Using this model Cisco has become the 900 pound gorilla in the market with some 70%+ market share.
Along with this basic topology was concepts like Spanning Tree to support multi-path HA at layer 2 and then TCP/IP at layer 3. Over the years speed have increased from 10 to 100Mbps, then 1Gbps and now 10Gbps & even 40Gbps links are available.
In the mid 2000's VMware's ESX product was taking off and in a partnership with Cisco introduced the Nexus 1000v "virtual switch". The virtual switch introduced the concept of Network Function Virtualization (NFV). Previous to that introduction - all traffic from virtual machines travelled out the virtual NIC to the physical host NIC and to TOR to get to the VM running on the same machine. The vSwitch changed that - now networking traffic from VM to VM on the same host would travel through that virtual switch.
As virtualized datacenters became more mature and customers wanted to leverage new capabilities like vMotion to move VMs from one server to another - then additional networking challenges can into play. Like how do I manage subnets and how do I route traffic from a VM that was on 1 subnet on host A to a different subnet on Host B.
Additionally in many application designs there were additional networking services like firewalls and load balancers that were traditionally served by dedicated appliances. So once again - traffic would need to flow out of the VM Host through the appliance and back.
In 2007 a company called Nicira was founded. Nicira took the vSwitch concept and added additional networking services like firewall and load balancing as well as introducing the concept of separation of the networking control plane from the data plane and extending the management of the virtual network from the VM Host to the physical network via the SDN Controller concept.
In the past the network control plane was managed at the individual switch level and implemented via the switch operating system like Cisco's CatOS and NexusOS. These Network OS are based on Unix/Linux and used custom ASICs to perform data plane functions.
Nicira's capabilities was so revolutionary that VMware ended up buying them in 2012 and the resultant product is now called NSX.
At the same time, many of those data plane capabilities that used to be implemented in custom ASIC are now available in commodity components that have dramatically dropped the cost while improving the capabilities of TOR switches. Even the concept of Access, Aggregation and Core is being replaced by a Spine/Leaf topology that reduces complexity and cost while improving Host to Host communications in large scale virtual farms.
Finally the large Web Scale providers like Google - who need extremely low cost and massive scale have been driving the use of commodity hardware switches driven by open source Linux Switch OSes like Cumulus. Just like the disruption of the server market by going to an ODM model of their own design, these providers are having the ODMs build custom switches based on commodity silicon and using open source software to control their network functions.
With virtualization - the TOR switch has now effectively been moved to the host VM and Network Virtualization Overlays support server to server networking. Add centralized policy based management across multiple switches in a server farm along with their external data path switches and the ability transparently support distributed workloads and a new data center model has been born.
SDN is disruptive and still maturing - but here is my expectation. First you will see more and more "commodity" switches becoming available. By Commodity I mean an x Port switch that has NO OS on it. Just like you buy a server. You'll then have the option to install an open-source OS like Cumulus or BigStack and then use NSX or OpenStack to manage it.
This model is going to change your decision making dramatically...
First - it will drive prices down dramatically. Expect to see for example 32 port 40Gbps switches for $12 - $16K - not the $30K - $60K you see today. 48 Port 10Gbps switches will be closer to $4K - maybe even less...like $2K - $3K. Oh and no more large chassis based switches with blade modules - think fixed port.
Second - you'll need fewer switches - using a Spine/Leaf model, virtualization and modern servers you could support 48 hosts each running 100 VMs for example with a whopping total of 6 switches and still have full redundancy and 40Gbps performance. How many of you can support 4800 VMs in just two or three racks.
Third - forget older layer 2 HA and redundancy - do it at layer 3. In my experience over the years - technologies like Spanning Tree don't work when you really need them. The merge is often buggy. Most of these new open source network OS don't use it.
Finally - upgrade your skills - SDN is coming. Yes Cisco is going to say oh yes we do SDN, but not the way I described it above. Many of you have developed your careers based on Cisco technologies and certifications. But avoid being cross armed on this - over the next 5 - 6 years I expect a dramatic shift towards this model and if you don't understand it - you'll be left behind.
Today most data center networks are defined by the Access, Aggregation and Core concept introduced by Cisco. With this concept, individual servers connected to an Access or Top of Rack (TOR) switch. The TOR switches consolidated to an End of Row (EOR) Aggregation switch which then consolidated to a large redundant set of Core switches. Using this model Cisco has become the 900 pound gorilla in the market with some 70%+ market share.
Along with this basic topology was concepts like Spanning Tree to support multi-path HA at layer 2 and then TCP/IP at layer 3. Over the years speed have increased from 10 to 100Mbps, then 1Gbps and now 10Gbps & even 40Gbps links are available.
In the mid 2000's VMware's ESX product was taking off and in a partnership with Cisco introduced the Nexus 1000v "virtual switch". The virtual switch introduced the concept of Network Function Virtualization (NFV). Previous to that introduction - all traffic from virtual machines travelled out the virtual NIC to the physical host NIC and to TOR to get to the VM running on the same machine. The vSwitch changed that - now networking traffic from VM to VM on the same host would travel through that virtual switch.
As virtualized datacenters became more mature and customers wanted to leverage new capabilities like vMotion to move VMs from one server to another - then additional networking challenges can into play. Like how do I manage subnets and how do I route traffic from a VM that was on 1 subnet on host A to a different subnet on Host B.
Additionally in many application designs there were additional networking services like firewalls and load balancers that were traditionally served by dedicated appliances. So once again - traffic would need to flow out of the VM Host through the appliance and back.
In 2007 a company called Nicira was founded. Nicira took the vSwitch concept and added additional networking services like firewall and load balancing as well as introducing the concept of separation of the networking control plane from the data plane and extending the management of the virtual network from the VM Host to the physical network via the SDN Controller concept.
In the past the network control plane was managed at the individual switch level and implemented via the switch operating system like Cisco's CatOS and NexusOS. These Network OS are based on Unix/Linux and used custom ASICs to perform data plane functions.
Nicira's capabilities was so revolutionary that VMware ended up buying them in 2012 and the resultant product is now called NSX.
At the same time, many of those data plane capabilities that used to be implemented in custom ASIC are now available in commodity components that have dramatically dropped the cost while improving the capabilities of TOR switches. Even the concept of Access, Aggregation and Core is being replaced by a Spine/Leaf topology that reduces complexity and cost while improving Host to Host communications in large scale virtual farms.
Finally the large Web Scale providers like Google - who need extremely low cost and massive scale have been driving the use of commodity hardware switches driven by open source Linux Switch OSes like Cumulus. Just like the disruption of the server market by going to an ODM model of their own design, these providers are having the ODMs build custom switches based on commodity silicon and using open source software to control their network functions.
With virtualization - the TOR switch has now effectively been moved to the host VM and Network Virtualization Overlays support server to server networking. Add centralized policy based management across multiple switches in a server farm along with their external data path switches and the ability transparently support distributed workloads and a new data center model has been born.
SDN is disruptive and still maturing - but here is my expectation. First you will see more and more "commodity" switches becoming available. By Commodity I mean an x Port switch that has NO OS on it. Just like you buy a server. You'll then have the option to install an open-source OS like Cumulus or BigStack and then use NSX or OpenStack to manage it.
This model is going to change your decision making dramatically...
First - it will drive prices down dramatically. Expect to see for example 32 port 40Gbps switches for $12 - $16K - not the $30K - $60K you see today. 48 Port 10Gbps switches will be closer to $4K - maybe even less...like $2K - $3K. Oh and no more large chassis based switches with blade modules - think fixed port.
Second - you'll need fewer switches - using a Spine/Leaf model, virtualization and modern servers you could support 48 hosts each running 100 VMs for example with a whopping total of 6 switches and still have full redundancy and 40Gbps performance. How many of you can support 4800 VMs in just two or three racks.
Third - forget older layer 2 HA and redundancy - do it at layer 3. In my experience over the years - technologies like Spanning Tree don't work when you really need them. The merge is often buggy. Most of these new open source network OS don't use it.
Finally - upgrade your skills - SDN is coming. Yes Cisco is going to say oh yes we do SDN, but not the way I described it above. Many of you have developed your careers based on Cisco technologies and certifications. But avoid being cross armed on this - over the next 5 - 6 years I expect a dramatic shift towards this model and if you don't understand it - you'll be left behind.
Comments
Post a Comment