BGP in the Data Center

If you think that BGP is a Server Provider (SP) only protocol and is not your business in the data center, then better think again because BGP is coming to your territory.

Traditionally for Enterprises and DCs; BGP commonly took part at the Internet edge in multi-homing scenarios (the internet from multiple ISPs) where BGP provided such networks with better control of upstream and downstream internet traffic as well as some other extra redundancy features as compared to basic default/static routing.

With the current Cloud evolution and demands, BGP can generally be used in many different solutions in the DC domain. Some of these solutions are already up and running in some DCs and are considered to be best practices, and others are still evolving and maturing and haven’t been widely adopted yet.

In this series of posts, we will try to shed some light on a few BGP deployments within the DC domain.

1- BGP as the underlay network routing protocol to provide IP reachability in IP CLOS, wherein a spine and leaf architecture, each leaf will advertise its direct connected server IP addresses using BGP (IBGP or EBGP) just to provide pure IPv4 reachability. Traditional IGPs like OSPF can fit in here as well, however, BGP offers better scalability plus rich control over the prefixes. The IP CLOS has become the de facto method of building large DCs that is truly based on open standards and can easily scale beyond all vendors solutions limitations (CISCO FEX and Fabric path, Juniper Qfabric and VCF, etc.).

2- BGP as the DCI (Data Center Interconnect) control plan protocol. Traditionally DCs used to be interconnected with L2VPN/L2 circuits or dark fiber (if feasible). Although such interconnection is very simple and straight forward as the DCs see the DCI connection as if it was just merely a cable, but this doesn’t provide any protection from a loop taking out all of your DCs plus the complexity it introduces when you have more than 2 or 3 DCs spread across many countries. For that BGP has an interesting role to play, for that role you will use the EVPN flavor of BGP where BGP would advertise MAC address among other interesting stuff, so the control plan would be BGP EVPN which will minimize the need for flooding (unknown unicast) plus it would offer many other features like active-active forwarding, proxy ARP, etc. On the other side, and when it comes to the forwarding plane, multiple options are available; VXLAN, MPLSoGRE, NVGRE, etc.

3- Safely stretching your L2 domain in between multiple zones, entities or vendor’s solutions, and this one also rely on the EVPN flavor of BGP. Imagine a scenario where you reach the limited number of TOR that any vendor’s solution support and you want to scale more, then the vendor would propose buying more entities from the same solution so you end up with 2, 3 or may be more entities which should make you think again about loops, number of MAC addresses to be hold, etc. The same problem you might pump into when the servers team or the customer requirements ask you to segment the DC into few l2 zones, then because of VM migration or certain new application you realize that you have to stretch VLAN or group of VLANS between two different zones/segments or even DCs. In such case, BGP EVPN emerges once again to provide a solution by offering the control plane for tunneling the L2 frames over IP in between these segments/zones or solutions.

4- Last but not least, BGP can play a role in the control plane of the overlay network. This is probably a very new solution for BGP and still need some time to evolve. In this case, we will also go for the EVPN flavor of BGP, but this time the BGP sessions would be running between the VXLAN Virtual Tunnel End Points (VTEPs); either the software VTEPs or the hardware VTEPS (software and hardware VTEP is not a precise technical term, however, it is widely used). The software VTEP would be running on the server that hosts the VMs, while the hardware VTEP would be the TOR switch in any of the following scenarios:

1- The server hosting the VMs is not capable of running the VTEP (commonly because of performance issues).
2- The server is bare metal server BMS (single tenant physical server).

3- In case we need to connect the virtualized world with physical devices (physical firewalls, load-balancers, etc.)
So BGP EVPN, in this case, would handle the end-to-end reachability between the VMs, assisting VTEPs to tunnel the L2 frames over IP (most solutions would go for VXLAN, although other tunneling methods are applicable as well; MPLSoUDP, MPLSoGRE, etc.). It is important to highlight that such functionality can be provided using other protocols either vendor proprietary (VMware NSX controller) or standards based (XMPP or OVSDB for example), however, BGP might have a shot being one of the industry’s most extensible and scalable protocols (especially for the hardware-only vendors).

Does all of this make sense to you?

Not necessary, because it is a topic that spans between the SP and DC arenas, so some people might be familiar with the SP side of the story, while others with the DC side. So yes it is normal not to be 100% comfortable with any of the above. If this is the case, then please follow our discussion for each of these topics in details in this series of posts.