VXLAN Tutorial

With the rapid growth of networking nowadays, traditional VLANs have to face some challenges. There are four main disadvantages of traditional VLANs:

+ The number of VLANs is small. A traditional VLAN ID is only 12-bit in length so it only provides 4096 VLANs. This number of VLANs is maybe enough in the old days but nowadays it is not, especially for service providers. Suppose each normal customer requires 10 VLANs then a service provider only has enough VLAN to divide for about 400 customers. If a large customer like a bank comes to you, he may require a few hundreds of VLANs. The lack of VLAN is same as the lack of IPv4 addressing space we are facing nowadays.

+ Spanning-tree blocks redundant ports to avoid loops. Another problem of traditional VLAN is it works purely on Layer 2 switch, which comes with Spanning-tree protocol (STP). STP helps block redundant links to prevent loops but they may be the links we have to hire, which cost money. Blocking them means we cannot fully use what we paid for.

+ Another problem is about limited MAC address table space. Nowadays with virtualization, each switchport may connect to a physical server which includes multiple virtual hosts inside. In other words, each physical server may contains many (virtual) MAC addresses. The burden of storing MAC addresses is not on the access-layer switches but on the distribution-layer switches as they have to remember all the MAC addresses that are stored on their connected access-layer switches.

+ The last problem we want to mention here is related to the mobility of virtualization. A key benefit of virtualization is the ability to move virtual machines (VMs) among data center servers while they are running. But to support this feature, VMs must remain in their native subnet. This guarantees network connectivity between the source and destination VM.

For example in the topology above, an administrator can easily move VM9 to VM10 (reside on the same subnet) but he cannot move VM250 to VM304 on a different subnet transparently. VXLAN can solve IP subnetting limitations, allowing the administrators to move VMs to any other servers in the data center, regardless of the data center’s subnetting scheme and we will explain it later.

What is VXLAN

VXLAN addresses the problems of traditional VLAN by the following factors:

VXLAN has a 24-bit VXLAN network identifier (VNI), which allows for up to 16 million (= 2²⁴) VXLAN segments to coexist within the same infrastructure. This surely solve the small number of traditional VLANs -> The first problem of VLAN has been resolved.

VXLAN is a virtual overlay network which runs on the top of a physical underlay network. The underlay network may use any Layer 3 routing protocol like OSPF, EIGRP, IS-IS… to route packets so no Spanning Tree Protocol is required -> One of the major benefits of VXLAN technology is that it allows creating virtual Layer 2 segments over Layer 3 routed networks.

When any connectivity issue occurs, the overlay network is still alive and unchanged, provided that the underlay network can find an alternative way to reach the destination. -> The second problem of VLAN has been resolved.

If you know how GRE tunnel works then maybe you are familiar with the term Overlay as GRE also runs on an overlay network.

It can also make good use of Equal-Cost Multi-Path (ECMP) links for load sharing and near-instant failure recovery. But ECMP is out of scope of this tutorial.

With VXLAN, our previous topology now looks like this:

To understand how VXLAN can do the magic, let’s have a closer look at the structure of a VXLAN packet.

VXLAN Packet Structure

VXLAN is often described as an overlay technology because it allows to stretch Layer 2 connections over an intervening Layer 3 network by encapsulating (tunneling) Ethernet frames in a VXLAN packet that includes IP addresses. Let’s see the structure of a VXLAN packet to understand how:

The key fields for the VXLAN packet in each of the protocol headers are:

+ Outer MAC header (14 bytes with 4 bytes optional) – Contains the MAC address of the source VTEP and the MAC address of the next-hop router. Each router along the packet’s path rewrites this header so that the source address is the router’s MAC address and the destination address is the next-hop router’s MAC address.

+ Outer IP header (20 bytes)- Contains the IP addresses of the source and destination VTEPs.
+ (Outer) UDP header (8 bytes)- Contains source and destination UDP ports:
– Source UDP port: The VXLAN protocol repurposes this standard field in a UDP packet header. Instead of using this field for the source UDP port, the protocol uses it as a numeric identifier for the particular flow between VTEPs. The VXLAN standard does not define how this number is derived, but the source VTEP usually calculates it from a hash of some combination of fields from the inner Layer 2 packet and the Layer 3 or Layer 4 headers of the original frame.
– Destination UDP port: The VXLAN UDP port. The Internet Assigned Numbers Authority (IANA) allocates port 4789 to VXLAN.

+ VXLAN header (8 bytes)- Contains the 24-bit VXLAN Network Identifiers (VNI).
+ Original Ethernet/L2 Frame – Contains the original Layer 2 Ethernet frame.

In total, VXLAN encapsulation adds between 50 and 54 bytes of additional header information to the original Ethernet frame. Because this can result in Ethernet frames that exceed the default 1514 byte MTU, best practice is to implement jumbo frames throughout the network.

In summary, we can see VXLAN packet is nothing more than a MAC-in-UDP encapsulated packet. The VXLAN header is added to the original Layer 2 frame and then placed in a UDP-IP packet. This encapsulation allows VXLAN packets to tunnel from Layer 2 network through Layer 3 network.