VXLAN Tutorial

VXLAN tunnel endpoints (VTEPs)

Maybe you still remember the picture about Overlay and Underlay network above. VTEPs connect between Overlay and Underlay network and they are responsible for encapsulating frame into VXLAN packets to send across IP network (Underlay) then decapsulating when the packets leaves the VXLAN tunnel. Therefore the underlay network does not know anything about VXLAN. It treats VXLAN packets like normal packets.

VTEP has two interfaces:
+ Local LAN interface: Provides local communication through bridging for endpoints connected to VTEP
+ IP interface: The interface connects to the underlay Layer 3 network (sometimes called Transport network). The IP address on the IP interface helps uniquely identify VTEP in the network.

Remote VTEP Discovery and Address learning

VTEP learns the remote VTEPs for its VXLAN segments and remote MAC Address-to-VTEP mappings through its IP interface. Let’s see how it learns them with simplified explanation of the process:

Suppose VTEP 1 & VTEP 2 were configured with the same VXLAN VNI 100 and joined the same multicast group 239.1.1.100 that is associated with VXLAN 100. VM1 connected to the (virtual) VTEP 1, which residing in the ESXi server, wishes to communicate with a host in VTEP 2 (hardware-based switch). At this moment, VM1 does not know the MAC address of the remote host, so:

1. VM1 sends Address Resolution Protocol (ARP) packet for remote host.
2. VTEP1 encapsulates this broadcast frame into a VXLAN header with destination IP as multicast group address 239.1.1.100.
3. All the VTEPs that joined above multicast group, including VTEP 2, receives this packet. They also learn about VTEP 1 that has originated this packet.
4. VTEP 2 first looks at the encapsulation header and if the VNI matches, it removes the encapsulation header and delivers the packet to the host.
5. The remote host in VTEP 2 sends a unicast ARP reply packet to VTEP 1 since VTEP 2 learned about VTEP 1 and VM1 (including VTEP 1 IP address and VM1 MAC address). VTEP 1 learns all necessary information about VTEP 2 and HostA (including VTEP 2 IP address and Host A MAC address) in this step.
6. VTEP 1 forms the neighbor relationship with VTEP 2 for ongoing inter-communications across Layer 3 networks.

From now, when VM1 sends an Ethernet frame to HostA, VTEP 1 uses the VNI and the destination MAC (HostA MAC address) to look up in its forwarding table which VTEP to send the packet to. Then VTEP 1 adds a VXLAN header that contains the VNI to the Ethernet frame, encapsulates the frame in a Layer 3 UDP packet, and routes the packet to VTEP 2 over the Layer 3 network. VTEP2 decapsulates the original Ethernet frame and forwards it to HostA. VM1 and HostA are completely unaware of the VXLAN tunnel and the Layer 3 network between them.

VXLAN Traffic Flow

In order to fully see how VTEPs sends real traffic to the destination after learning about VTEP and destination host described above, please see the picture below:

HostA and HostB in VXLAN segment 100 communicate with each other through the VXLAN tunnel between VTEP1 and VTEP4. This example assumes that address learning has been done on both sides (as described in the section “Remote VTEP Discovery and Address learning” above), and corresponding MAC-to-VTEP mappings exist on both VTEPs. Let’s see what will happen when HostA wants to communicate with HostB.

1. When HostA sends traffic to HostB, it forms Ethernet frames with the MAC B as the destination MAC address and IP B as the destination IP address and sends them out to VTEP1.

2. VTEP1, with a mapping of MAC B to VTEP4 in its mapping table, performs VXLAN encapsulation on the packets by adding VXLAN, UDP, and outer IP address header to it. In the outer IP address header, the source IP address is the IP address of VTEP1, and the destination IP address is the IP address of VTEP4.

VTEP1 then performs an IP address lookup for the IP address of VTEP4 to resolve the next hop in the transit network and subsequently uses the MAC address of the next-hop device to further encapsulate the packets in an Ethernet frame to send to the next-hop device.

3. The packets are routed toward VTEP4 through the transport network based on their outer IP address header, which has the IP address of VTEP4 as the destination address.

4. After VTEP4 receives the packets, it strips off the outer Ethernet, IP, UDP, and VXLAN headers, and forwards the packets to HostB, based on the original destination MAC address in the Ethernet frame.

Good resource and reference: https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/200262-Configure-VxLAN-Flood-And-Learn-Using-Mu.html