Proxmox SDN with VXLAN Overlay: Setting Up Multi-Tenant Networks Cleanly

VLANs have been the workhorse of network segmentation for more than twenty years. They work, they are understood everywhere, and they are perfectly sufficient for most mid-sized environments. But as soon as multiple tenants need to run on the same cluster, several sites have to be connected at Layer 2, or the 4094 usable VLAN IDs become scarce, the classic model hits its limits. This is where VXLAN comes in — and with Proxmox SDN, an overlay network becomes a tool that can be operated without a dedicated network specialist team.

This article shows how VXLAN works concretely in Proxmox VE 9, when a switch from pure VLAN mode pays off, and where the typical pitfalls are in production. The fundamentals around zones, VNets and subnets have already been covered in our article on Proxmox SDN as a concept — here, we focus on the VXLAN zone and its practical use.

Why VXLAN When VLANs Still Work?

VLANs operate at Layer 2 and are tied to the physical switch infrastructure. Every trunk port has to carry the relevant VLAN IDs, every switch involved has to know about them, and every change touches at least one network admin. For ten or twenty networks, this is unproblematic — for a hoster with fifty customers, a software house with twenty test environments, or a cluster that spans two data centers, it becomes tedious.

VXLAN (Virtual Extensible LAN) solves three concrete problems at once:

• Instead of a 12-bit VLAN ID, 24 bits are available as a VNI (VXLAN Network Identifier) — roughly 16 million networks instead of 4094.
• Overlay packets are encapsulated in UDP and flow over the regular IP infrastructure. The physical switches only need to route IP, not to know every customer VLAN ID.
• An overlay can be spanned across sites as long as there is IP connectivity between nodes. This makes geo-redundant clusters and stretched setups technically clean.

The price for this is overhead: VXLAN adds a 50-byte header to every frame. Without MTU adjustment in the underlay, you pay for this in fragmented packets and noticeable performance loss. We come back to this below.

Proxmox SDN VXLAN in Detail: Zone, VNet, Subnet

The SDN hierarchy in Proxmox VE 9 is three-tiered and stays the same with VXLAN: the zone as a technology container, the VNet as a virtual network within the zone, and the subnet as the IP configuration of that VNet. For a VXLAN zone, an additional property comes in — the list of peers, meaning the IP addresses of all cluster nodes participating in this overlay.

A typical setup for three nodes looks like this:

Datacenter -> SDN -> Zones -> Add -> VXLAN
  ID:        vx-tenant-a
  Peers:     10.10.0.11, 10.10.0.12, 10.10.0.13
  MTU:       1450   (with physical MTU 1500)

Then you create the individual virtual networks under VNets. Each VNet gets its own VNI, which acts as the separator between overlays. Two VNets with different VNIs in the same zone are completely isolated from each other, even though they share the same physical infrastructure.

VNets -> Add
  ID:   customer-a-prod
  Zone: vx-tenant-a
  Tag:  10001        (VNI)

Finally, optionally one subnet per VNet with CIDR and gateway. If you want DHCP for the VMs in the overlay, you enable the dnsmasq service per subnet — Proxmox then manages it automatically on the involved nodes.

After an Apply at the datacenter level, Proxmox creates the appropriate Linux bridges and VXLAN interfaces on each node. A VM is no longer attached directly to vmbr0, but to the new customer-a-prod bridge. Everything else happens transparently in the background.

Multi-Tenant Separation: One Cluster, Many Customers

The classic use case for VXLAN in mid-sized environments is hosting multiple tenants on the same Proxmox cluster. Each customer gets one or more VNIs that have no connection to each other — even if the VMs run on the same hypervisor. Isolation happens at the network level and is independent of firewall rules.

A workable naming scheme makes operations much easier. In projects, we typically use:

• Zone ID: vx-<short> for the actual VXLAN zone per site or tenant group
• VNet ID: <customer>-<purpose>, for example miller-dmz or smith-prod
• VNI allocation: in blocks per customer, e.g. 10001–10099 for customer A, 10100–10199 for customer B

This separation has three consequences in operations:

• A customer’s VMs do not see other customers’ broadcasts. ARP scans or stray DHCP servers stay within their own VNet.
• Live migration between nodes works without network reconfiguration because the VNet has the same name everywhere.
• When decommissioning a customer, it is enough to remove their VNet in the GUI. No switch configuration, no VLAN ID maintenance.

Important: this isolation is a network isolation, not full tenant isolation. If you need true multi-tenant security, combine it with Proxmox pools, separated users, and — for high requirements — physically separated clusters.

MTU, Underlay, and the Pitfalls Nobody Mentions

The most common reason why VXLAN setups behave strangely in the first attempt is MTU. VXLAN adds 50 bytes of overhead: 14 bytes Ethernet, 20 bytes IP, 8 bytes UDP, 8 bytes VXLAN header. With a physical MTU of 1500, that leaves 1450 bytes for the overlay frame. This MTU must be entered exactly like that in the VXLAN zone, otherwise large packets get fragmented or silently dropped.

A much more elegant path is jumbo frames in the underlay. If the physical network supports MTU 9000, you configure the VXLAN zone with MTU 8950 and get a full 8800 bytes for payload in the overlay. This pays off especially for storage and backup traffic. How we lay out our physical cluster networks for this is described in our article on planning the Proxmox cluster network properly.

Two more points matter in practice:

• Multicast is optional, but not trivial. For standard VXLAN, Proxmox uses what is called head-end replication mode — each node sends broadcast and multicast packets as unicast to all other peers. This works without any switch configuration but does not scale forever. From about a dozen nodes upwards, a true multicast underlay becomes interesting.
• Firewall and hairpinning. If two VNets need to talk to each other via an external firewall, traffic has to leave the overlay. This works via a Proxmox firewall VM with one leg in each VNet, or via a physical firewall with appropriate routing configuration.

EVPN: When VXLAN Alone No Longer Suffices

For isolated overlays, the standard VXLAN zone is enough. But as soon as routing between VNets is required, multiple sites are involved, or MAC addresses have to be managed centrally, EVPN (Ethernet VPN) enters the picture. Proxmox SDN has supported EVPN productively since version 7 and ships with all required components — FRRouting as the controller and BGP as the signaling protocol.

The most important differences in practice:

• Instead of head-end replication, the nodes exchange MAC and IP information via BGP. This reduces broadcast traffic and scales much better.
• An EVPN zone can carry multiple VRFs (Virtual Routing and Forwarding instances). Routing tables can therefore be separated per tenant, while all overlays use the same underlay.
• Between sites, EVPN works cleanly over Layer-3 links. Anyone building a stretched cluster or a geo-redundant setup will not get around EVPN in the medium term.

The price: configuration is noticeably more complex. ASN allocation, route-reflector design, VRF mapping — these are topics where things go wrong quickly without prior experience. In our projects, we use EVPN specifically where it is truly needed and advise against the “EVPN by default” approach.

When Is the Effort Worth It?

Not every environment needs VXLAN. A three-node cluster with five VLANs for production, backup, management, storage, and testing runs much more simply and performantly with classic VLANs on a configured trunk port. VXLAN unfolds its value in specific constellations:

• Multiple tenants on the same cluster where network isolation is a customer requirement
• Many short-lived networks — for example in software development with automated test environments
• Cross-site clusters that need Layer-2 continuity
• Environments where VLAN IDs are becoming scarce or switch configuration has become the bottleneck

For classic mid-market virtualization with a single data center and a manageable number of networks, the SDN VXLAN path tends to be overkill — clean VLAN separation is leaner and lower-maintenance there. If you are unsure which path best fits your own cluster, we are happy to walk through the concrete topology in an IT service consultation.

FAQ on Proxmox SDN with VXLAN Overlay

Do I need special switches to use VXLAN in Proxmox?

No. Since VXLAN operates at IP level, any switch that connects the nodes in a common IP segment is sufficient. Neither VXLAN hardware offload nor multicast configuration is strictly required. Switches with VXLAN and BGP support become interesting only for larger clusters or when deploying EVPN with many routes.

What distinguishes VXLAN from a VLAN trunk in everyday operations?

In day-to-day operations, you mostly feel the difference when making changes. Creating a new network with VLANs means: switch configuration, trunk port extension, bridge creation on the node. With Proxmox SDN and VXLAN, it is three clicks in the web interface, and all nodes pick up the change automatically. With stable, rarely changed networks the difference is small; with many tenants it is enormous.

Do I lose performance when switching to VXLAN?

Due to the extra header and encapsulation, VXLAN costs some CPU time on the nodes. Modern server CPUs and network cards with VXLAN offload capability reduce this significantly. The decisive factor is MTU configuration: set correctly, the overhead is barely noticeable in production. Set incorrectly — e.g. VXLAN MTU 1500 with underlay MTU 1500 — fragmentation occurs with visible performance problems.

Can I run VLANs and VXLAN in parallel?

Yes, in fact that is the rule. Management network, Corosync, storage, and backup typically stay on classic VLANs or dedicated physical interfaces, while VXLAN is used only for the VM networks. Proxmox separates this cleanly: SDN zones are independent of the physical bridges, which can still be used.

How does VXLAN in Proxmox relate to TrueNAS and NFS storage?

For performance and stability reasons, storage traffic does not belong in the overlay. NFS between Proxmox nodes and a TrueNAS system always runs on a dedicated storage VLAN on separate hardware or at least a separate interface. The VXLAN isolation is intended for VM-to-VM traffic, not for hypervisor-to-storage. Suitable hardware can be put together in our TrueNAS configurator.

What happens to the overlay in case of a node failure?

The VXLAN configuration is part of the Proxmox cluster state and remains available unchanged on all surviving nodes after a failure. VMs can be moved to other nodes via HA without any network reconfiguration — the VNet already exists there. The only important thing is that the zone’s peer list contains all potential target nodes.