ECMP on Linux

During a new server deployment I’ve come across the need to use ECMP on Linux. Turns out it’s actually really simple to configure.

What is ECMP?

ECMP, which stands for Equal-Cost Multi-Path routing is a method of network load balancing. Essentially a router connected to multiple gateways (for example multiple ISPs) can load balance outgoing connections to increase total bandwidth available.

It’s important to note that there were several breaking changes throughout the life of the Linux kernel and this feature works differently (or not at all) depending on which kernel you’re running. Cumulus Linux has a really nice write up if you’re interested in history and more information on ECMP implementation in Linux.
In short, newer kernels support more features and ECMP works better on them. At the time of writing, my router runs CentOS 7 with kernel 4.18.x.

Setting up ECMP

Let’s say that we want to load balance outgoing connectivity between two ISPs with the following setup:

• ISP 1:
  • Gateways:
    • 192.168.0.1
    • 2001:db8:a::1
  • Interface: ens2f0
• ISP 2:
  • Gateways:
    • 192.168.10.1
    • 2001:db8:b::1
  • Interface: ens2f1

Running these commands will replace the current default route with an ECMP one:

ip route replace default proto static scope global \
    nexthop dev ens2f0 via 192.168.0.1 weight 1 \
    nexthop dev ens2f1 via 192.168.10.1 weight 1

ip -6 route replace default proto static scope global \
    nexthop dev ens2f0 via 2001:db8:a::1 weight 1 \
    nexthop dev ens2f1 via 2001:db8:b::1 weight 1

It’s that simple. While these changes won’t survive a reboot, it’s possible (and highly recommended) to add them to if-up (or equivalent for your distribution) scripts to ensure proper working on network reloads, reboots etc.

Keep in mind additional configuration might be required if both gateways are present in the same subnet in order to achieve correct load balancing over both network ports, related to the standard Linux ARP behavior.

Bird ECMP configuration

Assuming we want to originate an ECMP route using a static protocol, the following configuration will achieve that:

protocol static {
    route 0.0.0.0/0
        via 192.168.0.1
        via 192.168.10.1;
}

To get ECMP routes exported to the kernel, the merge paths property also has to be enabled (default is off).

protocol kernel {
    ipv4 { export all; import none; };
    merge paths on;
}

The aforementioned syntax is valid for Bird 2.

Netplan ECMP configuration

As of August 2023, Netplan unfortunately does not support configuring ECMP. While there is an open feature request on the netplan bug tracker, it does not seem to be getting much attention.