ECMP on Linux

During a new server deployment I’ve come across the need to use ECMP on Linux. Turns out it’s actually really simple to configure.

What is ECMP?

ECMP, which stands for Equal-Cost Multi-Path routing is a method of network load balancing. Essentially a router connected to multiple gateways (for example multiple ISPs) can load balance outgoing connections to increase total bandwidth available.

It’s important to note that there were several breaking changes throughout the life of the Linux kernel and this feature works differently (or not at all) depending on which kernel you’re running. Cumulus Linux has a really nice write up if you’re interested in history and more information on ECMP implementation in Linux.
In short, newer kernels support more features and ECMP works better on them. At the time of writing, my router runs CentOS 7 with kernel 4.18.x.

Setting up ECMP

Let’s say that we want to load balance outgoing connectivity between two ISPs with the following setup:

  • ISP 1:
    • Gateways:
      • 192.168.0.1
      • 2001:db8:a::1
    • Interface: ens2f0
  • ISP 2:
    • Gateways:
      • 192.168.10.1
      • 2001:db8:b::1
    • Interface: ens2f1

 

Running these commands will replace the current default route with an ECMP one:

ip route replace default proto static scope global \
    nexthop dev ens2f0 via 192.168.0.1 weight 1 \
    nexthop dev ens2f1 via 192.168.10.1 weight 1

ip -6 route replace default proto static scope global \
    nexthop dev ens2f0 via 2001:db8:a::1 weight 1 \
    nexthop dev ens2f1 via 2001:db8:b::1 weight 1

It’s that simple. While these changes won’t survive a reboot, it’s possible (and highly recommended) to add them to if-up (or equivalent for your distribution) scripts to ensure proper working on network reloads, reboots etc.

Keep in mind additional configuration might be required if both gateways are present in the same subnet in order to achieve correct load balancing over both network ports, related to the standard Linux ARP behavior.

Bird ECMP configuration

Assuming we want to originate an ECMP route using a static protocol, the following configuration will achieve that:

protocol static {
    route 0.0.0.0/0
        via 192.168.0.1
        via 192.168.10.1;
}

To get ECMP routes exported to the kernel, the merge paths property also has to be enabled (default is off).

protocol kernel {
    ipv4 { export all; import none; };
    merge paths on;
}

The aforementioned syntax is valid for Bird 2.

Netplan ECMP configuration

As of August 2023, Netplan unfortunately does not support configuring ECMP. While there is an open feature request on the netplan bug tracker, it does not seem to be getting much attention.

8 Comments

Add yours

  1. this is nice write up ….appricicate. ..have you added two scope /etc/iproute2/ ? do we need to change anythinng in /etc/iproute2/tables ?

  2. On some routers, you can configure a hashing algorithm to determine which link a particular packet will be forwarded to. Have you seen anything similar in Linux?

    • Excellent question. Looking at Linux kernel docs reveals the following:

      fib_multipath_hash_policy – INTEGER
      Controls which hash policy to use for multipath routes.
      Default: 0 (Layer 3)
      Possible values:
      0 – Layer 3 (source and destination addresses plus flow label)
      1 – Layer 4 (standard 5-tuple)
      2 – Layer 3 or inner Layer 3 if present

      Seems that’s net.ipv4.fib_multipath_hash_policy is the setting you’re after.

  3. Looks like its not working, Everytime i do traceroute, it takes the same route again and again.

    • I’d double check the output of ip route that the ECMP route actually exists.
      One thing to keep in mind is that ECMP uses consistent hashing, so traceroute towards the same destination IP will generally only take one path (depending on the configured hash policy).

Leave a Reply...