ECMP on Linux

During a new server deployment I’ve come across the need to use ECMP on Linux. Turns out it’s actually really simple to configure.

What is ECMP?

ECMP, which stands for Equal-Cost Multi-Path routing is a method of network load balancing. Essentially a router connected to multiple gateways (for example multiple ISPs) can load balance outgoing connections to increase total bandwidth available.

It’s important to note that there were several breaking changes throughout the life of the Linux kernel and this feature works differently (or not at all) depending on which kernel you’re running. Cumulus Linux has a really nice write up if you’re interested in history and more information on ECMP implementation in Linux.
In short, newer kernels support more features and ECMP works better on them. At the time of writing, my router runs CentOS 7 with kernel 4.18.x.

Setting up ECMP

Let’s say that we want to load balance outgoing connectivity between two ISPs with the following setup:

  • ISP 1:
    • Gateways:
      • 2001:db8:a::1
    • Interface: ens2f0
  • ISP 2:
    • Gateways:
      • 2001:db8:b::1
    • Interface: ens2f1


Running these commands will replace the current default route with an ECMP one:

ip route replace default proto static scope global \
    nexthop dev ens2f0 via weight 1 \
    nexthop dev ens2f1 via weight 1

ip -6 route replace default proto static scope global \
    nexthop dev ens2f0 via 2001:db8:a::1 weight 1 \
    nexthop dev ens2f1 via 2001:db8:b::1 weight 1

It’s that simple. While these changes won’t survive a reboot, it’s possible (and highly recommended) to add them to if-up (or equivalent for your distribution) scripts to ensure proper working on network reloads, reboots etc.


Add yours

  1. this is nice write up ….appricicate. ..have you added two scope /etc/iproute2/ ? do we need to change anythinng in /etc/iproute2/tables ?

  2. On some routers, you can configure a hashing algorithm to determine which link a particular packet will be forwarded to. Have you seen anything similar in Linux?

    • Excellent question. Looking at Linux kernel docs reveals the following:

      fib_multipath_hash_policy – INTEGER
      Controls which hash policy to use for multipath routes.
      Default: 0 (Layer 3)
      Possible values:
      0 – Layer 3 (source and destination addresses plus flow label)
      1 – Layer 4 (standard 5-tuple)
      2 – Layer 3 or inner Layer 3 if present

      Seems that’s net.ipv4.fib_multipath_hash_policy is the setting you’re after.

Leave a Reply...

%d bloggers like this: