ECMP on Linux

During a new server deployment I’ve come across the need to use ECMP on Linux. Turns out it’s actually really simple to configure.

What is ECMP?

ECMP, which stands for Equal-Cost Multi-Path routing is a method of network load balancing. Essentially a router connected to multiple gateways (for example multiple ISPs) can load balance outgoing connections to increase total bandwidth available.

It’s important to note that there were several breaking changes throughout the life of the Linux kernel and this feature works differently (or not at all) depending on which kernel you’re running. Cumulus Linux has a really nice write up if you’re interested in history and more information on ECMP implementation in Linux.
In short, newer kernels support more features and ECMP works better on them. At the time of writing, my router runs CentOS 7 with kernel 4.18.x.

Setting up ECMP

Let’s say that we want to load balance outgoing connectivity between two ISPs with the following setup:

  • ISP 1:
    • Gateways:
      • 192.168.0.1
      • 2001:db8:a::1
    • Interface: ens2f0
  • ISP 2:
    • Gateways:
      • 192.168.10.1
      • 2001:db8:b::1
    • Interface: ens2f1

 

Running these commands will replace the current default route with an ECMP one:

ip route replace default proto static scope global \
    nexthop dev ens2f0 via 192.168.0.1 weight 1 \
    nexthop dev ens2f1 via 192.168.10.1 weight 1

ip -6 route replace default proto static scope global \
    nexthop dev ens2f0 via 2001:db8:a::1 weight 1 \
    nexthop dev ens2f1 via 2001:db8:b::1 weight 1

It’s that simple. While these changes won’t survive a reboot, it’s possible (and highly recommended) to add them to if-up (or equivalent for your distribution) scripts to ensure proper working on network reloads, reboots etc.

2 Comments

  1. this is nice write up ….appricicate. ..have you added two scope /etc/iproute2/ ? do we need to change anythinng in /etc/iproute2/tables ?

Leave a Reply...