During a new server deployment I’ve come across the need to use ECMP on Linux. Turns out it’s actually really simple to configure.
What is ECMP?
ECMP, which stands for Equal-Cost Multi-Path routing is a method of network load balancing. Essentially a router connected to multiple gateways (for example multiple ISPs) can load balance outgoing connections to increase total bandwidth available.
It’s important to note that there were several breaking changes throughout the life of the Linux kernel and this feature works differently (or not at all) depending on which kernel you’re running. Cumulus Linux has a really nice write up if you’re interested in history and more information on ECMP implementation in Linux.
In short, newer kernels support more features and ECMP works better on them. At the time of writing, my router runs CentOS 7 with kernel 4.18.x.
Setting up ECMP
Let’s say that we want to load balance outgoing connectivity between two ISPs with the following setup:
- ISP 1:
- Interface: ens2f0
- ISP 2:
- Interface: ens2f1
Running these commands will replace the current default route with an ECMP one:
ip route replace default proto static scope global \ nexthop dev ens2f0 via 192.168.0.1 weight 1 \ nexthop dev ens2f1 via 192.168.10.1 weight 1 ip -6 route replace default proto static scope global \ nexthop dev ens2f0 via 2001:db8:a::1 weight 1 \ nexthop dev ens2f1 via 2001:db8:b::1 weight 1
It’s that simple. While these changes won’t survive a reboot, it’s possible (and highly recommended) to add them to if-up (or equivalent for your distribution) scripts to ensure proper working on network reloads, reboots etc.
August 17, 2019 — 12:50 pm
this is nice write up ….appricicate. ..have you added two scope /etc/iproute2/ ? do we need to change anythinng in /etc/iproute2/tables ?
August 17, 2019 — 2:52 pm
Hi, you don’t need to change anything in
April 15, 2020 — 10:08 pm
On some routers, you can configure a hashing algorithm to determine which link a particular packet will be forwarded to. Have you seen anything similar in Linux?
April 15, 2020 — 10:15 pm
Excellent question. Looking at Linux kernel docs reveals the following:
net.ipv4.fib_multipath_hash_policyis the setting you’re after.
December 29, 2022 — 5:36 pm
Looks like its not working, Everytime i do traceroute, it takes the same route again and again.
January 1, 2023 — 10:29 pm
I’d double check the output of
ip routethat the ECMP route actually exists.
One thing to keep in mind is that ECMP uses consistent hashing, so traceroute towards the same destination IP will generally only take one path (depending on the configured hash policy).