While it is possible to use a Linux based router in an internet exchange setting, there are some tweaks one has to make in order to not cause any trouble.
Let’s say the IXP facing interface is ens1f1.
We don’t want to spam the IX network with unnecessary ARP/NDP requests. Increasing the reachable time to 1080000ms or more will decrease the amount of unnecessary background noise you’re sending.
net.ipv4.neigh.ens1f1.base_reachable_time_ms = 1080000 net.ipv6.neigh.ens1f1.base_reachable_time_ms = 1080000
Using proxy ARP on an IX is completely unacceptable. It must be disabled.
net.ipv4.conf.ens1f1.proxy_arp = 0
Useful if you have multiple NICs connected to the peering network in the same system.
net.ipv4.conf.ens1f1.arp_filter = 0
Reverse path filtering (uRPF) needs to be disabled to allow for asymmetric routing, which will definitely occur with some peers.
net.ipv4.conf.ens1f1.rp_filter = 0
Any sort of automatic configurations is a big no-no, including IPv6 autoconf.
net.ipv6.conf.ens1f1.autoconf = 0
Linux’s default behaviour regarding ARP can be weird at times. With the default settings in pretty much all distros, Linux will send out ARP requests for a given IP through all interfaces (see my previous post regarding ECMP issues caused by ARP). That is not desired and would cause unnecessary noise on the peering network.
net.ipv4.conf.ens1f1.arp_ignore = 2 net.ipv4.conf.ens1f1.arp_announce = 1
We do want to send gratuitous ARP requests when any configuration changes occur though.
net.ipv4.conf.ens1f1.arp_notify = 1
ICMP Redirects are dangerous and pretty much useless. Both sending and accepting should be disabled, let’s do it for all interfaces while we’re at it.
net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv6.conf.all.accept_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv6.conf.default.accept_redirects = 0
Miscellaneous settings, which are not IX specific but do come in handy in using a Linux system as an edge/core router.
net.ipv6.conf.all.accept_ra = 0 # Enough to fit a full DFZ for (hopefully) years to come net.ipv6.route.max_size = 1048576 net.ipv4.route.max_size = 8048576 # A router should be able to forward packets :) net.ipv6.conf.all.forwarding = 1 net.ipv4.ip_forward = 1
I would highly recommend disabling any form of conntrack as well, mainly for performance reasons.
Patrick
June 6, 2024 — 10:07 am
Good and helpful settings!
With a Linux router connected to multiple IXPs or upstreams, it will sometimes happen, that its “TTL exceeded” ICMP replies for a traceroute packet seem to be originated by the wrong source IP. Example: Assume the traceroute packet of one host towards another arrives at our router by IXP A, the packet’s TTL gets 0, but the route to the originating host is via IXP B. Then the IP address of the router at IXP B would be taken as a source IP for the ICMP error packet, misleading the traceroute user, as if IXP B would be on the forward route towards its target. This can be avoided by:
net.ipv4.icmp_errors_use_inbound_ifaddr = 1