In the past few months, quite a few people have asked me this question: “I bought a VPS from provider X, turned up a BGP session with them and my routes are all marked unreachable”. Today’s post will hopefully shed some light on what causes this phenomenon and how to deal with it.
The Problem
You’re setting up a shiny new BGP router at a cloud provider (like VMHaus or Vultr)
protocol bgp Provider {
local as 204543;
source address 2402:28c0:100:fa10::1;
import filter {
accept;
};
export filter {
if source = RTS_STATIC then { # Only announce our routes
accept;
}
reject;
};
export limit 50; # Just in case
graceful restart on;
multihop 3;
neighbor 2402:28c0:3::179 as 136620;
}
getting ready to accept the full table and propagate it to the kernel and … uh oh. You see the following sight:
unreachable 2c0f:f140:c000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f140:d000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f5f0:c000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f5f0:d000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f7b8:c000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f7b8:d000::/36 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f850:dc01::/48 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f850:dc03::/48 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:f850:dc05::/48 dev lo proto bird metric 1024 error -113 pref medium unreachable 2c0f:fb20:d000::/36 dev lo proto bird metric 1024 error -113 pref medium
All Bird supplied routes in the routing table are marked as “unreachable”. But why’s that?
Explanation
Bird exporting unreachable routes to the kernel is a sign that Bird can’t reach the next-hop of these routes. But wait, how is that possible, when they are supplied using BGP by our provider, who surely would not feed us bad routes?
The answer is actually pretty simple. This problem occurs as the BGP session we have with the provider is “multihop” – in other words, there are multiple network hops between the BGP router and our VM. Bird can only route to the first one, and not the other ones.
Solution
As Bird can’t automagically resolve past the first hop, we have to do it manually.
The first thing we need to do is to see which hops are in the way. The way we do this is by doing a traceroute to our BGP neighbour:
# traceroute6 -n 2402:28c0:3::179 traceroute to 2402:28c0:3::179 (2402:28c0:3::179), 30 hops max, 80 byte packets 1 2402:28c0:100:fe00::1 0.374 ms 0.275 ms 0.200 ms 2 2402:28c0:100:ffff:ffff:ffff:ffff:fffd 0.280 ms 0.460 ms 0.420 ms 3 2402:28c0:3::179 0.375 ms 0.322 ms 0.335 ms
2402:28c0:100:fe00::1 is the server’s default gateway, so we needn’t worry about it. The other two hops have to be addressed though.
One of the easiest ways is to just manually define these routes in Bird:
protocol static noAnnounce {
import all;
route 2402:28c0:100:ffff:ffff:ffff:ffff:fffd/128 via 2402:28c0:100:fe00::1;
route 2402:28c0:3::179/128 via 2402:28c0:100:fe00::1;
}
We’re now telling Bird that these hops are reachable via 2402:28c0:100:fe00::1, the server’s default gateway.
Last but not least, we need to make sure we don’t export these routes to the outside world, as that could cause some problems. We can achieve this by modifying the BGP session configuration like so:
protocol bgp Provider {
local as 204543;
source address 2402:28c0:100:fa10::1;
import filter {
accept;
};
export filter {
if proto = "noAnnounce" then reject; # Do not announce local routes
if source = RTS_STATIC then { # Only announce our routes
accept;
}
reject;
};
export limit 50; # Just in case
graceful restart on;
multihop 3;
neighbor 2402:28c0:3::179 as 136620;
}
That is hopefully all that needs to be done.
Conclusion
With a simple birdc6 configure, everything should be working correctly now.
2c0f:ff98::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffa0::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffa8::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffb8::/33 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffc8::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffd0::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:ffd8::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium 2c0f:fff0::/32 via 2402:28c0:100:fe00::1 dev eth0 proto bird src 2a0d:1a40:6300::1 metric 1024 pref medium