This just took me a little longer to figure out than I’d like to admit.
TL;DR: Stick this in your hosts file:
127.0.0.1 localhost.aviatrix.comWorkings Out
Every time I tried to start Aviatrix I’d get this error:
So, I tried stracing it to see what’s up. Run the strace, search the output for the error string and see what’s on the immediately-preceeding lines:
strace -ff -e trace=network /usr/bin/AVPNC_bin/AVPNC 2>&1
[...]
[pid 20708] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 31
[pid 20708] setsockopt(31, SOL_IP, IP_RECVERR, [1], 4) = 0
[pid 20708] connect(31, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 20708] sendto(31, "\34\10\1\0\0\1\0\0\0\0\0\0\tlocalhost\10aviatrix\3"..., 40, MSG_NOSIGNAL, NULL, 0) = 40
[pid 20708] recvfrom(31, "\34\10\201\200\0\1\0\0\0\0\0\0\tlocalhost\10aviatrix\3"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28 => 16]) = 40
[pid 20708] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 31
[pid 20708] setsockopt(31, SOL_IP, IP_RECVERR, [1], 4) = 0
[pid 20708] connect(31, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 20708] sendto(31, "\314\224\1\0\0\1\0\0\0\0\0\0\tlocalhost\10aviatrix\3"..., 50, MSG_NOSIGNAL, NULL, 0) = 50
[pid 20708] recvfrom(31, "\314\224\201\203\0\1\0\0\0\1\0\0\tlocalhost\10aviatrix\3"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, [28 => 16]) = 92
[Errno -5] No address associated with hostname
[...]So it’s trying to send a DNS request to 127.0.0.53, and presumably finding that unreachable?
It turns out that this is the address that systemd-resolved listens on:
$ netstat -tunlap | grep 127.0.0.53
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 657/systemd-resolve
udp 0 0 127.0.0.53:53 0.0.0.0:* 657/systemd-resolve
But I can reach that and query that:
$ dig +short @127.0.0.53 bbc.co.uk
151.101.192.81
151.101.64.81
151.101.0.81
151.101.128.81So it’s likely that what’s happening here is Aviatrix is trying to resolve some name that’s not known to systemd-resolved, but it’s not immediately clear from the strace or the error messages what that name is.
So let’s dump the query and grab a string out of it:
$ sudo tcpdump -n -t udp port 53
[sudo] password for avi:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wlp0s20f3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
IP 192.168.0.26.41750 > 192.168.0.1.53: 44084+ A? api.snapcraft.io. (34)
IP 192.168.0.26.34737 > 192.168.0.1.53: 5327+ AAAA? api.snapcraft.io. (34)
IP 192.168.0.1.53 > 192.168.0.26.34737: 5327 0/1/0 (98)
IP 192.168.0.1.53 > 192.168.0.26.41750: 44084 4/0/0 A 185.125.188.58, A 185.125.188.54, A 185.125.188.55, A 185.125.188.59 (98)
IP 192.168.0.26.44166 > 192.168.0.1.53: 51169+ A? localhost.aviatrix.com. (40)
IP 192.168.0.1.53 > 192.168.0.26.44166: 51169 0/0/0 (40)
IP 192.168.0.26.41438 > 192.168.0.1.53: 37626+ A? localhost.aviatrix.com.fritz.box. (50)
IP 192.168.0.1.53 > 192.168.0.26.41438: 37626 NXDomain* 0/1/0 (92)
[...]Lots more follows, but the likely looking queries are for localhost.aviatrix.com and, presumably once that’s failed localhost.aviatrix.com.fritz.box – my router is a Fritz Box, so the searchdomain is fritz.box).
systemd-resolved does not have a record with those names, so I think this is likely the problem:
$ dig +short @127.0.0.53 localhost.aviatrix.com
$So, stick that into /etc/hosts and it all works! Exactly how Aviatrix expected this to be here (and how this worked for the many months up until now) I’m not really sure.