Hardcoded external DNS breaks internal DNS resolver about robox

[ekohl]

For me it's because Ubuntu doesn't publish libvirt images. Otherwise I'd prefer the official ones.

[ladar]

@jradmacher There are reasons for setting up known (presumably good) DNS servers, so before I get too far, is there a reason you can't override the DNS config using the Vagrantfile?

[ladar]

@jradmacher for a precise example, I'd need to know which hypervisor you're using, but you can find generic help here. For the 18.04 and 18.10 images, you need to change the 'addresses' line in /etc/netplan/01-netcfg.yaml and run netplan generate and then possibly restart networking services.

[jradmacher]

I'm using libvirt on our CI/CD server, but locally I also tested Virtualbox. Both have a DNS-Proxy, which is managed by Virtualbox.

change the 'addresses' line in /etc/netplan/01-netcfg.yaml and run netplan generate and then possibly restart networking services.

netplan generate && netplan apply to be exact. My Problem was that deleting the file does not work.
You also need to revert the changes in /etc/systemd/resolved.conf and run systemctl restart systemd-resolved.service

My debugging showed that while it is working for you, you may not have the DNS resolver you expect all the time. The resolver set by Virtualbox is still present and may get used:

systemd-resolve --status       
Global
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
                      208.67.222.222
                      192.168.121.1
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes
         DNS Servers: 4.2.2.1
                      4.2.2.2
                      208.67.220.220
                      208.67.222.222
                      192.168.121.1

I still think it would be better to not add an unexpected workaround to an image called "generic/ubuntu1804".
I know about those ISPs, which redirect you to "search pages" instead of returning NXDOMAIN, but I think this should be fixed on the host (or local network) running your VMs.

[ladar]

I just had a random thought. The roboxes are all supposed to be purging their SSHD host keys before they get shut them down, and then generating gresh 4K RSA host keys when the user, aka you,, spin them up the first time. If that's the case, you should be able to reboot the box quickly, without having to recompile the kernel...?

It was just a random thought, which might explain why your CI/CD server is slower. It's also worth confirming that you are using qemu-kvm and not the unaccelerated qemu-x86-64 runtime. The system I'm at the moment uses a relatively current QEMU binary as well, from the centos-qemu-ev repo, and not the main line CentOS/RHEL version:

[ladar@darla robox]$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.7.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[jradmacher]

Huh? I think your comment was meant to go to a different issue.

[ladar]

@jradmacher d'oh. I think your right about the wrong topic. I saw your libvirt and Virtualbox sentence, and I thought I had pulled up the issue with Fedora 29 booting slowly on libvirt. Disclaimer: I was up all night playing with the RHEL v8 beta. Hopefully I can include it in my next build cycle.

As for the DNS issue you mentioned, I setup the default DNS servers, because if I let random DNS get assigned via DHCP, I sometimes end up with servers that return responses for domains which don't exist. And that in turn breaks several of my unit tests, and the unit tests for some of my dependencies. So I went with the 'known good' by default strategy.

I see what your saying about the DHCP servers getting appended to the list, but I haven't run into the issue, and I don't know of a quick fix. The 18.04 and 18.10 configs were my first attempt at using netplan ... so I don't know all of the tricks yet.

Have you tried adding a second virtual NIC in the Vagrantfile? If you do, then eth0 should the default network config (which you can disconnect, or stick on a host only subnet), and the new eth1 interface should use the system defaults... which I presume is a vanilla DHCP configuration. At least that's how it works on Red Hat systems...

Which is why, when I'm mobile, and using a new network, or WiFi hotspot, I end up with a new network config on my host, and that config tends to use xyz as it's DNS servers (as supplied by the DHCP server). I have to leave it that way, so get I redirected to the network auth portal. As such, if I leave the DNS configuration up to the host config, I found my builds would randomly fail, and nothing is worse than compiling, and running the unit tests for magma and all her dependencies (which can take 30-90 minutes) only to find out that a handful of random tests failed. Multiply that across 10-20 different distros and it's a big pain.

All that said, I'm open to suggestions for improvement. Any ideas? Anything that won't break the 'good' by default paradigm, but make things easier for edge cases like yours?

[ladar]

@jradmacher any suggestions for a fix? I'm about going to be building a new version soon.

If I don't hear back soon, I'll proceed with the build. Based on our discussion, it sounds like solving this problem via your Vagrantfile? Or there is no solution which will improve the situation.. and if that's the case, I'll close the issue.

[jradmacher]

Currently I'm build my own box, because removing it in the Vagrantfile is a real pain. (Automated yaml editing, restarting services, making sure caches are cleared...)

IMHO the best solution would be a DNS option in vagrant, which allow setting the DNS servers used by the DNS-proxys of the various providers. It will take some time for me to follow this path and dig deeper into vagrant itself.
So go ahead with your builds, building two different boxes seems like most sane choice for now.

[ladar]

@jradmacher editing the yaml file could be tricky. If I were you I'd just overwrite the whole file using a shell script block. That's how the network.sh script does it in the first place. Something like:

sudo cat <<-EOF > /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      dhcp6: false
      optional: true
      nameservers:
        addresses: [4.2.2.1, 4.2.2.2, 208.67.220.220]
EOF

netplan generate
systemctl restart systemd-networkd.service
systemctl restart ifplugd.service

Just modify the above to suit, and place it inside your Vagrantfile, wrapped up as:

  config.vm.provision "shell", run: "always", inline: <<-SHELL
#[SCRIPT GOES HERE]
  SHELL

As for modifying the settings directly using Vagrant... I know Vagrant configures the machine networking during the provision stage. I don't know off the top of my head if it can manipulate DNS servers... and I suspect it is provider specific.

[sandervanburken]

This script, executed by the Vagrant provision step, resolved the issue in my case whereby the corporate firewall would not allow external dns traffic.

# Reset netplan config, not really needed; just to clearly indicate no fixed dns is used
tee <<EOF > /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      dhcp6: false
      optional: true
      nameservers:
        addresses: []
EOF
netplan generate

# Remove fixed DNS entries and disable DNSSEC
tee <<EOF > /etc/systemd/resolved.conf
[Resolve]
DNS=
FallbackDNS=
Domains=
#LLMNR=no
#MulticastDNS=no
DNSSEC=no
Cache=yes
DNSStubListener=yes
EOF

systemctl daemon-reload
systemctl restart systemd-resolved

[skyscooby]

Just out of curiosity what is driving people to using these boxes instead of the Official Ubuntu ones? Are those broken in ubuntu/bionic64 for example? . It's pretty clear some less than generic decisions have been made by the core developer of these ones and it's not certain we are going to get fixes in any timely manner.

[mloskot]

Here is my variant (copied from other issues here and around) of DNS fix for the generic boxes with Ubuntu:

# Apply bunch of fixes for DNS resolver on Ubuntu 18.04/19.04
# https://github.com/hashicorp/vagrant/issues/10701
# https://github.com/lavabit/robox/issues/35
sudo sed -i "s/DNS=.*/DNS=/g" /etc/systemd/resolved.conf
sudo sed -i "s/addresses\:.*/addresses: [8.8.4.4,8.8.8.8]/g" /etc/netplan/01-netcfg.yaml
sudo netplan generate
sudo netplan apply
sudo systemctl restart systemd-networkd.service systemd-resolved.service
# https://github.com/docker/libnetwork/issues/2187
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

[kawaway]

If use synced-folder with nfs, vagrant try to install nfs-common but Installation fails if resolv.conf contains an inaccessible dns server addresses. same as smb(cifs-utils), and other packages that install before provision.

Then I wrote vagrant-plugin vagrant-dnsconf.
This plugin replace dns configuration.

example

Vagrant.configure("2") do |config|

  config.dnsconf.replaces = [
          {:before => "4.2.2.1", :after => "192.168.100.10"},
          {:before => "4.2.2.2", :after => "192.168.100.11"}
  ]
  config.dnsconf.deletes = [
        "208.67.220.220"
  ]
end

[elreydetoda]

this works for the debian 9 box

sudo sed -i '/dns-.*/d' /etc/network/interfaces

[elreydetoda]

@ladar is it possible to pin this message to the issues, so people can easily find it if they have problems with dns stuff?

[elreydetoda]

Here is my variant (copied from other issues here and around) of DNS fix for the generic boxes with Ubuntu:

# Apply bunch of fixes for DNS resolver on Ubuntu 18.04/19.04
# https://github.com/hashicorp/vagrant/issues/10701
# https://github.com/lavabit/robox/issues/35
sudo sed -i "s/DNS=.*/DNS=/g" /etc/systemd/resolved.conf
sudo sed -i "s/addresses\:.*/addresses: [8.8.4.4,8.8.8.8]/g" /etc/netplan/01-netcfg.yaml
sudo netplan generate
sudo netplan apply
sudo systemctl restart systemd-networkd.service systemd-resolved.service
# https://github.com/docker/libnetwork/issues/2187
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

So @mloskot , for me this doesn't work because I block all outbound dns (except for specific devices (i.e. my dns server)), so here is my suggestion.

sudo sed -i -e '/nameservers:/d' -e '/addresses:/d' /etc/netplan/01-netcfg.yaml
sudo netplan generate && sudo netplan apply
sudo sed -i 's/^[[:alpha:]]/#&/' /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved.service

which instead of replacing it with google's dns it

1. deletes the added dns lines
2. comments all the uncommented lines in systemd-resolved service

this lets the vm default to what it pulls from dhcp (which vagrant sets up when it creates the hypervisor network). so then it just follows system defaults all the way up to my dns server for my network, which means no hosts talking direct to another dns server except mine.

I add this as an extra provisioner that is set to never run, so I can manually invoke it to fix the dns when I want to (i.e.: Vagrantfile then command)

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.provision 'fix-dns', type: "shell", run: 'never' do |script|
    script.inline = <<-SHELL
      sudo sed -i -e '/nameservers:/d' -e '/addresses:/d' /etc/netplan/01-netcfg.yaml
      sudo netplan generate && sudo netplan apply
      sudo sed -i 's/^[[:alpha:]]/#&/' /etc/systemd/resolved.conf
      sudo systemctl restart systemd-resolved.service
    SHELL
  end
end

vagrant provision --provision-with fix-dns