TypeError: Cannot read property 'length' of null about docker-dns

[bnfinet]

Thanks for the report(s)!

Could you please add the --privileged=true arg to the docker run and see if that makes the difference?

docker run -d -t   \ 
    --privileged=true \
    --name docker-dns -p 172.17.42.1:53:53/udp \
    -v /var/run/docker.sock:/var/run/docker.sock  \
    -v /home/cristian/.docker-dns/docker-dns/docker/log:/var/log/supervisor \
    -v /home/cristian/.docker-dns/docker-dns/docker/./config/config.js:/opt/docker-dns/config/config.js  \
     bfoote/docker-dns

Thanks!

[CristianCantoro]

Hi @bnfinet,

that seems to make some difference, here's what I get:

[~/.docker-dns/docker-dns]$ docker run -d -t \
>     --privileged=true \
>     --name docker-dns -p 172.17.42.1:53:53/udp \
>     -v /var/run/docker.sock:/var/run/docker.sock  \
>     -v /home/cristian/.docker-dns/docker-dns/docker/log:/var/log/supervisor \
>     -v /home/cristian/.docker-dns/docker-dns/docker/./config/config.js:/opt/docker-dns/config/config.js  \
>      bfoote/docker-dns
83998f152f1001ce41fce0772c5da78d65e9ae4504c00019f7d0b6d700b834f1

and

[~/.docker-dns/docker-dns]$ docker ps
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS                    NAMES
83998f152f10        bfoote/docker-dns:latest   "supervisord -n"    6 seconds ago       Up 5 seconds        172.17.42.1:53->53/udp   docker-dns

Running the two commands you suggest to run in the helper text of your run_docker.sh script I get:

[~/.docker-dns/docker-dns]$  dig -t SRV \* @172.17.42.1 
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.9.5-4.3ubuntu0.1-Ubuntu <<>> -t SRV * @172.17.42.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44135
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;*.             IN  SRV

;; ANSWER SECTION:
*.          5   IN  SRV 0 10 53 83998f152f10.local.dockerA.docker.local.
*.          5   IN  SRV 0 10 53 83998f152f10.public.dockerA.docker.local.

;; Query time: 7 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Mon Dec 15 09:56:35 CET 2014
;; MSG SIZE  rcvd: 140

and

[~/.docker-dns/docker-dns]$ dig -t SRV _domain._udp\* @172.17.42.1                                                                                                                           [master] 
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.9.5-4.3ubuntu0.1-Ubuntu <<>> -t SRV _domain._udp* @172.17.42.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4162
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;_domain._udp*.         IN  SRV

;; ANSWER SECTION:
_domain._udp*.      5   IN  SRV 0 10 53 83998f152f10.local.dockerA.docker.local.
_domain._udp*.      5   IN  SRV 0 10 53 83998f152f10.public.dockerA.docker.local.

;; Query time: 0 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Mon Dec 15 09:59:11 CET 2014
;; MSG SIZE  rcvd: 176

The problem then, as I see it, is the following:

$ ping 83998f152f10.public.dockerA.docker.local
ping: unknown host 83998f152f10.public.dockerA.docker.local

and if I start some containers I cannot ping them. For example:

$ docker ps
CONTAINER ID        IMAGE        COMMAND        CREATED        STATUS        PORTS NAMES
e271fc9e7adc        cristiancantoro/hadoop-mrjob:latest   "/etc/bootstrap.sh -   8 minutes ago       Up 8 minutes        50010/tcp, 50020/tcp, 8030/tcp, 50070/tcp, 50075/tcp, 22/tcp, 8088/tcp, 8033/tcp, 49707/tcp, 50090/tcp, 8031/tcp, 8040/tcp, 8042/tcp, 8032/tcp   elegant_blackwell   
77759268965c        sequenceiq/ambari:1.6.0               "/usr/local/serf/bin   13 minutes ago      Up 13 minutes       7373/tcp, 7946/tcp, 0.0.0.0:49153->8080/tcp                                                                                                      ambari-singlenode   
83998f152f10        bfoote/docker-dns:latest              "supervisord -n"       16 minutes ago      Up 16 minutes       172.17.42.1:53->53/udp                                                                                                      docker-dns

also executing the two dig commands above I still get the same result. Is this right?

[bnfinet]

in order to ping 83998f152f10.public.dockerA.docker.local reliably you'll need to add 172.17.42.1 to /etc/resolv.conf locally and also configure your docker daemon to use 172.17.42.1 for dns

/etc/resolv.conf

nameserver 172.17.42.1
nameserver 8.8.8.8
search local.dockerA.docker.local  

then:

ping docker-dns
PING 5884b4efc33f.local.dockerA.docker.local (172.17.0.72) 56(84) bytes of data.
64 bytes from 172.17.0.72: icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from 172.17.0.72: icmp_seq=2 ttl=64 time=0.118 ms

make sure nameserver 172.17.42.1 is first

these may be helpful:
https://github.com/bnfinet/docker-dns#setup---configure-your-docker-daemon
https://github.com/crosbymichael/skydock#setup

[bnfinet]

@CristianCantoro did that help any?

[bnfinet]

closing

[CristianCantoro]

Hi,

I'm sorry for not giving a reply earlier.

After editing /etc/resolv.conf it seems that I am able to ping containers with their name:

[~/.docker-dns/docker-dns]$ docker run -d -t \
     --privileged=true \
     --name docker-dns -p 172.17.42.1:53:53/udp \
     -v /var/run/docker.sock:/var/run/docker.sock  \
     -v /home/cristian/.docker-dns/docker-dns/docker/log:/var/log/supervisor \
     -v /home/cristian/.docker-dns/docker-dns/docker/./config/config.js:/opt/docker-dns/config/config.js  \
      bfoote/docker-dns
5d9b5cddcbf6a8f5241a2caac41694300ecfe0626f0a649b82fc41984daf15f0
[~/.docker-dns/docker-dns]$ docker ps
CONTAINER ID        IMAGE                      COMMAND             CREATED              STATUS              PORTS                    NAMES
5d9b5cddcbf6        bfoote/docker-dns:latest   "supervisord -n"    About a minute ago   Up About a minute   172.17.42.1:53->53/udp   docker-dns          
[~/.docker-dns/docker-dns]$ ping 5d9b5cddcbf6.public.dockerA.docker.local
ping: unknown host 5d9b5cddcbf6.public.dockerA.docker.local
[~/.docker-dns/docker-dns]$ sudo nano /etc/resolv.conf
[~/.docker-dns/docker-dns]$ ping docker-dns
PING 5d9b5cddcbf6.local.dockerA.docker.local (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.084 ms
[...]
64 bytes from 172.17.0.2: icmp_seq=13 ttl=64 time=0.097 ms
64 bytes from 172.17.0.2: icmp_seq=14 ttl=64 time=0.159 ms
^C
--- 5d9b5cddcbf6.local.dockerA.docker.local ping statistics ---
14 packets transmitted, 14 received, 0% packet loss, time 22006ms
rtt min/avg/max/mdev = 0.043/0.100/0.161/0.032 ms

and if I start another container

$ docker run -i -t  cristiancantoro/hadoop-mrjob /etc/bootstrap.sh -bash
[...]

then I can ping it as well:

[~/.docker-dns/docker-dns]$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                CREATED             STATUS              PORTS                                                                                                                                            NAMES
e0d2cb635c0b        cristiancantoro/hadoop-mrjob:latest   "/etc/bootstrap.sh -   4 minutes ago       Up 4 minutes        50010/tcp, 8042/tcp, 50070/tcp, 8031/tcp, 49707/tcp, 50075/tcp, 8033/tcp, 22/tcp, 8040/tcp, 8088/tcp, 50090/tcp, 50020/tcp, 8032/tcp, 8030/tcp   sharp_yonath        
5d9b5cddcbf6        bfoote/docker-dns:latest              "supervisord -n"       10 minutes ago      Up 10 minutes       172.17.42.1:53->53/udp                                                                                                                           docker-dns          
[~/.docker-dns/docker-dns]$ ping sharp_yonath
PING e0d2cb635c0b.local.dockerA.docker.local (172.17.0.4) 56(84) bytes of data.
64 bytes from 172.17.0.4: icmp_seq=1 ttl=64 time=0.098 ms
64 bytes from 172.17.0.4: icmp_seq=2 ttl=64 time=0.045 ms
[...]
64 bytes from 172.17.0.4: icmp_seq=12 ttl=64 time=0.056 ms
^C
--- e0d2cb635c0b.local.dockerA.docker.local ping statistics ---
12 packets transmitted, 12 received, 0% packet loss, time 20008ms
rtt min/avg/max/mdev = 0.041/0.055/0.098/0.019 ms

But (and that's quite a big but) if I do so I lost the ability to resolve domains from the internet so if I ping, say, [google.com] this is what I get:

[~/.docker-dns/docker-dns]$ ping google.com
ping: unknown host google.com

I need to have this line in my (host's) resolv.conf

nameserver 127.0.1.1

but if I put it there then everything else stops working and I can not ping docker-dns or other containers anymore.

Furthermore it seems that my conrtainers have lost the ability to resolve urls, for example if I do (in the container above named sharp_yonath:

bash-4.1# yum update
Loaded plugins: fastestmirror, keys, protect-packages, protectbase
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was
14: PYCURL ERROR 6 - "Couldn't resolve host 'mirrorlist.centos.org'"
[...]

but the container is able to contact the internet using IP addresses (74.125.232.137 is one IP address for [google.com]):

bash-4.1# ping google.com
ping: unknown host google.com
bash-4.1# ping 74.125.232.137
PING 74.125.232.137 (74.125.232.137) 56(84) bytes of data.
64 bytes from 74.125.232.137: icmp_seq=1 ttl=55 time=197 ms
64 bytes from 74.125.232.137: icmp_seq=2 ttl=55 time=291 ms
64 bytes from 74.125.232.137: icmp_seq=3 ttl=55 time=1258 ms
64 bytes from 74.125.232.137: icmp_seq=4 ttl=55 time=574 ms
^C
--- 74.125.232.137 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4211ms
rtt min/avg/max/mdev = 197.909/580.777/1258.440/415.115 ms, pipe 2
bash-4.1# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=13.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=17.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=24.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=56 time=16.9 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4151ms
rtt min/avg/max/mdev = 13.614/17.276/24.036/3.769 ms

Also, I can not find a way to restore the ability of my containers to resolve urls.
Any ideas?

[bnfinet]

Hello again!

docker-dns is not a "forwarding dns resolver"

Any domain names which docker-dns does not know about will not resolv.

Before showing you a couple options I'll add the caveat that dns setups can be tricky and what might work in one environment may not work in another.

add multiple dns servers to /etc/hosts

nameserver 8.8.8.8
nameserver 172.17.42.1
search public.dockerA.docker.local

run a more robust dns server in front of docker-dns such as dnsmasq

add these lines to dnsmasq

server=/public.dockerA.docker.local/172.17.42.1
server=8.8.8.8

In addition you should have an understanding of how the dns environment get constructed inside the containers....
https://docs.docker.com/articles/networking/

The --dns and --dns-search arguments passed to the docker daemon at system startup populates the /etc/resolv.conf inside each container.

I had thought of providing dns forwarding features in docker-dns, but it's a slippery slope, and life is too short to get mired in RFC documents. :)

Please do post back here to share your ventures.

[CristianCantoro]

A quick update.

Following the advice in this AskUbuntu post I disabled dnsmasq and restarted the newtwork manager.

Here's how my new /etc/resolv.conf looked like after reconnecting:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.206.180
nameserver 192.168.200.20
search unitn.it

(unitn.it is the network of my university where I am now).

Then I added the following:

nameserver 172.17.42.1
nameserver 8.8.8.8
nameserver 192.168.206.180
nameserver 192.168.200.20
search unitn.it
search local.dockerA.docker.local  

and both local and external DNS resolution work.

[~/.docker-dns/docker-dns]$ ping docker-dns
PING 04f275bbaa40.local.dockerA.docker.local (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.050 ms
^C
--- 04f275bbaa40.local.dockerA.docker.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.050/0.066/0.083/0.018 ms
[~/.docker-dns/docker-dns]$ ping google.com
PING google.com (173.194.78.138) 56(84) bytes of data.
64 bytes from wg-in-f138.1e100.net (173.194.78.138): icmp_seq=1 ttl=47 time=34.5 ms
64 bytes from wg-in-f138.1e100.net (173.194.78.138): icmp_seq=2 ttl=47 time=34.8 ms
64 bytes from wg-in-f138.1e100.net (173.194.78.138): icmp_seq=3 ttl=47 time=35.9 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 34.555/35.100/35.924/0.630 ms

The last question that I have is: how can I configure everything such taht I do not have to modify manually resolv.conf every time? As the file itself says, it gets rewritten every time the Network manager starts a new connection.

[bnfinet]

this is one of the tricky parts

your /etc/resolv.conf is getting manipulated by resolvconf, and possibly with the help of a dhcp client

[CristianCantoro]

(I have seen now the answer you posted while I was writing my other comment. I will try to firstly understand it 😅 and then try again).

[CristianCantoro]

I think that the easiest solution is to add the line:

search public.dockerA.docker.local

to the file:

/etc/resolvconf/resolv.conf.d/base

In this way the NetworkManager produces a (/etc/)resolv.conf file which has the correct search domain (in addition to the ones you obtain from other sources, say DHCP) and this seems to actually work, i.e. the host can ping containers using their name and the outside world is also reachable.

For completeness, ,my attempt at configuring dnsmasq failed. I had put this lines:

server=/public.dockerA.docker.local/172.17.42.1
server=8.8.8.8

in a file called docker-dns in /etc/dnsmasq.d, but I was not able to ping containers.
(Also, for reference, note that using the full dnsmasq with NetworkManager on Ubuntu has some known problems.)

My last point would be to point out that I cannot ping a container (say docker-dns from inside another container), this seems reasonable since I wasn't unable to tell to dnsmasq on the host to lock for the public.dockerA.docker.local (right? Am I making sense?)