crosbymichael / skydock Goto Github PK

View Code? Open in Web Editor NEW

1.1K 62.0 88.0 81 KB

Service discovery via DNS for docker

License: MIT License

Go 89.31% JavaScript 7.08% Nginx 3.61%

skydock's People

Contributors

Stargazers

Watchers

Forkers

ehazlett svendowideit rasputnik inthecloud247 asbjornenge pombredanne ernoaapa mainyaa kgoudeaux 1uptalent kballenegger bketelsen drnic tianon cogfor gesellix brimstone fabware digitalsatori ubergarm sebible kyxap1 rayleyva ms123s 2325407504 ozdemircili cloyne-archives mitar peerlibrary ianmiell alemic thinktandem maximf cedrictools sudokeys asandwich vjekoslav lycano itsbp webdev1001 bkrolikowski my-docker zofuthan manics johnhsieh adl1398 dekstroza yonglehou yuriyf mba811 alexwild 5haman artemkaint biswajitpain cloudxtreme dockernuts krylon360 barogi xflash linguahouse 0-t-0 eiencore dvocker 0xcryptowang michaelkuk maniacs-ops bravechou2009 siddharthasingh jinqiqiqi krodyx qianlifu12345 hh9net vermuz sportsbite enterstudio nuaays olitvin victorhuangsh laokan carol8421 enginelabs horizon1997 firebitsbr iq-scm

skydock's Issues

skydock CPU usage question

/go/bin/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns

constantly takes about 24-26% CPU usage. is that normal?

Stops resolving running containers after skydock: EOF under heavy system loads

Executive Summary:

When running a VPS and maxing out RAM and SWAP it might eventually cause skydns+skydock to stop resolving running containers. A manual docker restart skydock will get the services back into skydns and everything resolves happily again after that.

Obvious solution

Throw more hardware at it!
Limit amount of resources containers can take to prevent system starvation.

Preferred Solution

Create a way for skydock+skydns stack to gracefully recover after being temporarily strangled.

Production Setup

My production box in this case is a 2x CPU, 4GB RAM, 4GB SWAP, Digital Ocean VPS running an apache+php container, a mysql container, a skydock container, a skydns container, and nginx on the host.

Overview

I noticed problems right at the end of the month (as site usage peaks) when I started getting 500 errors which required a manual docker restart skydock to get skydns resolving properly again.

Correlating the logs and metrics led me to observe EOF in the skydock logs right when the 500 errors started with graphs showing high system load.

[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF

The best reference I could find was from docker irc chat where a rather overloaded VPS system threw the same errors.

The error seems like it is coming from heartbeat() -> updateService(). So perhaps the connection between skydock and skydns craps out allowing the services to time out?

I was able to repeat it two out of two tries on a fresh VPS test install with the latest docker/skydock/skydns stack. Increasing the TTL from 30 up to 300 made the system stay up longer, but eventually it hung long enough to crap out.

Stress testing provided by by starting multiple gitlab containers: thanks Ruby! :)

Details

Steps to reproduce Skydock EOF and loss of registered services from skydns causing failure to resolve container names.

Test Hardware

Digital Ocean Droplet
1x CPU
~512MB RAM
~512MB Swap

Test OS

Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-29-generic x86_64)
Linux skydock-test 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Docker Version

Installed cgroup-lite and lxc-docker from deb https://get.docker.io/ubuntu docker main

Docker Version:
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

Docker Daemon OPTS

/etc/init/docker.conf -- DOCKER_OPTS="-dns 172.17.42.1 -bip 172.17.42.1/16"

`ulimit -a`

ulimit -a as root:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3781
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3781
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Pull Images

$ docker pull crosbymichael/skydns
$ docker pull crosbymichael/skydock
$ docker pull crosbymichael/redis
$ docker pull sameersbn/gitlab:latest

Start Containers

$ docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker
$ docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 10 -environment dev -s /docker.sock -domain docker -name skydns
$ docker run -d --name redis1 crosbymichael/redis
$ docker run -d --name redis2 crosbymichael/redis

Setup hosts resolv.conf

$ sudo sed -i '1s/^/nameserver 172.17.42.1\n/' /etc/resolv.conf

Test Base Config

Everything starts out resolving fine.

$ ping redis.dev.docker    
$ ping redis1.redis.dev.docker    
$ ping redis2.redis.dev.docker    
$ ping skydock.dev.docker    
$ ping skydns.dev.docker

Introduce Heavy Load

Keep spinning up gitlab containers to thrash the machine until DNS breaks.

$ docker run -d sameersbn/gitlab:latest
...
$ docker run -d sameersbn/gitlab:latest

Skydock logs

[debug] 1399049971 skydock: received event (die) 5226e81f50219a1928f10bd50a9184d54f4b2c2d1e2aaf6503e2ec12c36e0c8f sameersbn/gitlab:latest
[info] 1399049971 skydock: removing 5226e81f50 from skydns
[error] 1399050033 skydock: read tcp 172.17.0.2:8080: connection reset by peer
[error] 1399050034 skydock: EOF
[error] 1399050034 skydock: EOF
[error] 1399050041 skydock: EOF
[debug] 1399050069 skydock: received event (die) 84e06d358662703ddadb7f1835ec7cfce0c317ece05888d625abfc36140b9d90 sameersbn/gitlab:latest
[info] 1399050069 skydock: removing 84e06d3586 from skydns
[debug] 1399050339 skydock: received event (die) 45ad35fad38dad6c56eda5513113fe70dcbff3afa32c61d4a807bdb8924e5f0f sameersbn/gitlab:latest
[info] 1399050339 skydock: removing 45ad35fad3 from skydns

Skydns logs

2014/05/02 16:58:53 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:48430"
2014/05/02 16:58:59 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:56676"
2014/05/02 16:58:59 Calling 0 callback(s) for service 84e06d3586
2014/05/02 16:58:59 Removed Service: 84e06d3586
2014/05/02 16:58:59 Updated Service TTL: 87a190cf50 10
2014/05/02 16:58:59 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:58:59 Updated Service TTL: ccb9859838 10
2014/05/02 16:58:59 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:58:59 Updated Service TTL: 44f871be83 10
2014/05/02 16:58:59 Updated Service TTL: 5226e81f50 10
2014/05/02 16:58:59 Updated Service TTL: 1b16f8703a 10
2014/05/02 16:58:59 Calling 0 callback(s) for service 1b16f8703a
2014/05/02 16:58:59 Removed Service: 1b16f8703a
2014/05/02 16:58:59 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:58:59 Calling 0 callback(s) for service 44f871be83
2014/05/02 16:58:59 Removed Service: 44f871be83
2014/05/02 16:58:59 Calling 0 callback(s) for service 5226e81f50
2014/05/02 16:58:59 Removed Service: 5226e81f50
2014/05/02 16:58:59 Calling 0 callback(s) for service 9233663fd1
2014/05/02 16:58:59 Removed Service: 9233663fd1
2014/05/02 16:59:00 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:44552"
2014/05/02 16:59:01 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:01 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:47767"
2014/05/02 16:59:01 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:02 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:02 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:55471"

2014/05/02 16:59:02 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:02 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:03 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:03 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:03 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:59246"
2014/05/02 16:59:03 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:04 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:06 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:41331"
2014/05/02 16:59:06 Error:  Service does not exist in registry
2014/05/02 16:59:06 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:55604"
2014/05/02 16:59:06 Error:  Service does not exist in registry
2014/05/02 16:59:08 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:52937"
2014/05/02 16:59:08 Error:  Service does not exist in registry
2014/05/02 16:59:08 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:47870"
2014/05/02 16:59:08 Error:  Service does not exist in registry
2014/05/02 16:59:10 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:10 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:11 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:11 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:12 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:14 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:47845"
2014/05/02 16:59:14 Error:  Service does not exist in registry
2014/05/02 16:59:14 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:44293"
2014/05/02 16:59:14 Error:  Service does not exist in registry
2014/05/02 16:59:18 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:18 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:19 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:19 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:20 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:30 Calling 0 callback(s) for service 4a52804fb8
2014/05/02 16:59:31 Removed Service: 4a52804fb8
2014/05/02 16:59:31 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:31 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:31 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:31 Calling 0 callback(s) for service ccb9859838
2014/05/02 16:59:31 Removed Service: ccb9859838
2014/05/02 16:59:31 Calling 0 callback(s) for service 8c65d2828b
2014/05/02 16:59:31 Removed Service: 8c65d2828b
2014/05/02 16:59:31 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:31 Calling 0 callback(s) for service 45ad35fad3
2014/05/02 16:59:31 Removed Service: 45ad35fad3
2014/05/02 16:59:31 Calling 0 callback(s) for service 87a190cf50
2014/05/02 16:59:31 Removed Service: 87a190cf50
2014/05/02 16:59:35 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:55834"
2014/05/02 16:59:35 Error:  Service does not exist in registry

skydns goes zombie

$ docker stop skydns  # nothing after a while hit <cntrl><c>
$ docker kill skydns  # nothing, just hangs

Docker Daemon Logs

2014/05/02 15:41:00 POST /v1.10/containers/87a190cf501c/stop?t=10
[/var/lib/docker|8d7c52cd] +job stop(87a190cf501c)
2014/05/02 15:41:10 Container 87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5 failed to exit within 10 seconds of SIGTERM - using the force
[/var/lib/docker|8d7c52cd] +job release_interface(87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5)
[/var/lib/docker|8d7c52cd] -job release_interface(87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5) = OK (0)
2014/05/02 16:14:48 POST /v1.10/containers/skydns/kill?signal=KILL
[/var/lib/docker|8d7c52cd] +job kill(skydns, KILL)
2014/05/02 16:14:58 Container ccb9859838ab failed to exit within 10 seconds of kill - trying direct SIGKILL

`ps aux`

root      3020  0.1  0.0      0     0 ?        Zsl  12:23   0:20 [skydns] <defunct>

May be related to the kernel version

curl skydns

root@skydock-test:~# curl -X GET -L 172.17.0.2:8080/skydns/services/*
Service does not exist in registry

Repeat Everything with TTL=300 seconds

Same final result but it took more load and time before system hung long enough for problems to occur.

Skydock Log

[debug] 1399064109 skydock: received event (create) 3c0272ebfc8df84d4250b9fbf6a3826e71291b674d46d913ae3fc3930b0e3651 sameersbn/gitlab:latest
[debug] 1399064109 skydock: received event (start) 3c0272ebfc8df84d4250b9fbf6a3826e71291b674d46d913ae3fc3930b0e3651 sameersbn/gitlab:latest
[info] 1399064109 skydock: adding 3c0272ebfc (gitlab) to skydns
[info] 1399064110 skydock: updating ttl for 42482f363d
[info] 1399064110 skydock: updating ttl for a8b74bf97f
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF

Skydns Log

2014/05/02 20:52:44 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:52:44 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:46400"
2014/05/02 20:52:44 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:52:51 Updated Service TTL: 2c1da75ec2 300
2014/05/02 20:52:54 Updated Service TTL: 894c4385c6 300
2014/05/02 20:53:07 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:59028"
2014/05/02 20:53:08 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:53:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:45508"
2014/05/02 20:53:15 Error: Failure to Forward DNS Request "dial udp 8.8.8.8:53: i/o timeout"
2014/05/02 20:53:16 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:51719"
2014/05/02 20:55:08 Error: Failure to Forward DNS Request "dial udp 8.8.8.8:53: i/o timeout"
2014/05/02 20:55:08 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:45692"
2014/05/02 20:55:08 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:44059"
2014/05/02 20:55:09 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:39885"
2014/05/02 20:55:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:38250"
2014/05/02 20:55:09 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:55:09 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:09 Updated Service TTL: a978419b17 300
2014/05/02 20:55:09 Updated Service TTL: c20c19453e 300
2014/05/02 20:55:09 Updated Service TTL: fdda8dd6f2 300
2014/05/02 20:55:09 Updated Service TTL: 206af0ea2f 300
2014/05/02 20:55:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:56460"
2014/05/02 20:55:09 Calling 0 callback(s) for service fdda8dd6f2
2014/05/02 20:55:09 Removed Service: fdda8dd6f2
2014/05/02 20:55:09 Added Service: {3c0272ebfc gitlab clever_fermi dev  172.17.0.7 22 300 2014-05-02 21:00:09.667536585 +0000 UTC map[]}
2014/05/02 20:55:09 Updated Service TTL: 8c1a3c129c 300
2014/05/02 20:55:09 Updated Service TTL: 198847fca4 300
2014/05/02 20:55:09 Updated Service TTL: 4b10f487ac 300
2014/05/02 20:55:09 Updated Service TTL: ca82643f15 300
2014/05/02 20:55:09 Updated Service TTL: b7dda2baa5 300
2014/05/02 20:55:09 Updated Service TTL: 4b59b2c0d3 300
2014/05/02 20:55:09 Updated Service TTL: 1c05e58b3c 300
2014/05/02 20:55:09 Updated Service TTL: 8b15f1d0b2 300
2014/05/02 20:55:09 Updated Service TTL: a7ae9c16b2 300
2014/05/02 20:55:09 Calling 0 callback(s) for service 2c1da75ec2
2014/05/02 20:55:09 Removed Service: 2c1da75ec2
2014/05/02 20:55:09 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:09 Added Service: {de17559007 gitlab jolly_engelbart dev  172.17.0.21 22 300 2014-05-02 21:00:09.672444668 +0000 UTC map[]}
2014/05/02 20:55:09 Updated Service TTL: c0566e3579 300
2014/05/02 20:55:09 Calling 0 callback(s) for service 206af0ea2f
2014/05/02 20:55:09 Removed Service: 206af0ea2f
2014/05/02 20:55:09 Calling 0 callback(s) for service a978419b17
2014/05/02 20:55:09 Removed Service: a978419b17
2014/05/02 20:55:09 Calling 0 callback(s) for service c20c19453e
2014/05/02 20:55:09 Removed Service: c20c19453e
2014/05/02 20:55:09 Calling 0 callback(s) for service a7ae9c16b2
2014/05/02 20:55:09 Removed Service: a7ae9c16b2
2014/05/02 20:55:09 Calling 0 callback(s) for service c0566e3579
2014/05/02 20:55:09 Removed Service: c0566e3579
2014/05/02 20:55:09 Calling 0 callback(s) for service 8c1a3c129c
2014/05/02 20:55:09 Removed Service: 8c1a3c129c
2014/05/02 20:55:10 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:57337"
2014/05/02 20:55:10 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:51288"
2014/05/02 20:55:10 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:10 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:55:10 Updated Service TTL: 42482f363d 300
2014/05/02 20:55:10 Updated Service TTL: a8b74bf97f 300

Ignore images that are not tagged

This will help us filter when ppl run docker build

No DNS for containers with untagged images

I've found that if I have containers just running latest, and then I update the image, thus untagging the image currently in use, Skydock does not add a DNS entry for it.

Not sure if this is solvable with Skydock or not outside of storing an image ID's tag for later.

Issues under load with binding to docker0

Putting a bit of load on my applications is causing the docker daemon to hang. It seems to be the binding to docker0 for skydns (-p 172.17.42.1:53:53/udp) that is causing the issues.

/var/log/upstart/docker.log says:

2014/01/30 12:20:52 Can't proxy a datagram to udp/172.17.0.2:53: dial udp 172.17.0.2:53: too many open files

...and goes on...

2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 5ms
2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 10ms
2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 20ms

...same for the socket...

2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 5ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 10ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 20ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 40ms

If I instead avoid the port binding for skydns, and rather pass

-dns $(docker inspect --format='{{.NetworkSettings.IPAddress}}' skydns)

to my containers, I can put as much load as I want on it without getting these errors.

Might be something specific to my setup, but I also tried a vanilla vm without any network tampering and saw the same.

Can this be used for zero-downtime deployments? How to properly proxy from nginx?

Say I'm using skydock and running a few ruby web apps in docker:

app1.myrubyapp.production.myapp.com
app2.myrubyapp.production.myapp.com
app3.myrubyapp.production.myapp.com

I think I have two questions:

How do I properly set up nginx to proxy between the three apps? The nginx config example in the repo only looks to proxy back to one:

set $dns app-1.ruby.live.docker;
....
proxy_pass http://$dns:80;

If I bring a container down to redeploy, with a TTL of 30 seconds will skydock know that the container is down and re-route accordingly?

Please excuse me if the questions are basic... I'm just getting into web dev stuff.

Example should include instructions to start from the docker example vm

In particular, in my setup, it is running upstart, with which I am not familiar. Instructions for modifying either the Vagrantfile or the file system would be appreciated.

Running on a Mac, 10.9.1.
Vagrant 1.3.5
Virtual box 4.3.0 r89960
Using the docker image from the demo on the website. Hacked docker.conf until it didn't start on reboot & ran docker by hand with options from your example.

Can skydns do the load-balancing too?

Hi, I was wondering whether SkyDns will also do the load-balancing like HA-Proxy?

Or do I still need to add HA-proxy layer?

Terminal hanging on example for redis login

docker run -t -i crosbymichael/redis-cli -h redis1.redis.dev.docker
redis1.redis.dev.docker:6379>

I get no response from the terminal. ^C in the terminal kills the session to the redis host, but I'm not at all sure what's really happening there.

[Support] ping/dig work to SkyDNS, any other command fails

For some reason the SkyDNS domain names don't resolve when using anything but nslookup/dig/ping, any idea why? See output below:

[root@docker opt]# ping skydns.dev.dockerdev
PING skydns.dev.dockerdev (172.17.0.7) 56(84) bytes of data.
64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 172.17.0.7: icmp_seq=2 ttl=64 time=0.055 ms
--- skydns.dev.dockerdev ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.024/0.039/0.055/0.016 ms

[root@docker opt]# dig skydns.dev.dockerdev

; <<>> DiG 9.9.2-P2 <<>> skydns.dev.dockerdev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28414
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;skydns.dev.dockerdev.         IN      A

;; ANSWER SECTION:
skydns.dev.dockerdev.  23      IN      A       172.17.0.7

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jan 30 22:26:27 2014
;; MSG SIZE  rcvd: 76

However a curl or wget results in:

[root@docker opt]# curl http://skydns.dev.dockerdev:8080
curl: (6) Could not resolve host: skydns.dev.dockerdev

[root@docker opt]# wget skydns.dev.dockerdev
--2014-01-30 22:27:39--  http://skydns.dev.dockerdev/
Resolving skydns.dev.dockerdev (skydns.dev.dockerdev)... failed: Name or service not known.
wget: unable to resolve host address 'skydns.dev.dockerdev'

All that resolv.conf contains is

nameserver 127.0.0.1

And the VMs are created using:

docker run -d -p 53:53/udp -name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain dockerdev
docker run -d -v /var/run/docker.sock:/docker.sock -name skydock -link skydns:skydns crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain dockerdev

Skydock DNS loadbalancing

I'm looking to balance load between containers of the same service. So let's say I have two api containers, api1 and api2. I would like to distribute load between these two containers.

What would be a good approach?

According to this there are quite a few considerations and client variations here. Does anyone know how skydns picks the order in which servers are listed?

I wouldn't want to wait until clients are unable to connect to api1 before moving some of the traffic to api2. An idea could be a -balance "[api,es,redis]" flag for skydock which would do a DELETE & PUT instead of PATCH when updating TTL. Depending on how skydns orders things.

Ideas? Thoughts? Am I missing something obvious?

Skydock thinks an image is untagged if the it has 2 names

If you tag the same commit with 2 differents image's name, when you start the containers with image A, docker may register image B un the Config section. It's not a problem except for skydock who thinks that the image is not tagged and returne a ErrImageNotTagged error.

$ docker images | grep d3b7d9f39158
foo-mailcatcher                                       latest                d3b7d9f39158        5 days ago          381.3 MB
bar-mailcatcher                                      latest                d3b7d9f39158        5 days ago          381.3 MB

$ docker run -d --name foo foo-mailcatcher
b9c81f106ad0509edc15d34305bb26ba766a465462d4a3a0f20f0f75e239e069
$ docker run -d --name bar bar-mailcatcher
80a7fe67d59d92b88d2121a75ce3b4e3bb9c19105e2f167bfaa79ceb288ca78a

$ docker inspect --format '{{ .Config.Image }}' foo
bar-mailcatcher

$ docker inspect --format '{{ .Config.Image }}' bar
bar-mailcatcher

The first image will raise a ErrImageNotTagged in skydns

Cannot use real top-level domain

I am using -environment server and -domain example.com and if I try to resolve server.example.com I get back all IPs inside the environment, instead of real server.example.com IP for that server. Even crazier, if I try to resolve example.com, I get back 0.0.0.0.

I think that everything from environment up (inclusive) should be recursed to provided DNS server.

Support SkyDNS version 2

SkyDNS version 1 dev slowed down to a crawl, exiting new stuff is being added to SkyDNS version 2. Could you add support for this in skydock?

Multiple hosts

After thinking some more about multiple host support, I don't think there is an elegant DNS solution to the problem.

One idea I have is to allow listening on multiple docker hosts, e.g. supporting multiple -s parameters, and leaving the networking between the hosts to some other mechanism.

This would keep skydock simple, focused and clean.

A setup similar to http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/ might work...? Or any other setup. The more flexible skydock would not make assumptions about it and simply get out of it's way.

Thoughts?

SkyDock & Boot2docker & DNS resolving under mac

First: Great work with SkyDock! 👍 I think service discovery via DNS is the way to go.

Second: Here is what I like to do:

$ docker run -d -p 8090:80 mattes/hello-world-nginx

# this works:
# from within docker container
$ docker run -it ubuntu /bin/bash
$ apt-get update && apt-get install —y curl
$ curl hello-world-nginx.dev.docker
Hello world!

# this works:
# from within boot2docker
$ boot2docker ssh
sudo vi /etc/resolv.conf 
 nameserver 172.17.42.1
$ curl hello-world-nginx.dev.docker
Hello world!

# from my mac (this won't work)
$ boot2docker ip
192.168.59.103
$ sudo sh -c "echo 'nameserver 192.168.59.103' >> /etc/resolver/docker"
# more info about /etc/resolver under mac: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/resolver.5.html
$ curl hello-world-nginx.dev.docker
(hangs)
$ dig @192.168.59.103 +short hello-world-nginx.dev.docker
172.17.0.32

So the problem here is that hello-world-nginx.dev.docker resolves to 172.17.0.32. Which is actually correct, but makes no sense from the mac side.

Side note: I started skydock with docker run -p 53:53/udp .... So no binding to -p 172.17.42.1:53:53/udp.

Any ideas how to make this work?

start/stop race cond?

I'm using upstart to manage my container services.
If I use upstart's restart command to restart the container most of the time I'll see the dns entry was removed from skydns but never added back in... at least not until I stop it again... wait... then start it back up.

Not sure if this would also be an issue with Docker's restart command.

Is it possible to run more than one skydock instance in a single machine?

Hello everyone, first of all, thanks for creating this awesome tool.

Let me explain the experiment I'm trying to accomplish; basically I want to spin up a set of services instances under different environments (with some sed'ing to create specific conf files out of some templates), let's say I run the skydock instances as follows:

    docker run -d -v /var/run/docker.sock:/docker.sock --name skydock_qa --link skydns:skydns crosbymichael/skydock -ttl 30 -environment prod -s /docker.sock -domain company.local

    docker run -d -v /var/run/docker.sock:/docker.sock --name skydock_uat --link skydns:skydns crosbymichael/skydock -ttl 30 -environment uat -s /docker.sock -domain company.local

What I have found (docker ps) is that only the second instance is active, while the first one is stopped, causing the services instances that attempt to reach each other using the "qa" env. to not be able to communicate between them.

Is this the intended behavior?
Should I approach it differently?

Updating the value of hostname inside container

Hi,

Is it possible to update the hostname of the container to reflect its DNS name?
I'm trying to setup a RabbitMQ cluster with a few containers, and they are very particular about hostnames.

Do not bind http port to 0.0.0.0

When using a cluster of SkyDNS servers, the server used for the -http parameter is also used for redirecting API requests on followers to the master.

This means that any API request performed on a follower will most likely fail, since skydock will not find the master.

Issues with 60 sec TTL

I don't know if this is still an issue with the latest code.

But I received this error when using 60 sec TTL, no issue when reverting back to 30 sec TTL

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x40175c]

goroutine 8 [running]:
main.heartbeat(0xc200136d20, 0xa)
        /home/ubuntu/docker/builder/tmp/skydock/main.go:126 +0x2fc
created by main.sendService
        /home/ubuntu/docker/builder/tmp/skydock/main.go:179 +0x2b8

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc200162068)
        /usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc200112180)
        /usr/lib/go/src/pkg/sync/waitgroup.go:109 +0xf2
main.main()
        /home/ubuntu/docker/builder/tmp/skydock/main.go:281 +0x5c1

goroutine 2 [syscall]:

goroutine 4 [syscall]:
os/signal.loop()
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x2f

goroutine 13 [chan receive]:
main.heartbeat(0xc200136dc0, 0xa)
        /home/ubuntu/docker/builder/tmp/skydock/main.go:116 +0x18e
created by main.sendService
        /home/ubuntu/docker/builder/tmp/skydock/main.go:179 +0x2b8

goroutine 14 [IO wait]:
net.runtime_pollWait(0x7f94aad93e60, 0x72, 0x0)
        /usr/lib/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001be3e0, 0xb, 0xc200111930)
        /usr/lib/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2001be360, 0xc20016a000, 0x1000, 0x1000, 0x0, ...)
        /usr/lib/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc2001620a0, 0xc20016a000, 0x1000, 0x1000, 0x0, ...)
        /usr/lib/go/src/pkg/net/net.go:123 +0xc3
bufio.(*Reader).fill(0xc2001b6ae0)
        /usr/lib/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).ReadSlice(0xc2001b6ae0, 0xa, 0x0, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/bufio/bufio.go:262 +0x202
net/http.readLine(0xc2001b6ae0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/net/http/chunked.go:91 +0x57
net/http.(*chunkedReader).beginChunk(0xc2001bd390)
        /usr/lib/go/src/pkg/net/http/chunked.go:47 +0x28
net/http.(*chunkedReader).Read(0xc2001bd390, 0xc20017be00, 0x200, 0x200, 0x200, ...)
        /usr/lib/go/src/pkg/net/http/chunked.go:65 +0x8f
net/http.(*body).Read(0xc2001bc7c0, 0xc20017be00, 0x200, 0x200, 0x0, ...)
        /usr/lib/go/src/pkg/net/http/transfer.go:535 +0x94
encoding/json.(*Decoder).readValue(0xc2001b5540, 0xc2000b90a0, 0x7f94aad50010, 0x420853)
        /usr/lib/go/src/pkg/encoding/json/stream.go:124 +0x46a
encoding/json.(*Decoder).Decode(0xc2001b5540, 0x6c3700, 0xc2001620c8, 0xc200167570, 0xc2001bc7c0, ...)
        /usr/lib/go/src/pkg/encoding/json/stream.go:44 +0x60
github.com/crosbymichael/skydock/docker.func·002()
        /home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:186 +0x607
created by github.com/crosbymichael/skydock/docker.(*dockerClient).GetEvents
        /home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:196 +0xa8

goroutine 15 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
        /home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
        /home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f

goroutine 16 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
        /home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
        /home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f

goroutine 17 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
        /home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
        /home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f

goroutine 18 [chan receive]:
github.com/crosbymichael/skydock/docker.func·001()
        /home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:174 +0x54
created by github.com/crosbymichael/skydock/docker.func·002
        /home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:181 +0x551

Resolv directly with container name without the service component

We're running our environment in a way that all our containers have already an unique name and thus the 'service' component in the domain name gets annoyingly in the way.

Could skydock offer a command line argument which registers the container without the service name component?

Invalid HTTP response using docker 0.11.1

[info] 1399526496 skydock: adding f6647e8847 (consulserver) to skydns
[error] 1399526496 skydock: error adding f6647e8847 to skydns: Invalid HTTP response

On ubuntu 12.4. Anything else you want to know, I'd be happy to try and find it.

Support SERVICE_NAME to override image name

Sometimes, I don't want the image name to be used as the service name. Redis is a great example. I want to do this:

$ docker run -d --name redis-1 -e SERVICE_NAME=redis-cache redis
$ docker run -d --name redis-2 -e SERVICE_NAME=redis-cache redis
$ docker run -d --name redis-3 -e SERVICE_NAME=redis-cache redis
$
$ docker run -d --name redis-1 -e SERVICE_NAME=redis-session redis
$ docker run -d --name redis-2 -e SERVICE_NAME=redis-session redis
$ docker run -d --name redis-3 -e SERVICE_NAME=redis-session redis
$
$ host redis-cache.dev.skydns
redis-cache.dev.skydns has address 172.18.102.31
redis-cache.dev.skydns has address 172.18.102.30
redis-cache.dev.skydns has address 172.18.102.29
$
$ host redis-session.dev.skydns
redis-session.dev.skydns has address 172.18.102.18
redis-session.dev.skydns has address 172.18.102.14
redis-session.dev.skydns has address 172.18.102.6

Having to create new image tags for this seems like overkill.

This idea is stolen from: https://github.com/progrium/registrator

Question on redis-cli in the youtube video

https://www.youtube.com/watch?v=Nw42q1ofrV0
At 15:40 you have talked about redis-cli doing service discovery. Do you have the code for the redis-cli on the github ? Just wanted to understand how redis-cli is using skydock under the hoods..

Stopped containers are continuing to be advertised

I stopped my redis1, redis2 and redis3 containers. Also rmed the names.

The DNS entries are still appearing.

$ dig @172.17.42.1 redis.dev.docker

; <<>> DiG 9.8.1-P1 <<>> @172.17.42.1 redis.dev.docker
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63974
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;redis.dev.docker.      IN  A

;; ANSWER SECTION:
redis.dev.docker.   16  IN  A   172.17.0.4
redis.dev.docker.   7   IN  A   172.17.0.5
redis.dev.docker.   10  IN  A   172.17.0.6

Remove log fatals

containers started with a tag (other than the default / latest) are not captured by skydock

if a container is started with a specific tag ( i had a container with a numeric tag 1.2.3), it looks like skydock doesnt catch the event..
If i start the same container without a tag (by giving it a tag - latest), it works fine..

Unable to get README example to work

Not long ago I was able to get things to work but after reformating my machine I'm unable to get Skydock / SkyDNS working properly on an Ubuntu 14.04 host (dunno if it matters but it worked on my previous 13.10 installation). While I can't get things to work properly, I didn't change my /etc/default/docker to include --dns, and this is why you'll notice that I pass it around on the examples below.

These are the commands I ran on separate bash sessions and a "live demo" can be found on https://asciinema.org/a/9338

# SkyDNS
docker run -ti --rm \
           -p 172.17.42.1:53:53/udp \
           --dns 172.17.42.1 \
           --name skydns \
           crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker

# Skydock
docker run -ti --rm \
           -v /var/run/docker.sock:/docker.sock \
           --dns 172.17.42.1 \
           --name skydock \
           crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns

# Redis server
docker run -ti --rm \
           --name redis1 \
           --dns 172.17.42.1 \
           crosbymichael/redis

# Finnaly the redis-cli
docker run -ti --rm \
           --dns 172.17.42.1 \
           crosbymichael/redis-cli -h redis1.redis.dev.docker
# that gives me a
Could not connect to Redis at redis1.redis.dev.docker:6379: Temporary failure in name resolution

More information:

$ docker images | grep sky
crosbymichael/skydock            latest              e985023521f6        4 weeks ago         510.7 MB
crosbymichael/skydns             next                79c99a4608ed        5 weeks ago         525 MB
crosbymichael/skydns             latest              1923ce648d4c        3 months ago        137.5 MB

$ docker info
Containers: 1
Images: 270
Storage Driver: aufs
 Root Dir: /home/fabio/docker/aufs
 Dirs: 288
Execution Driver: native-0.1
Kernel Version: 3.13.0-24-generic
Username: fgrehm
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

$ docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0

If there's any other information you guys want from me to help debugging please LMK. Thanks in advance!

`open /var/run/docker.sock: no such device or address`

Sorry but I'm clueless about what's going on :/

$ docker run -d -v /var/run/docker.sock:/docker.sock -name skydock -link skydns:skydns crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker
0b18412a05ef15244e6d8f050dd71f8fb2b46c8ceb52165240c5303a28f97f1d
2014/01/29 23:51:18 Error: start: Cannot start container 0b18412a05ef15244e6d8f050dd71f8fb2b46c8ceb52165240c5303a28f97f1d: open /var/run/docker.sock: no such device or address
$ ls -l /var/run/docker.sock 
srw-rw---- 1 root docker 0 Jan 30 00:17 /var/run/docker.sock
$ docker -v
Docker version 0.7.1, build 88df052
$ docker images | grep crosbymichael
crosbymichael/skydock   latest              2016b214a02c        7 days ago          137.2 MB
crosbymichael/skydns    latest              fbdbaca11389        9 days ago          139.6 MB

Drop to non-root user

I know that sykdns and skydock processes are running inside an LXC, but still, there is no need for them to really run as a root. Processes could drop to non-root user after opening sockets.

Skydock don't add containers from private registry

If my image came from a private registry, SkyDock don't add it to SkyDns

docker run -d --name apache my-private-registry.private.network/apache
[debug] 1415354852 skydock: received event (create) cd85709041e6e461b2565d1089ed26744ea7ede36076620cab892b875cc94a11 apache:latest
[debug] 1415354852 skydock: received event (start) cd85709041e6e461b2565d1089ed26744ea7ede36076620cab892b875cc94a11 apache:latest

docker pull my-private-registry.private.network/apache
docker run -d --name apache apache
[debug] 1415354888 skydock: received event (create) 22e4a95e1baada889a5f5e70bea40e2df250ba5e7e0233c60ab55ee2c8a967e8 apache:latest
[debug] 1415354888 skydock: received event (start) 22e4a95e1baada889a5f5e70bea40e2df250ba5e7e0233c60ab55ee2c8a967e8 apache:latest
[info] 1415354888 skydock: adding 22e4a95e1b (apache) to skydns

Host value as a argument

Hi,
I'm playing with Skydock and it looks good! Do you have planned to support alternative host -value?
For example: now it reports 172.17.0.XX as a host to Skydns, but I would like to use public ip address as a host value. What do you think?

can't get it working :(

I have followed all instructions on the readme, but i cannot seem to reach the containers by hostname is there anyway to debug it where it is failing?

The ip is added on resolv.conf in the container and from there it does seem to work

DNS resolution fails after 20 hours uptime

I have just set this up for a development environment and have ran into a problem after 20 hours.

No containers have been stopped, skydns and skydock are up running the latest images.
redis instance was functioning just fine
waited 20 hours and now it is not resolving, yet it should be

I grabbed specific logs for the skydock service and it shows a stop and die message for the not running redis instance. restarting redis instance fixes the issue.

skydock is not usable in this condition.

Somehow instances are failing TTL and not being put back into DNS.

two issues in the latest CoreOS / docker 0.8.1

Sorry for having a crappy subject line and combining two issues into one, but both fixes are needed to make skydock more usable in CoreOS, and they might be related...

I haven't found a way to start skydock and skydns in systemd reliably. I have skydock Requires=skydns.service and After=skydns.service but it looks like running docker start -a skydock is not enough to guarantee the container is actually running. I get "Cannot link to a non running container". If I wait until the container is running (at least a few seconds after start -a skydns) I get no problem linking. Maybe you have a better way of starting these services on boot. There is no treatment in the README.md. (Help!)
This is a new issue with docker 0.8.1, I think I read that the behavior of -link has changed: I get "failed to send 88a03c289d to skydns" in my journalctl logs. The full text with a few running containers:

[info] 1394224127 skydock: loading plugins from /plugins/default.js
[debug] 1394224127 skydock: starting restore of containers
[info] 1394224127 skydock: adding 0378034680 (skydock) to skydns
[error] 1394224127 skydock: failed to send 0378034680 to skydns on restore: Put http://:8080/skydns/services/0378034680: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 88a03c289d (ntp) to skydns
[error] 1394224127 skydock: failed to send 88a03c289d to skydns on restore: Put http://:8080/skydns/services/88a03c289d: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 47ba5e9b9b (skydns) to skydns
[error] 1394224127 skydock: failed to send 47ba5e9b9b to skydns on restore: Put http://:8080/skydns/services/47ba5e9b9b: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 763611feaf (registry) to skydns
[error] 1394224127 skydock: failed to send 763611feaf to skydns on restore: Put http://:8080/skydns/services/763611feaf: dial tcp :8080: connection refused
[debug] 1394224127 skydock: starting main process

It seems like skydock is not getting the address of skydns for some reason. I am not sure if the change in link semantics I heard about is responsible. This CoreOS release was only less than 24 hours ago, but Docker 0.8.1 has been out for a while, did you use it to test skydock already?

Skydns/Skydock fails to start under Docker 0.9

I get nothing when I try pulling the published skydock/skydns images under Docker 0.9.

That is, starting them outputs absolutely nothing in logs. It seems to immediately return as well, even when -d isn't specified.

This is with a fresh install of Docker 0.9 on Ubuntu 13.10. A downgrade to 0.8.1 resolved everything.

I'll test further later, but I hit this when trying to roll out a fresh deployment, so figured I'd at least log it. I'll test the environment in Vagrant and add more details as I get them. Wish I had more time to provide better details now, but maybe someone else has encountered this and has more.

No need for link

Given that skydock has access to the Docker socket, it should be possible to query Docker for the IP address given an agreed-upon name of skydns (or one configurable via an environment variable). That would alleviate the need to use container links and generally make setup simpler.

Event Lost on few simultaneous docker start ?

Background:

I'm testing this on an m1.small aws ec2.
Docker is configured not to resume container state (-r=false).
Container is started via upstart.
First is skydns, then skydock.
After that redis, mysql, fpm, and nginx
When rebooting, some of the containers are not registered by skydock.

Here is the log:

[info] 1391926655 skydock: loading plugins from /plugins/containerEnv.js
[debug] 1391926655 skydock: starting restore of containers
[info] 1391926655 skydock: adding 62cc880410 (skydock) to skydns
[info] 1391926655 skydock: adding 9ec3e07522 (skydns) to skydns
[debug] 1391926655 skydock: starting main process
[debug] 1391926656 skydock: received event (create) f8aa137b623fe5854037bf96f955df5f086b059bce69bcaff82fdff576b83837 atn-mysqld:latest
[debug] 1391926656 skydock: received event (create) 8a29be7375656d3ccc3ca44ac259b14b0df396dabb5c62731b280a9179f18ba9 atn-redis:latest
[debug] 1391926657 skydock: received event (create) e38405f92010678abc2fcc9139acf020db6d1a02d0fe3ab4bfb797f22a5a1b51 atn-fpm:latest
[debug] 1391926657 skydock: received event (create) 3ec3e1cc6bb27d68e886f74656fdd8180cc53a4747bd18ecfc088f91e5c554d4 atn-nginx:latest
[debug] 1391926657 skydock: received event (start) 8a29be7375656d3ccc3ca44ac259b14b0df396dabb5c62731b280a9179f18ba9 atn-redis:latest
[info] 1391926658 skydock: adding 8a29be7375 (redis) to skydns
[debug] 1391926658 skydock: received event (start) 3ec3e1cc6bb27d68e886f74656fdd8180cc53a4747bd18ecfc088f91e5c554d4 atn-nginx:latest
[info] 1391926658 skydock: adding 3ec3e1cc6b (www) to skydns
[debug] 1391926658 skydock: received event (start) e38405f92010678abc2fcc9139acf020db6d1a02d0fe3ab4bfb797f22a5a1b51 atn-fpm:latest
[info] 1391926658 skydock: adding e38405f920 (fpm) to skydns

After some time I stop the mysql container

[debug] 1391935317 skydock: received event (die) f8aa137b623fe5854037bf96f955df5f086b059bce69bcaff82fdff576b83837 atn-mysqld:latest
[info] 1391935317 skydock: removing f8aa137b62 from skydns

I'm not sure yet if this is skydock or docker fault.
Thank u for sharing this. Skydock looks promising.

Support filters for events

will post details soon

Trouble with domain to container resolution on OS X

Hi,

I'm having trouble connecting to containers by hostname from the OS X side.

Is there anything I'm doing wrong here? Any help/ideas greatly appreciated!

Thx! Gijs

The following is what I know so far.

Running dig from the OS X terminal replies as such:

$ dig elasticsearch.dev.docker.dev

; <<>> DiG 9.8.3-P1 <<>> elasticsearch.dev.docker.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23145
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;elasticsearch.dev.docker.dev.  IN  A

;; ANSWER SECTION:
elasticsearch.dev.docker.dev. 18 IN A   172.17.0.13

;; Query time: 0 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Mon Sep  8 13:02:02 2014
;; MSG SIZE  rcvd: 90

And I can query the container using curl using the IP address:

$ curl 172.17.0.13:9200

which returns:

{
  "status" : 200,
  "name" : "Ogre",
  "version" : {
    "number" : "1.2.2",
    "build_hash" : "9902f08efc3ad14ce27882b991c4c56b920c9872",
    "build_timestamp" : "2014-07-09T12:02:32Z",
    "build_snapshot" : false,
    "lucene_version" : "4.8"
  },
  "tagline" : "You Know, for Search"
}

But using curl to query by domain...

$ curl http://elasticsearch.dev.docker.dev:9200

...gives:

curl: (6) Could not resolve host: elasticsearch.dev.docker.dev

Details:

CoreOS (alpha channel) on OS X Mavericks using Vagrant/Virtualbox (installed with Homebrew and Cask)
I've added 172.17.0.3 and 172.17.42.1 alongside the DHCP-provided IP's to the DNS Servers in OS X's System Preferences under Network > Advanced
The added DNS servers show up in /etc/resolv.conf:

#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain nens.local
nameserver 172.17.0.3
nameserver 172.17.42.1
nameserver 192.168.1.2
nameserver 192.168.1.4

Added a route to VirtualBox using $ sudo route -n add -net 172.17.0.0 10.2.0.10
Output from $ docker logs skydns:

[debug] 1410171742 skydock: received event (start) 467159a7239d7314a8fa675f3ec781f30a6d4c69590c22c43e66b20f9a4e16fd gijs/elasticsearch:1.0
[info] 1410171742 skydock: adding 467159a723 (elasticsearch) to skydns

Output from $ docker logs skydock:

2014/09/08 11:08:03 Received DNS Request for "elasticsearch.dev.docker.dev." from "10.2.0.1:61645"

Curl'ing the SkyDNS registry:

$ curl -XGET http://172.17.0.10:8080/skydns/services/

gives:

[
  {
    "UUID": "8e04b49ff8",
    "Name": "skydock",
    "Version": "skydock",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.11",
    "Port": 80,
    "TTL": 19,
    "Expires": "2014-09-08T10:37:55.252560655Z"
  },
  {
    "UUID": "3b24e79710",
    "Name": "skydns",
    "Version": "skydns",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.10",
    "Port": 53,
    "TTL": 19,
    "Expires": "2014-09-08T10:37:55.254102479Z"
  },
  {
    "UUID": "08707bf365",
    "Name": "pggis",
    "Version": "pggis",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.12",
    "Port": 49156,
    "TTL": 29,
    "Expires": "2014-09-08T10:38:05.14786093Z"
  },
  {
    "UUID": "467159a723",
    "Name": "elasticsearch",
    "Version": "elasticsearch",
    "Environment": "dev",
    "Region": "",
    "Host": "172.17.0.13",
    "Port": 49157,
    "TTL": 14,
    "Expires": "2014-09-08T10:37:49.989654756Z"
  }
]

Output from $ scutil --dns:

DNS configuration

resolver #1
  search domain[0] : docker.dev
  search domain[1] : dev
  search domain[2] : nens.local
  nameserver[0] : 172.17.0.3
  nameserver[1] : 172.17.42.1
  nameserver[2] : 192.168.1.2
  nameserver[3] : 192.168.1.4
  flags    : Request A records
  reach    : Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : docker.dev
  search domain[1] : dev
  search domain[2] : nens.local
  nameserver[0] : 172.17.0.3
  nameserver[1] : 172.17.42.1
  nameserver[2] : 192.168.1.2
  nameserver[3] : 192.168.1.4
  if_index : 4 (en0)
  flags    : Scoped, Request A records
  reach    : Reachable

Another thing I noticed is that whenever I open a website in a browser, something like this shows up in SkyDNS's logs:

2014/09/08 11:14:06 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:07 Received DNS Request for "en.wikipedia.org." from "10.2.0.1:50015"
2014/09/08 11:14:07 Updated Service TTL: 467159a723 30
2014/09/08 11:14:08 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:09 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:10 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:11 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"

...however, the website did continue to work (wikipedia in this case)

Question about Skydock Setup

Hi there,

I'm new to docker (and skydock obviously) and trying to get set up. In the README, you state the following:

First get the IP address of the docker0 bridge. We will need to know the IP of the bridge so that we can tell skydns to bind it's nameserver to that IP.

Is there a simple way to get the IP Address of the docker0 bridge? I can obviously spin up container and do ifconfig, but wondering if docker had a command to get this information out so that in an upstart script it would be easy to run the skydns container on the right IP.

Thanks!

Where can I get that skydnsctl executable as shown in ur youtube video

In your video at around 13:30, you use that skydnsctl cli.
I couldn't find any such thing to install.
Can you make it clear on how can I get it and install?

reverse dns resolve in docker

Does skydock support reverse dns resolve ?
In my docker container, it prints that

bash-4.1# nslookup 172.17.0.7
Server: 172.17.42.1
Address: 172.17.42.1#53

** server can't find 7.0.17.172.in-addr.arpa.: NXDOMAIN

But for the hostname. it works
bash-4.1# nslookup s2.centos65.dev.docker
Server: 172.17.42.1
Address: 172.17.42.1#53

Name: s2.centos65.dev.docker
Address: 172.17.0.7

How do I run 2 environments simultaneously?

First I fired up those two instances.

docker run -d -p 8080:8080 -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker

docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns

And this setup is working as expected.

Now I want to spawn another production environment. This time I only fired another skydock container with the env production as follows.

docker run -d -v /var/run/docker.sock:/docker.sock --name skydock-prod crosbymichael/skydock -ttl 30 -environment prod -s /docker.sock -domain docker -name skydns

Querying the api doesn't show the production skydoc.

curl $(docker-ip):8080/skydns/services/

And now I am wondering on how to setup the production version of skydock.

Do I have to run in separate docker-host?
If I fire up in the same docker host, in which DNS url entry will the new containers be available?
Do I have to pass some flags/variables when I fire new containers to be available in the production env?

Container not getting added to SkyDNS after retagging

Docker Version: 0.8.1
SkyDock version: Both the latest image from index, and the master branch tried.
SkyDNS version: latest image.

After a container's image ends up untagged (Primary reason, image being rebuilt), and then retagging it SkyDock does not seem to detect the new image name, I've tried the following:

Start/stop the container that was re-tagged (Also tried a restart)
Restart SkyDock
Recreate the SkyDock container

The containers with retagged images:

4c6568f28cdc        internal-old/nginx:latest        /start                 18 hours ago         Up 13 minutes       80/tcp                           restore-test                                                                               
8851c7056f5b        internal-oldold/nginx:latest     /start                 18 hours ago         Up 13 minutes       80/tcp                           satis-test

The log output from SkyDock on restart is

[info] 1394187898 skydock: adding 450666b2d4 (router) to skydns
[info] 1394187898 skydock: service already exists for 450666b2d4. Resetting ttl.
...
[info] 1394187898 skydock: adding f05a1bdd17 (mysql) to skydns
[info] 1394187898 skydock: service already exists for f05a1bdd17. Resetting ttl.

But no mention of nginx or the above containers.

Stop/starting the containers with retagged images:

[debug] 1394188038 skydock: received event (stop) 8851c7056f5b86fff433cb1ca041fa9888516bd4a0213bd8b350a5bf30b9706b internal-oldold/nginx:latest
[info] 1394188038 skydock: removing 8851c7056f from skydns
[debug] 1394188041 skydock: received event (start) 8851c7056f5b86fff433cb1ca041fa9888516bd4a0213bd8b350a5bf30b9706b internal-oldold/nginx:latest

But no mention of it being added or the name (nginx) being detected.

Querying the SkyDNS server for any of these also returns no results.

Example fails to include a pull line for crosbymichael/redis-cli

'nuff said.

No DNS created for images hosted on custom registry

When adding images tagged with the following format, no DNS entry is added.

myregistry.mycompany.com/myrepository/imagename:tag

SkyDock / containers unable to resolve under RHEL 6.5

Using docker 0.9.1

Following the skydns/skydock setup I am able to

resolve redis1.redis.dev.docker with skydns from the host
all containers have 172.17.42.1 as their nameserver under /etc/resolv.conf
ping 172.17.42.1

But from within a container with skydns as a server I'm unable to ping either a local domain or any other domain (unknown host google.com)

I've changed the host's /etc/resolv.conf to include skydns which allows the resolution of redis1.redis.dev.docker from the host, but the container still doesn't resolve (unknown host google.com)

I've also disabled SELinux

Changing the dns to 8.8.8.8 and ping works normally.

Testing under Ubuntu 13.04 everything works as expected.

Will test again once Docker 0.1.0 rpms are out as they include SELinux and other bug fixes relating to RHEL.

the language

may you use a programming language please?, not a no-no 60 years deja-vue attempt of bullcrap.

Thank You.