After upgrading atomic host some running containers were removed about atomic

[rhatdan]

No idea. Were all running containers destroyed? Some corruption in devmapper storage?

[roldancer]

Hi @rhatdan, no, just some of them ... is there any docker daemon log ? how can I troubleshoot ?

[rhatdan]

Yes you can run docker logs to look at it, but I doubt you will find anything.

[roldancer]

docker logs is for containers, I need docker daemon's logs ...

I think I will open a support ticket in Redhat, it's a really serious issue.

Thanks.

[rhatdan]

Then look at journalctl

journalctl -u docker.service

[rhvgoyal]

@roldancer

Can you please paste output of "docker info". First thing I am trying to figure out what's the storage setup you are using.

Also need docker logs from journal as rhatdan said, to see if there is any message which can give information. We should not be losing containers over upgrade.

[roldancer]

hi @rhvgoyal , docker info output

-bash-4.2# docker info
Containers: 7
Images: 172
Storage Driver: devicemapper
Pool Name: rah_vmlbcatoml01-docker--pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 6.223 GB
Data Space Total: 26.43 GB
Data Space Available: 20.21 GB
Metadata Space Used: 7.885 MB
Metadata Space Total: 33.55 MB
Metadata Space Available: 25.67 MB
Udev Sync Supported: true
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.1.2.el7.x86_64
Operating System: Employee SKU
CPUs: 2
Total Memory: 1.797 GiB
Name: vmlbcatoml01.lvtc.gsnet.corp
ID: RHZI:DYCI:3ZUO:EAWE:62SV:VAJJ:A46D:O3NM:T73V:CPQF:MSSZ:LPBF

[roldancer]

Hi All, here is the journalctl -u docker.service output, also the boot time, by the way the system was restarted after upgrading.
There are some errors, any idea ?

-bash-4.2# who -b
system boot Apr 14 12:46

Apr 14 12:46:58 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:46:58+02:00" level="info" msg="+job serveapi(tcp://180.133.160.14:5320)"
Apr 14 12:46:58 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:46:58+02:00" level="info" msg="Listening for HTTP on tcp (180.133.160.14:5320)"
Apr 14 12:46:58 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:46:58+02:00" level="info" msg="/!\ DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING /!"
Apr 14 12:47:00 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:00+02:00" level="info" msg="+job init_networkdriver()"
Apr 14 12:47:01 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:01+02:00" level="info" msg="-job init_networkdriver() = OK (0)"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:02+02:00" level="info" msg="Loading containers: start."
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: ....time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 28e9154c734dfe1f1dcca86d52df333efd9a91e4612e676402f802fcb47cc902: open /var/lib/docker/containers/28e9154c734dfe1f1dcca86d52df333efd9a91e4612e676402f802fcb47cc902/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 29370f79b73c6a6b691abb7e2d328a64b9ff70c5dad423eab0bd2a2b4c052652: open /var/lib/docker/containers/29370f79b73c6a6b691abb7e2d328a64b9ff70c5dad423eab0bd2a2b4c052652/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 363436f5c7abbcb206021b71ca5f1a5e16a66f4931d31ce6f8ca91257db105bb: open /var/lib/docker/containers/363436f5c7abbcb206021b71ca5f1a5e16a66f4931d31ce6f8ca91257db105bb/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 36fe02e626bc35ad8b6bda94210de1a86506feb41cf1ce7b5f5fafc55c1aeafe: json: cannot unmarshal string into Go value of type mode.Mode"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: ...time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 762c544d9cf6d629cd11db6d80e557a3c82363939f57bf16d7c74e8f9677b9b5: open /var/lib/docker/containers/762c544d9cf6d629cd11db6d80e557a3c82363939f57bf16d7c74e8f9677b9b5/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: ...time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container c25deb30f063e73819942d2c641a910eee3247762555ff870e025fd5793c8711: open /var/lib/docker/containers/c25deb30f063e73819942d2c641a910eee3247762555ff870e025fd5793c8711/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container c43f0653e13a11a620ad41874065302ab60210dcf8f18c848b590d84ecdc62a2: open /var/lib/docker/containers/c43f0653e13a11a620ad41874065302ab60210dcf8f18c848b590d84ecdc62a2/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: ..time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container e187a11653c1eef0030db4a682bc25c177d3988f466fb7e4b024e46fd636ef29: open /var/lib/docker/containers/e187a11653c1eef0030db4a682bc25c177d3988f466fb7e4b024e46fd636ef29/config.json: no such file or directory"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container fd554a342cb8307a7d2ec1f04b0ec3693ceded582d442255072d229af4ff89a3: json: cannot unmarshal string into Go value of type mode.Mode"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:02+02:00" level="info" msg="Loading containers: done."
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:02+02:00" level="info" msg="docker daemon: 1.5.0-dev fc0329b/1.5.0; execdriver: native-0.2; graphdriver: devicemapper"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:02+02:00" level="info" msg="+job acceptconnections()"
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:47:02+02:00" level="info" msg="-job acceptconnections() = OK (0)"
Apr 14 12:57:01 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:01+02:00" level="info" msg="GET /v1.18/containers/json?all=1"
Apr 14 12:57:01 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:01+02:00" level="info" msg="+job containers()"
Apr 14 12:57:01 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:01+02:00" level="info" msg="-job containers() = OK (0)"
Apr 14 12:57:38 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:38+02:00" level="info" msg="POST /v1.18/containers/02cd3c2dc93f/start"
Apr 14 12:57:38 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:38+02:00" level="info" msg="+job start(02cd3c2dc93f)"
Apr 14 12:57:38 vmlbcatoml01.lvtc.gsnet.corp docker[953]: time="2015-04-14T12:57:38+02:00" level="info" msg="+job allocate_interface(02cd3c2dc93f27af60289af09f371c843a191f24358fd2573df79f06327ad67e)"

[rhvgoyal]

@roldancer

Following errors are the problem.
Apr 14 12:47:02 vmlbcatoml01.lvtc.gsnet.corp docker[953]: .time="2015-04-14T12:47:02+02:00" level="error" msg="Failed to load container 363436f5c7abbcb206021b71ca5f1a5e16a66f4931d31ce6f8ca91257db105bb: open /var/lib/docker/containers/363436f5c7abbcb206021b71ca5f1a5e16a66f4931d31ce6f8ca91257db105bb/config.json: no such file or directory"

So we are trying to load containers again and then this metadata file config.json is missing for some containers. It might help to just go in that directory (/var/lib/docker/containers/) and look around that how many containers are there and for how many containers config.json is missing.

Why this file is missing, is a mystery so far.

[jeremyeder]

sosreport should help us get further...to run that on atomic, you use the rhel-tools container:

# atomic run registry.access.redhat.com/rhel7/rhel-tools
# sosreport

Then the tarball will be in /host on your atomic system. It's also documented here: https://access.redhat.com/articles/1336853

Also just snooping around in /var/lib/docker/containers i.e.
# docker ps -a ; find /var/lib/docker -name config.json

[rhatdan]

Seems to be an old issue, I am supposing this is working fine now, If not I will reopen.