docker-archive / migrator

Tool to migrate Docker images from Docker Hub or v1 registry to a v2 registry

docker/migrator

Tool to migrate Docker images from Docker Hub or v1 registry to a v2 registry including Amazon Elastic Container Registry (ECR)

https://hub.docker.com/r/docker/migrator/

Usage

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY=v1.registry.fqdn \
    -e V2_REGISTRY=v2.registry.fqdn \
    docker/migrator

Environment Variables

The following environment variables can be set:

Required

• V1_REGISTRY - DNS hostname of your v1 registry or Docker Hub (Do not include https://)
  • If migrating images from Docker Hub, use docker.io
• V2_REGISTRY - DNS hostname of your v2 registry (Do not include https://)

Optional

• AWS_ACCESS_KEY_ID - AWS Access Key supplied as either an environment variable or as a part of your credentials file.
• AWS_REGION - AWS Region, must be specified if using ECR
• AWS_SECRET_ACCESS_KEY - AWS Secret Access Key supplied as either an environment variable or as a part of your credentials file.
• ERROR_ACTION - Sets the default action on error for pushes and pulls
  • prompt - (Default) Prompt for user input as to what action to take on error
  • retry - Retry the failed action on error (may cause infinite loop of failure)
  • skip - Log the error and continue migration on error
  • abort - Abort the migration on error
• MIGRATION_INCREMENT - Breaks up migration in chunks of n images
  • Defaults to migrating all images at once if not specified
  • Must be a positive integer
  • Only works if source and destination are not the same FQDN
• USER_PROMPT - Sets the default action for user prompts (non-error)
  • true - (Default) Prompts user for input/validation
  • false - Skips user prompt and automatically proceeds
• NO_LOGIN
  • true - Skips docker login for both the v1 and v2 registries
  • false - (Default) Prompts user to login to the v1 and v2 registries
• V1_NO_LOGIN
  • true - Skips docker login for the v1 registry
  • false - (Default) Prompts user to login to the v1 registry
• V2_NO_LOGIN
  • true - Skips docker login for the v2 registry
  • false - (Default) Prompts user to login to the v2 registry
• USE_INSECURE_CURL
  • true - Allows curl to perform insecure SSL connections for querying APIs
  • false - (Default) Require curl to perform secure SSL connections for querying APIs
• USE_HTTP
  • true - Allows curl to connect to both the v1 and v2 registries over HTTP
    • Note: daemon must also have --insecure-registry option set
  • false - (Default) Requires curl to connect to v1 and v2 registries over HTTPS
• V1_USE_HTTP
  • true - Allows curl to connect to v1 registry running over HTTP
    • Note: daemon must also have --insecure-registry option set
  • false - (Default) Requires curl to connect to v1 registry over HTTPS
• V2_USE_HTTP
  • true - Allows curl to connect to v2 registry running over HTTP
    • Note: daemon must also have --insecure-registry option set
  • false - (Default) Requires curl to connect to v2 registry over HTTPS
• DOCKER_HUB_ORG - Docker Hub organization name to migrate images from
  • Defaults to the username used to login to Docker Hub if not provided
• V1_FULL_REPO_LIST
  • If provided, this allows the user to provide a whitespace separated list of repos for migration. This allows skipping the V1 call to _search (some setups might have search disabled)
• V1_REPO_FILTER - Search filter to limit the scope of the repositories to migrate (uses grep basic regular expression interpretation)
  • Note: This only filters the repositories returned from the source registry search API, not the individual tags
• V1_TAG_FILTER - Search filter to limit the scope of the tags to migrate (Plain text matching).
• LIBRARY_NAMESPACE - Sets option to migrate official namespaces (images where there is no namespace provided) to the library/ namespace (Note: must be set to true for DTR 1.4 or greater)
  • true - (Default) Adds library namespace to image names
  • false - Keeps images as they are without a namespace
• SKIP_EXISTING_TAGS - Option to skip tags that exist at the target repository
  • true - Do not migrate tags that exist at the target repository
  • false - (Default) Do not skip any tags
• Custom CA certificate and Client certificate support - for custom CA and/or client certificate support to your v1 and/or v2 registries, you should utilize a volume to share them into the container by adding the following to your run command:
  • -v /etc/docker/certs.d:/etc/docker/certs.d:ro
• V1_USERNAME - Username used for docker login to the v1 registry
• V1_PASSWORD - Password used for docker login to the v1 registry
• V1_EMAIL - Email used for docker login to the v1 registry
• V2_USERNAME - Username used for docker login to the v2 registry
• V2_PASSWORD - Password used for docker login to the v2 registry
• V2_EMAIL - Email used for docker login to the v2 registry

Note: You must use all three variables (V1_USERNAME, V1_PASSWORD, and V1_EMAIL or V2_USERNAME, V2_PASSWORD, and V2_EMAIL) for the given automated docker login to function properly. Omitting one will prompt the user for input of all three.

Prerequisites

This migration tool assumes the following:

• You have a v1 registry (or Docker Hub) and you are planning on migrating to a v2 registry
• The new v2 registry can either be running using a different DNS name or the same DNS name as the v1 registry - both scenarios work in this case. If you are utilizing the same DNS name for your new v2 registry, set both V1_REGISTRY and V2_REGISTRY to the same value.

It is suggested that you run this container on a Docker engine that is located near your registry as you will need to pull down every image from your v1 registry (or Docker Hub) and push them to the v2 registry to complete the migration. This also means that you will need enough disk space on your local Docker engine to temporarily store all of the images. If you have limited disk space, it is suggested that you use the MIGRATION_INCREMENT option to migrate n number of images at a time.

If you're interested in migrating to an Amazon Elastic Container Registry (ECR) you will additionally need to supply your AWS API keys to the migrator tool. This can be accomplished in one of the two following ways:

docker run -it \
    -v ~/.aws:/root/.aws:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY=v1.registry.fqdn \
    -e V2_REGISTRY=v2.registry.fqdn \
docker/migrator

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e AWS_ACCESS_KEY_ID=<key> \
    -e AWS_SECRET_ACCESS_KEY=<secret> \
    -e V1_REGISTRY=v1.registry.fqdn \
    -e V2_REGISTRY=v2.registry.fqdn \
docker/migrator

How Migration Works

The migration occurs using an automated script inside of the Docker container. Running using the above usage will work as expected.

1. Login to the v1 registry or Docker Hub (Optional)
   • If you do not have authentication enabled, leave the username blank when prompted
2. Query the v1 registry or Docker Hub for a list of all repositories
3. With the list of images, query the v1 registry or Docker Hub for all tags for each repository. This becomes the list of all images with tags that you need to migrate
4. Using a Docker engine, pull all images (including each tag)
5. Once all images are pulled, there are a few options for next steps:
   1. If the same DNS record will be used for the v1 and v2 registries:
      • Have user switch the DNS record over to the new server's IP or if same box to be used, stop the v1 registry and start the v2 registry
   2. If a different DNS record will be used for the v1 and v2 registries:
      • Re-tag all images to change the tagging from the old DNS record to the new one
6. Login to the v2 registry (Optional)
   • If you do not have authentication enabled, leave the username blank when prompted
7. Push all images and tags to the v2 registry
8. Verify v1 to v2 image migration was successful (not yet implemented)
9. Cleanup local docker engine to remove images

Logging Migration Output

If you need to log the output from migrator, add 2>&1 | tee migration.log to the end of the command shown above to capture the output to a file of your choice.

migrator's Issues

migrator attempts to login via v2 API on V1_REGISTRY

On a machine with Debian 9.5 and docker-ce 18.06.1~ce~3-0~debian just installed, I am not able to login the V1 registry. It seems the migrator is accessing the old registry via the v2 protocol...

So, it's trying to access /v2/.

I'm using the FQDN of the v1 registry, without https://.

The v1 registry is running registry:0.9.1. I can see the request made by the migrator:

registry_1  | 172.17.0.8 - - [17/Sep/2018:09:16:54 +0000] "GET /v2/ HTTP/1.1" 404 233 "-" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.0-8-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.6.2)"

The same happens when I try to do docker login (with a up-to-date client), but I guess that's normal as new clients need a v2 registry...

What am I doing wrong?

Add resuming on failed migrations

On failed pushes or pulls, migrator exits immediately. It might be better to log the error and continue, outputting the failed step(s) for further troubleshooting.

Add more detailed status info during migration

Add more detailed information about the status during of the migration to indicate how far along the migration has progressed.

Add checks for env vars

Add checks to make sure the V1_REGISTRY_URL and V2_REGISTRY_URL environment variables are set before running script.

V2_REGISTRY returns lowercase header and fails

In migrator.sh line 811

https://github.com/docker/migrator/blob/5e7ef10ace96e34d33e8870173f230e738a292cf/migrator.sh#L811

The "if" expects "Docker-Distribution-Api-Version: registry/2.0", but it fails with my current registry 2.6.2, because the returned header is the following:

...
docker-distribution-api-version: registry/2.0
...

Am I using some wrong version of the migrator/registry or is the check maybe outdated?

Thanks!

Lucas

[ERROR] v2 registry is not available

Hi again,

In fact it's not working. It fails at the push step.
I used:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY="my.old.registry" \
    -e V2_REGISTRY="10.x.x.x:5001" \
    -e USE_HTTP=true \
    docker/migrator

It fails with:

[ERROR] v2 registry (10.x.x.x:5001) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

From what you explained to me, I think I understand what's going on, but I do not know how to work around it using migrator:
I cannot use localhost:5001 as my V2_REGISTRY since it is executed in the docker container in which localhost is a different computer. That's why I used either the IP address or different DNS names. But it is only localhost that is trusted by default by the docker/registry:2

I guess I could finish the pull manually by executing a push loop on the host using localhost, but is there any way to make it work via the migrator ?

Thanks

Trap for Docker images with encoded characters in their names.

We have found there to be a few Docker images in our registry with encoded characters in their names, like '/' encoded as "%2F".
This is causing issues when trying to do the API curls resulting in 404 errors.
It would be good if there was an option to skip these images without failing the migration.

docker migrator work with public images?

Does the docker migrator work with public images, not under the users account?

Add NO_LOGIN automated test

Add a test that uses NO_LOGIN migration.

Namespaces returns owners only, not read access

When doing a GET on https://hub.docker.com/v2/repositories/namespaces
it returns only namespaces in which members are owners, rather than read access. Thus causing this code block to raise an exception incorrectly

https://github.com/docker/migrator/blob/master/migrator.sh#L430-L436

Custom CA's

How do I add custom CA's into the tool?

HTTP Auth not supported

Hi,

We use an nginx container for authentication to our docker registry. As of now there isn't anyway to authenticate with this service before pulling images. It'd be nice if there was similar support for http_auth as there is for docker.io.

Maybe http_auth_user and http_auth_pass as environment variables?

Thanks

image tag

Hello,

we have a problem with the migration process. When the migraton script pulls the images from the old registry a directory is added. The tags of the images for the new registry contain this added directory. This look something like this: /library/imagename:tag. since its complicated to pull those not correct named images from the new registry it would be nice to change the tag so that they are tagged as image name:tag as usual. Does anyone knows about this "bug" or could tell me where it comes from?

Regards, DrDth

Update V1_REGISTRY and V2_REGISTRY environment variables

The environment variables used, V1_REGISTRY and V2_REGISTRY should be changed to something that is more accurate such as SOURCE_REGISTRY and DESTINATION_REGISTRY as the source may not necessarily be a v1.

Missing MAINTAINERS file

I'm working on preparing the open source repositories for the new centralized maintainers file, but noticed this repository does not yet have a MAINTAINERS file.

I can create a PR to fix this, but I'm not sure who should be included as maintainer(s).

@mbentley are you the only maintainer, or should I add other people here?

for reference, see docker/opensource#35 and moby/moby#18321

Ability to start from a certain image

When moving thousands of images, you don't always want to retry from start. Is it possible to add this feature?

Change environment variables to make more sense

The environment variables used don't make complete sense due to the fact that they are not URLs that you provide. Instead of V1_REGISTRY_URL and V2_REGISTRY_URL, V1_REGISTRY and V2_REGISTRY would make much more sense.

can i migrate from v2 registry to another ?

i have a docker registry:2 private registry which is running for quite sometime in my testing environment with auth ( httpasswd ) and i want to change it's auth method to token based. can i use the migrator to migrate its database ( images, tags, layers everything ? ) or is there another way to do it.

Migrator can't detect my password-protected destination registry

I'm running docker 1.9.1 and migrating from an old registry to v2.2.1 registry. The migrator pulls in all the images from the old repository, but fails to recognize the v2 repository is present

[ERROR] v2 registry (docker2.example.com:5000) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

The registry is running behind an nginx front end which provides the SSL termination. I can access it in a browser, and if I provide the username and password, I can request https://docker2.example.com:5000/v2/ and get a {} response, with a Docker-Distribution-Api-Version: registry/2.0 header.

If I look at what the migrate script is doing, in my nginx logs, I can see a HEAD request for /v2/ failing with a 401, indicating no credentials supplied - this request is made here https://github.com/docker/migrator/blob/master/migrator.sh#L456, and tracing back the V2_OPTIONS doesn't appear to use the V2_USERNAME or V2_PASSWORD

Will this script work with v2 and token based authentication?

We used migrator script to migrate from v1 to a v2 registry which was setup without any authentication/authorization. This worked great, thanks!

Now we have a v2 registry setup with both token-based authentication along with authorization, i.e. using teams and authorized namespaces (public/private) via the Portus project:
https://github.com/SUSE/Portus/

I'm thinking that even if the migrator script can authenticate, I can't see how it would handle authorizations. Any thoughts?

migrate from DTR 1.4.3 to DTR 2.0

Hello,
I used the migrator to migrate from registry V1 to DTR 1.4.3, but now we are planning on upgrading to DTR 2.0, and there is no path to migrate the images. is it possible to support V2 to V2 migration?

Thanks

Add support for DTR 1.4

In DTR 1.4, repositories have to be created prior to pushing images. There are a few challenges that need to be addressed:

• Determine whether or not the v2 registry is DTR or not (need to find endpoint to test against)
• Adding the repos via the DTR API; adding an organization if the user namespace does not exist

Migration error - could not verify layer data

We have a few thousands of images to migrate and 60% of them fail with an error like:

FATA[0051] could not verify layer data for: sha256:5362dab0aca263b53324da67a2769502e7e2f99417ae39594d7a15171e194b75. This may be because internal files in the layer store were modified. Re-pulling or rebuilding this image may resolve the issue 

A similar issue is reported by someone else on Docker forums as well:
https://forums.docker.com/t/could-not-verify-layer-data-error-when-trying-to-push-image-with-content-trust/21790

Remove non-sensical statement

"impossible right now since we can't query the v2 registry" - this makes no sense.

Migrate from V1 to V2 with S3 storage without copying images

Is it possible to migrate from V1 registry (V0.9.1) to the new V2 registry without pushing and pulling all images? I know that this is not the normal migration path, but for what reason do the image need to get pulled? We have a lot of images in our private registry and it would be great not to generate too much traffic if there is no good reason. Would it be possible to convert existing meta-data only ?

Add migration verification

At the moment, there is no verification performed to ensure that each tag is present in the v2 registry after migration; it is assumed based off of the exit codes from each push. While this is relatively safe, it would be better to verify.

SKIP_EXISTING_TAGS compares incomplete list on ECR

The current implementation only makes a single query to the v2 repository to get the list of tags. Some implementations (like Amazon's ECR) hard-limits the number of tags it reports to 100, so at most 100 tags will be skipped.

We need to check if the query returns a Link; rel="next" header and loop trough all pages until we have a complete list.

Expect a PR fixing this in a few hours.

Errors pulling when repository of origin is bigger than 100G.

Error pulling image (59882a691179728f576b972a8af29f9ec6936196) from $myPrivateV1Repo/image, ApplyLayer exit status 1 stdout: stderr: open
/usr/include/openssl/stack.h: read-only file system

This error pops when the machine executing the script reaches 100G of used space by this process.

I tried running the migrator.sh script instead of the docker thinking that maybe I will be able to see something more but to no avail.

Doesn't work with client certificates

When both registries use client certificates the migrator doesn't work, it fails with curl => API failure because curl doesn't use Dockers client certificates.

problem to push images in registry v2

Hi,
I'm trying to migrate my images from a v1 registry to a v2 registry. To do so, I use a reverse proxy to add SSL but I'm not using authentication.
I've got no problem to pull images from registry v1 but when I push I've got this error :

[OK] Verified v2 registry (registry.si3si.com) is available
[!!] Please login to registry.si3si.com:
Username:
FATA[0001] Error response from daemon: no successful auth challenge for https://registry.si3si.com/v2/ - errors: []

I also tried to add NO_LOGIN=true env variable but in this case I've got another error :

cat: /root/.dockercfg: No such file or directory

[ERROR] Non-zero exit code
[ERROR] Migration from v1 to v2 failed!

I've got no other idea of what I can tried

If you need any other information, do not hesitate

Gilles DEHAUDT

REPO_FILTER is not working

I have an image under my organization that I know exists. When I type the name in REPO_FILTER or part of the name, it returns zero results for migration. I'm able to find older images under my org so it is weird that some images are not found.

Migrator still using https despite of V1_USE_HTTP=true specified

Hi,

I'm migrating a v1 registry without SSL to a v2 registry with SSL. Both registries don't need authentication. Here is the command I ran:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY=10.76.2.13:5002 \
    -e V2_REGISTRY=10.76.2.13:5001 \
    -e NO_LOGIN=true -e V1_USE_HTTP=true docker/migrator

and it fails with

FATA[0004] Error response from daemon: Get https://10.76.2.13:5002/v1/_ping: EOF

Migrating from library namespace does not work

Migrating from any workspace other than "library" name space works fine. For some reason it drops the library namespace:

[INFO] Retagging all images from 'dockerdev1.mitre.org' to 'dockeradm1.mitre.org'
[OK] dockerdev1.mitre.org/alpine:2.0 > dockeradm1.mitre.org/alpine:2.0
[OK] dockerdev1.mitre.org/alpine:2015-02-27 > dockeradm1.mitre.org/alpine:2015-02-27
[OK] Successfully retagged all images

my alpine image is stored as dockerdev1.mitre.org/library/alpine

Disk Space Issue : Migrate one image at a time

Instead of downloading the entire v1 registry with all images and tags, can we download one at a time and push to the v2 registry?

The new process would look like :

1. Query v1 registry for all repositories and their tags
2. Pull one image (with a particular tag) to local docker engine
3. Retag it and push to v2
4. Delete the image from local docker engine
5. Repeat this for all images in v1

This addresses the major issue of Disk space on the docker engine host not being sufficient to hold entire v1 registry.

We use S3 for storing our images and all the images cannot fit on a single host's disk.

Issue in Registry Migration

I tried to migrate the registry V1 to V2 our own registry. I'm facing an issue like

[INFO] Getting a list of images from cmtools

[ERROR] curl => API failure
[ERROR] Migration from v1 to v2 failed!

My command is :-

sudo docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=admin -e V2_REGISTRY=test -e NO_LOGIN=true -e V1_NO_LOGIN=true -e V2_NO_LOGIN=true -e USE_INSECURE_URL=true -e V1_USE_HTTP=true -e V2_USE_HTTP=true docker/migrator

Kindly help me to do the migration.

Thanks,

Kaleesastha

[ERROR] curl => API failure Migration from v1 to v2 failed!

Hello,
I'm trying to migrate from a 0.9.0 registry running in a docker, bound on port 5000, to a registry v2 running in a docker on port 5001.

I tried:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY="localhost:5000" \
    -e V2_REGISTRY="localhost:5001" \
    -e USE_HTTP=true \
    docker/migrator
[!!] Please login to localhost:5000:
Username: 
WARNING: login credentials saved in /root/.dockercfg.
Account created. Please see the documentation of the registry http://localhost:5000/v1/ for instructions how to activate it.

[INFO] Getting a list of images from localhost:5000

[ERROR] curl => API failure
[ERROR] Migration from v1 to v2 failed!

I also tried using NO_LOGIN=true with the same result.

What can I do ?

Add support for registry without auth

Add support for a v1 registry without auth

Will the script work by using Docker version 1.10.2?

The newest docker version creates a cfg file under .docker/config.json, while the script searches the user information under .docker.cfg file.

I have to do migration of a V1 registry to a v2 registry by using the new client version. This script seems to use the old client.

Does the migrator works for V1 -> Docker Trusted registry 1.2.0

The tool complains that [ERROR] v2 registry (http://r10a-docker7c.mitre.org) is not available

Add cleanup on local docker engine

Add cleanup of images pulled to local docker engine

Very long migrations fail

If your migration takes more than 12 hours, the ECR token expires and subsequent pushes will fail.

If search module is not enabled the migrator doesn't work

moby/moby#10906
https://github.com/docker/docker-registry#search-engine-options

If you don't have search enabled the migrator script will return an empty list of images as the list of images you have to migrate over to the new repo.

Add support for Docker Hub > v2 registry

Add support to migrate from Docker Hub organization/user namespace to a v2 registry

Problem with v2 availability check when using ERROR_ACTION=abort

When ERROR_ACTION=abort is set and the v2 availability check is failing, it will get stuck in an infinite loop for as long as the check fails. This should error out on first failure when ERROR_ACTION=abort is set.

migrator not able to pull from v1 registry

I am attempting to migrate images from a 0.8.1 registry running on a rhel 6.5 server with docker engine 1.4 to a v2 registry running on a rhel 6.7 server with docker engine 1.7.1. I am able to pull the list of images but when the script tries to pull the images I get an error response from the daemon : invalid registry endpoint <v1_registry>/v0/:unable to ping registry endpoint.
I get the same error when trying to pull from an individual command. I noticed the registry adds https in front of the registry name even though I have added the --insecure-registry flag on both machines.
Here is my command:

  sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=<v1_registry> -e V2_REGISTRY=<v2_registry> -e ERROR_ACTION=prompt -e NO_LOGIN=true -e USE_HTTP=true docker/migrator

Any help will be greatly appreciated.