rverchere / gitlab-migrator

Scripts to help migrate gitlab from one instance to another (users, groups and projects)

Gitlab Export/Import

Some scripts to export and import groups, projects and users from/to a gitlab instance.

You need some python dependencies to make it work:

• python-gitlab

Note: this script as not been tested so much, enhancements are welcome!

Group and Projects

Export

The gitlab-export.py script exports a group wih all its projects and subprojects, given a group name. You must be owner of the group you want to export.

When using it, it generates a folder in exports/ with the name of the group, plus:

• _group folder containing all groups information
• GROUP_NAME.tgz: group name archive
• subgroup\_projects\ folders containing all projects information
• PROJECT_PATH__PROJECT_NAME.tgz: project archive, given the name and path

Note: the folders are important, as every one created is associated to sub project

Usage

usage: gitlab-export.py [-h] -H HOSTNAME -t TOKEN -g GROUP [-o OUTPUT] [-d DELAY]

Gitlab project exporter

optional arguments:
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname HOSTNAME
                        hostname of gitlab server
  -t TOKEN, --token TOKEN
                        private token
  -g GROUP, --group GROUP
                        root group
  -o OUTPUT, --output OUTPUT
                        output folder
  -d DELAY, --delay DELAY
                        delay between two download tentative

Import

The gitlab-import.py script import a group wih all its projects and subprojects, given a root group name. The group will be created under this one.

It uses same folder architecture than the export script.

Usage

usage: gitlab-import.py [-h] -H HOSTNAME -t TOKEN [-r ROOT] -i INPUT [-d DELAY]

Gitlab project importer

optional arguments:
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname HOSTNAME
                        hostname of gitlab server
  -t TOKEN, --token TOKEN
                        private token
  -r ROOT, --root ROOT  root group (must exists). Must not end with '/'
  -i INPUT, --input INPUT
                        input path. Must end with '/'
  -d DELAY, --delay DELAY
                        delay between two download tentative

Users

Export

The gitlab-users-export.py exports users in a JSON format to a file, with the minimum required fields

Usage

usage: gitlab-users-export.py [-h] -H HOSTNAME -t TOKEN -o FILENAME

Gitlab users exporter

optional arguments:
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname HOSTNAME
                        hostname of gitlab server
  -t TOKEN, --token TOKEN
                        private token
  -o FILENAME, --output FILENAME
                        filename

JSON Result

    {
        "email": "[email protected]",
        "name": "Remi",
        "username": "Verchere",
        "bio": "",
        "linkedin": "",
        "twitter": "",
        "organization": null,
        "reset_password": true
    }

Import

The gitlab-users-import.py import users from a JSON file, with the minimum required fields

Usage

usage: gitlab-users-import.py [-h] -H HOSTNAME -t TOKEN [-i INPUT]

Gitlab users exporter

optional arguments:
  -h, --help            show this help message and exit
  -H HOSTNAME, --hostname HOSTNAME
                        hostname of gitlab server
  -t TOKEN, --token TOKEN
                        private token
  -i INPUT, --input INPUT
                        input file

License

Apache-2.0 License

gitlab-migrator's Issues

Import hangs after internal server error

Thank you for the helpful scripts.

Whilst using gitlab-import.py I experienced an internal server error that did not raise an exception, but meant that the project did return an import_status of failed. This status isn't handled by the line

gitlab-migrator/gitlab-import.py

Line 26 in 8fd4c8f

while project_import.import_status != 'finished':

so I hung inside this loop.

I added a quick modification to attempt another import_project with overwrite=True after a failed status. This resulted in an exception breaking me out of the loop, but did not import the project.

export error

Hello,

After an succesfull export i have trying to reinport it. I have just changed group name g1 to g4 in foldze
python gitlab-import.py --hostname https://gitlab.com --token TOKEN --input folderi2/ --root g2pcs

Import folderi2/ in parent group g2pcs
Parent group g2pcs with id 83864468 found
Importing g4 to folderi2 with parent_id 83864468
Importing Project 'p1' to 'p1' in 'g2pcs/folderi2' (folderi2/_projects)
Importing Project 'p1' to 'p1' in 'g2pcs/folderi2/s1g4' (folderi2/s1g4_projects)
Traceback (most recent call last):
File "C:\python\python379\lib\site-packages\gitlab\exceptions.py", line 336, in wrapped_f
return f(*args, **kwargs)
File "C:\python\python379\lib\site-packages\gitlab\v4\objects\projects.py", line 904, in import_project
"/projects/import", post_data=data, files=files, **kwargs
File "C:\python\python379\lib\site-packages\gitlab\client.py", line 1028, in http_post
**kwargs,
File "C:\python\python379\lib\site-packages\gitlab\client.py", line 797, in http_request
response_body=result.content,
gitlab.exceptions.GitlabHttpError: 404: 404 Namespace Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "gitlab-import.py", line 128, in
import_projects_from_groups(args.root, args.input)
File "gitlab-import.py", line 53, in import_projects_from_groups
import_projects(r_group, r_path)
File "gitlab-import.py", line 23, in import_projects
output = gl.projects.import_project(file=p_file, name=p_name, path=p_path, namespace=p_namespace)
File "C:\python\python379\lib\site-packages\gitlab\exceptions.py", line 338, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabImportError: 404: 404 Namespace Not Found

Could you help me
Thanks

certificat error [SSL: CERTIFICATE_VERIFY_FAILED] gitlab-export.py

Hi,
How could i avoid these error on gitlab-export.py
requests.exceptions.SSLError: HTTPSConnectionPool(host='gitlab.seflhosted.subpart', port=443): Max retries exceeded with url: /api/v4/groups (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

Thanks

export error

Hi

I have tried to export group and subfolder content from gitlab.com without succes and without error message
python gitlab-export.py --hostname https://gitlab.com --group 83830994 --token MONTOKEN --output foldero1
=> I don't found folder on my local drive ?

Regards
Philippe