jimhester / test-glfs

Example of Git LFS on GitHub, with a GitLab LFS backend

GitHub's implementation of git lfs is unfortunately hampered by a few major problems. The pricing model makes it challenging to use for popular open source projects, and as of 2015 it did not work with public forks of repositories. GitLab also provides an implementation of git lfs and they also provide as far as I can tell free hosting of files.

This repository demonstrates using GitLab to host your git lfs files, while continuing to push commits to GitHub as normal, with an option to mirror the repository on GitLab as well.

The key part to getting this working is when pushing commits to GitHub you will be prompted for a username and password when the lfs files are pushed. This will be crendentials for your Gitlab account, even if you are pushing to the GitHub remote.

Setup git lfs globally

Follow the directions at https://git-lfs.github.com/, should simply be running git lfs install after installing the command line extension.

New repository setup

1. Add regular files, git lfs tracked files as usual.
2. Add an .lsconfig entry for the gitlab lfs url. Replace user and repo with your values respectively. git config -f .lfsconfig lfs.url https://gitlab.com/user/repo/info/lfs
3. Create a new GitLab project
4. Add the GitLab project as a gitlab remote git remote add gitlab [email protected]:user/repo.git
5. Push to the gitlab remote first, to setup git lfs there. git push gitlab master
6. Create a new GitHub project.
7. add it as origin remote. git remote add origin [email protected]:user/repo.git
8. STOP, before pushing to GitHub you need to do one of two things. If your GitLab account is authenticated by a simple password you should be prepared to enter that. Otherwise you need to create a personal access token with api (read/write) access to use as a password for the upcoming step.
9. Push to the GitHub remote, this will then prompt you for a username and password. This is your GitLab username and password / PAT generated in the last step. git push gitlab master Username: user Password: xyz1234
10. Interact with GitHub normally from here on out! You can optionally set the GitLab to mirror the GitHub project from this point at https://gitlab.com/jimhester/test-glfs/settings/repository.

Existing GitHub repository setup

1. Track the files with git lfs tracked files you want to use.
2. Add an .lsconfig entry for the gitlab lfs url. Replace user and repo with your values respectively. git config -f .lfsconfig lfs.url https://gitlab.com/user/repo/info/lfs
3. Import your GitHub project to GitLab, (do not set up mirroring initially).
4. Add the GitLab project as a gitlab remote git remote add gitlab [email protected]:user/repo.git
5. Push to the gitlab remote, to setup git lfs there. git push gitlab master
6. STOP, before pushing to GitHub you need to do one of two things. If your GitLab account is authenticated by a simple password you should be prepared to enter that. Otherwise you need to create a personal access token with api (read/write) access to use as a password for the upcoming step.
7. Push to the GitHub remote, this will then prompt you for a username and password. This is your GitLab username and password / PAT generated in the last step. git push gitlab master Username: user Password: xyz1234
8. Interact with GitHub normally from here on out! You can optionally set the GitLab to mirror the GitHub project from this point at https://gitlab.com/jimhester/test-glfs/settings/repository.

test-glfs's Issues

error: batch response: Repository or object not found

I'm trying to implement this for use in my on work, but it keeps failing when I try to push to GitLab. The relevant repos are https://github.com/jeffreyhanson/test-gitlab-lfs and https://gitlab.com/jeffreyhanson/test-gitlab-lfs. I'm using Ubuntu Xenial, and the relevant commands/outputs are below:

xxxx:~/GitHub/test-gitlab-lfs$ git --version

 git version 2.7.4

xxxx:~/GitHub/test-gitlab-lfs$ git push gitlab master

Git LFS: (0 of 1 files) 0 B / 2 B                                                                           

batch response: Repository or object not found: https://gitlab.com/jeffreyhanson/test-gitlab-lfs/info/lfs/objects/batch

Check that it exists and that you have proper access to it

error: failed to push some refs to 'http://gitlab.com/jeffreyhanson/test-gitlab-lfs'

It looks like git lfs isn't set up on GitLab, but I can't find anything under the repo's settings to do this. Is there a specific file in the repo that I need to push first to initialize lfs in the GitLab repo?

testing on self-hosted gitlab: cannot push to github?

@jimhester thanks for this, looks really cool, just testing it out now on a self-hosted gitlab instance (https://gitlab.carlboettiger.info/cboettig/data).

For some reason, instead of getting prompted for my gitlab user name / password when I go to push to github, I'm just given the error:

$ git push --set-upstream origin master
Git LFS: (0 of 1 files) 0 B / 273.37 MB                                                                                     
batch response: Repository or object not found: https://gitlab.carlboettiger.info/cboettig/data/info/lfs/objects/batch
Check that it exists and that you have proper access to it

Any ideas? I think it should be prompting for a user/password for the the gitlab, but instead it seems to just try to push directly and fails.

AFIK lfs is working as expected on gitlab, and I can still git push gitlab master just fine.

(Also, wasn't clear from the your directions or maybe I missed it: do you git commit the .lfsconfig file or .gitignore it? (I'm guessing the latter, since I think committing it would break things for other users who might work on their own forks and who didn't have write access to the gitlab account? not quite clear how that works).