pote / gpm

Barebones dependency manager for Go.

Go Package Manager

Go Package Manager (or gpm, for short) is a tool that helps achieve reproducible builds for Go applications by specifying the revision of each external Go package that the application depends on.

Being simple and unobstrusive are some of the most important design choices for gpm: go get already provides a way to fetch dependencies, and relies on versions control systems like Git to do it, gpm adds the additional step of setting each dependency repo to the desired revision, neither Go or your application even know about any of this happening, it just works.

To achieve this, gpm uses a manifest file which is assumed to be called Godeps (although you can name it however you want), running gpm fetches all dependencies and ensures each is set to a specified version, down to revision level.

Basic usage

For a given project, running gpm in the directory containing the Godeps file is enough to make sure dependencies in the file are fetched and set to the correct revision.

However, if you share your GOPATH with other projects running gpm each time can get old, my solution for that is to isolate dependencies by manipulating the GOPATH, see the workspaces section for details.

You can see gpm in action under this workflow in the following gif:

Installation options

In OSX with Homebrew

$ brew install gpm

In Arch Linux - AUR

$ yaourt -S go-gpm

or

$ packer -S go-gpm

Caveat: you'll use go-gpm instead of just gpm in the command line, as there is a general purpose linux package under that name already.

Manually with a one-liner

Latest stable release:

$ wget https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm && chmod +x gpm && sudo mv gpm /usr/local/bin

Manually on *nix using the makefile.

$ git clone https://github.com/pote/gpm.git && cd gpm
$ git checkout v1.4.0 # You can ignore this part if you want to install HEAD.
$ ./configure
$ make install

Use directly from GitHub

As gpm is a bash script you can always use it directly from GitHub via wget or curl, this is particularly useful for CI servers and other automated environments.

## With wget
$ wget -qO- https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm | bash

## With cURL
$ curl -s https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm | bash

The Godeps file

gpm expects you to have a file called Godeps in the root of your Go application in the format <import path> <tag/revision>.

Once this file is in place, running the gpm tool will download those packages and check out the specified versions.

Packages

You can specify packages with the <import path> <version> format, where version can be a revision number (a git/bazaar/mercurial/svn revision hash) or a tag.

$ ls .
Godeps  foo.go  foo_test.go

$ cat Godeps
github.com/nu7hatch/gotrail               v0.0.2
github.com/replicon/fast-archiver         v1.02
launchpad.net/gocheck                     r2013.03.03   # Bazaar repositories are supported
code.google.com/p/go.example/hello/...    ae081cd1d6cc  # And so are Mercurial ones

When specifying your dependencies please keep in mind how gpm and the go tool operate: importing a package is setting the version of a cloned repo to a specific revision, so if you are importing several subpackages that are hosted under the same repo only one of them (the top level) should be specified in your Godeps file, in cases where there are no Go packages in the root of the dependency repository you can get Go to fetch the code anyway by appending /... to the import path (see last line in the example above)

Comments

The Godeps file accepts comments using a # symbol. Everything to the right of a # will be ignored by gpm, as well as empty lines.

Extensibility

As a convention comments can be used to specify lines that gpm core should ignore but are instead intended to affect how a given gpm plugin behaves.

For example: a hypothetical gpm-track plugin that makes sure a given package is always updated to its last possible version would leverage a line like this one:

#[gpm-track] github.com/nu7hatch/gotrail

This convention makes the Godeps file format extensible, just as with plugins this can help identify common needs that might later on be merged into core without having to sacrifice code simplicity in order to explore new features.

Private Repos

Both gpm and go get support using private GitHub repositories! Here's what you need to do in order for a specific machine to be able to access them:

• Generate a GitHub access token by following these instructions.
• Add the following line to the ~/.netrc file in your home directory.

machine github.com login <token>

You can now use gpm (and go get) to install private repositories to which your user has access! :)

Completeness

Any dependency not specified in the Godeps file will be installed by the Go tool to whatever revision the master branch of its hosting repository is pointing at that given moment, as reproducibility is the main goal of gpm it is suggested to be exhaustive and list all your dependencies in the file, with a specific revision.

Do it once, reproduce it anytime, it pays off.

Commands

gpm has the following commands:

$ gpm             # Same as 'install'.
$ gpm get         # Parses the Godeps file, gets dependencies and sets them
                  # to the appropriate version but does not install them.
$ gpm install     # Parses the Godeps file, installs dependencies and sets
                  # them to the appropriate version.
$ gpm version     # Outputs version information
$ gpm help        # Prints this message

Plugins

As of version v1.1.1 gpm supports plugins, the intent of which is the ability to add powerful non-core features to gpm without compromising the simplicity of its codebase.

The way gpm plugin works is simple: whenever an unknown command is passed into gpm it will look for an executable in your $PATH called gpm-<command> and if it exists it will run it while passing all extra arguments to it, simple yet powerful.

This brings a lot to the table: plugins can be written in anything, they can be Go binaries, bash scripts, Ruby gems, Python packages, you name it. gpm wants to make it easy for you to extend it. :)

Installing plugins through Homebrew

I maintain a repository with homebrew formulae for gpm plugins that you can add to your system with the brew tap command:

$ brew tap pote/gpm_plugins

After you've done this you can install plugins as you would with any other homebrew packge.

$ brew install gpm-bootstrap

Known Plugins

If you have written a gpm plugin and want it included please send a pull request to the repo! I love how people have taken to explore possible features using plugins so if you've written one there is about a 99% chance I will include it here. :)

Name and Link	Author	Short Description	Type
gpm-bootstrap	pote	Creates an initial Godeps file	official
gpm-git	technosophos	Git management helpers	third party
gpm-link	elcuervo	Dependency vendoring	third party
gpm-local	technosophos	Usage of local paths for packages	third party
gpm-prebuild	technosophos	Improves building performance	third party
gpm-all	pote	Installs multiple sets of deps	official
gpm-lock	zeeyang	Lock down dependency versions	third party

There is no real difference on official/third party plugins other than the willingness of the gpm core team to support each, plugins labeled as third party will be supported (or not) by their authors.

Workspaces

A question that comes up time and time again is how to handle different workspaces for Go projects.

This question has many answers, and gpm should be compatible with most of them. My personal way to solve it is to have an environment file per project, which I use to manipulate the GOPATH whenever I switch to a given project.

$ cd my_project
$ cat .env
export GOPATH="$PWD"/.dependencies:"$PWD"
$ source .env

After sourcing the env file (in which I usually keep other project-specific configuration variables, like database urls, secret keys, etc) the active GOPATH is a local one: this means that I don't need to run gpm again to make sure my dependencies are in the correct version and there is no danger of conflicting dependency versions across different projects. Everything is isolated and can be easily wiped clean if needed.

Further Reading

The creator for the gpm-git and gpm-local and an alternative package manager called Glide wrote a fantastic blog post explaining the usage and rationale of gpm, it sums up explanations for several of the design decisions behind both tools.

Contributing

Lots of people have contributed to make gpm what it is today, if you want to take your time to play around with the code please do so! Opening issues on bugs, feature requests or simple food for thought are a great way to contribute, if you send a pull request please be a good citizen and do things in a tidy manner.

• Create a feature branch with a meaningful name.
• Make sure your commit messages and PR comments are informative.
• Write a test for your feature if applicable.
• Always remember to run the test suite with make test before comitting.

Either way, thank you very much for any form of contribution, even if a patch ends up not being merged the fact that it was sent and forced us to think about it is a contribution in itself.

License

Released under MIT License, check LICENSE file for details.

Authorship/Inspiration/Hugs

This tool is inspired by Ruby's dep gem - authored by @cyx and @soveran, big thanks to them and to all the contributions made by the many wonderful people in our contributors page.

gpm is maintained by @pote and @elcuervo.

gpm's Issues

Parallel execution of 'go get' may lead to an error

Running go get in parallel for packages that depend on each other sometimes(?) leads to an error.

Here is an example. I have a Godeps file with the following content:

golang.org/x/oauth2 7fdf09982454086d5570c7db3e11f360194830ca
golang.org/x/net/context 242b6b35177ec3909636b6cf6a47e8c2c6324b5d

When running gpm get, it reports an error:

>> Getting package golang.org/x/oauth2
>> Getting package golang.org/x/net/context
# cd /Users/me/tmp/src/golang.org/x/net; git pull --ff-only
From https://go.googlesource.com/net
 * [new branch]      master     -> origin/master
 * [new branch]      release-branch.go1.6 -> origin/release-branch.go1.6
 * [new branch]      release-branch.go1.7 -> origin/release-branch.go1.7
 * [new branch]      release-branch.go1.8 -> origin/release-branch.go1.8
 * [new branch]      release-branch.go1.9 -> origin/release-branch.go1.9
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream-to=origin/<branch> master

package golang.org/x/net/context: exit status 1
>> Failed getting package golang.org/x/oauth2
>> Setting golang.org/x/oauth2 to version 7fdf09982454086d5570c7db3e11f360194830ca
>> Setting golang.org/x/net/context to version 242b6b35177ec3909636b6cf6a47e8c2c6324b5d

When issuing the go getcommands sequentially, no error is reported.

Error sometimes with `gpm install` on existing projects

I just noticed this a few times today. It looks like sometimes a race condition happens:

>> Setting github.com/lann/builder to version
# cd /Users/mbutcher/Code/Go/src/bitbucket.org/mobiplug/sugarmill/.godeps/src/github.com/lann/builder; git checkout master
fatal: Unable to create '/Users/mbutcher/Code/Go/src/bitbucket.org/mobiplug/sugarmill/.godeps/src/github.com/lann/builder/.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
package github.com/lann/squirrel
    imports github.com/lann/builder: exit status 128

I am having trouble reproducing this, but every once in a while I see the same error (on different projects each time).

Path names are not escaped

If the path to gpm's working directory contains white spaces, I see the following errors:

>> Setting github.com/mattn/go-sqlite3 to version 
/usr/local/bin/gpm: line 66: cd: /Users/codewerft/Codewerft/Kunden/03: No such file or directory
/usr/local/bin/gpm: line 66: cd: /Users/codewerft/Codewerft/Kunden/03: No such file or directory

The working directory in the case above was /Users/codewerft/Codewerft/Kunden/03 - XYZ/Entwicklung/reactor.backend.

Add 'vendoring' functionality

In ruby I'd use tools like the gs gem to create a hidden directory inside the repository of a project containing all the libraries needed.

Adding something like this in gpm would be trivial, the command would create a .godeps directory and switch the GOPATH variable to that directory so when gpm is run normally all the packages are installed there.

Thoughts? A good name for the command? gpm isolate maybe?

Packages with repo on different level are not checked out correctly

gpm assumes that the source code repository is always located at a specific level, i.e. when downloading package host/foo/bar the repo has to be in the bar directory. Although this is true for many project (e.g. hosted on github.com), there are exceptions.

Here is an example. I have a Godeps file with the following content:

cloud.google.com/go/compute/metadata v0.7.0

When running gpm get, it shows the following output:

>> Getting package cloud.google.com/go/compute/metadata
>> Setting cloud.google.com/go/compute/metadata to version v0.7.0

But when I switch to the corresponding directory and issue a git status, I can see that the repo is on branch master (and not on v0.7.0).

This is due to the fact that the Git repository is located at cloud.google.com/go, i.e. one level above the expected one.

MinGW support

One of our devs is using Windows / msys for his go development. gpm doesn't seem to work with the bash, since it doesn't support full process redirection.

(This is mainly a placeholder to remind me to work on this.)

If go get fails, gpm continues - it should fail

Hello.

Could you please exit gpm with go get's return code if it fails to get a package?

GPM support for Windows

Hi
Would like to know if GPM is supported for Windows ?

Thanks
Tank

Dev vs Production Deps?

gpm supports custom Godeps naming, so you can have Godeps and Godeps.dev (or whatever), but is there any desire to see this integrated tighter? Perhaps in a single Godeps file, you could define production / development deps?

github.com/foo/bar v1
gitlab.com/foo/bar v2

-dev
github.com/baz/some-test-suite

I dislike adding complexity to the simple Godeps, but i also feel a standardized way to handle devdeps would be handy.

Generate the dependencies file by scanning the git repo

I want to generate the file in one simple command

Easier installation and plugin management

In my opinion, gpm is being too reliant on home-brew for distributing itself to users. If gpm is rewritten in go, we can distribute it easily as a binary. And also a lot of other features can be added easily.

I am pledging my time to do this alongside with you.

git: You are not currently on a branch

I found gpm and wanted to give it a try. It looks like it does not support pinning dependencies based on commits. I have this in my Godeps file.

github.com/alicebob/miniredis               bb985e34f0942b1d770bf96e87f830cf38804abe

This is what I get.

# cd /path/to/workspace/src/github.com/alicebob/miniredis; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package github.com/alicebob/miniredis: exit status 1

Is this intended? Can we make it work?

Rewrite test suite with BATS

We currently use an assert script that somebody wrote and a custom script to execute all tests, it's kind of messy, I'd prefer if we had something more polished, enter BATS

I'll do this at some point if I have the time and the drive, if not I welcome anyone who wants to take a crack at doing it!

Simplify JSON parsing

Right now our two GitHub API calls do horrible things to parse the output, these should probably be simplified by using less piping and cuts and less commands overall so they are more maintainable.

Doesn't set revision when using subpackage of repository

For example, if my Godeps file looks like

github.com/dotcloud/docker/pkg/namesgenerator c9f216d

the namesgenerator package will still be at master since that directory doesn't contain a .git folder. gpm should traverse through the parent directories of the package until it find a version control system indicator, or fail if one doesn't exist.

Git Updating: Branches vs. Tags/Commits

I've been struggling with using GPM for trees that I want to keep updated to the latest commit on a particular branch.

Use case:

I want a repository to stay on the latest checkout on master, and each time I run gpm install I want it to update to the latest commit

If my Godeps file looks like this:

github.com/Masterminds/cookoo

Then master is checked out initially, and the tree is updated each time gpm install is run, but the repo is always pointed to whatever commit I got when I initially ran gpm install.

Same thing happens if I do this:

github.com/Masterminds/cookoo

The relevant line in GPM is this:

https://github.com/pote/gpm/blob/master/bin/gpm#L63

I had the same problem a while ago with gpm-git, and ended up doing this:

      # Figure out, based on ref, what type of reference this is.
      local vtype="commit"
      git show-ref -q "origin/$version" && vtype="branch"
      git show-ref -q "tags/$version" && vtype="tag"

      echo ">> Setting $package to $vtype $version"
      cd $install_path
      [ -d .git ] && git checkout -q "$version"

      # Handle case where branch changed. We need to get to the tip
      # of that branch.
      [ $vtype == "branch" ] && git merge --ff-only origin $version

It's far more verbose, but it seems to do the trick.

If I have the time this week, I will work up a patch and submit a pull request. Feel free, of course, to find a better way of doing it (or to simply say that this is out of scope for GPM).

As always, thanks for a great tool.

Support other VCSs

Some users have made public their interest in adding bazaar support, I'm opening this issue as a discussion forum for this and other VCSs we might want to add support to.

Do you want us to support svn? mercurial? bazaar? please state so in here and we'll plan our roadmap accordingly to the interest show on each. :)

simplify installing non-head depdendencies

I would be much easier if if gpm install would print the hash of library that gets installed.
So one could start off with no hash/tag, do the install and then set the deps to that.
Or even better:

gpm install github.com/golang/glog --save

which would add the dep (include the hash) to the Godep file.

gpm install crashes terminal

Regardless of if I use zsh or bash in xterm it just silently crashes it. Nothing in the logs just bam.

I just run:

source gpm in

I tried both the latest 1.3.0 tag as well as master to no avail.

Any idea what I can try to debug it?

Check integrity of Godeps file

This action should compare the Godeps packages with the go list command (which shows all imports) to see if there is anything missing in Godeps.

Maybe there could be a flag that also adds the latest release for all these packages to Godeps?

Add support to list outdated dependencies in the Godeps file

Something along the lines of bundle outdated that ruby has but for go. This would help a lot in terms of keeping up to date with the latest versions when using version constraints in the Godeps file.

How to set a svn address

How to set a svn address in Godeps file.
and set a username for svn login.
Please give some demo, thanks!

Errors from git commands when a pkg and its sub-pkg is in Godeps

We have a case where we need the top-level pkg of a given Go pkg for use in our application and also one of its sub-pkgs b/c it's a CLI. It seems gpm produces errors in output because it tries to git checkout the sub-pkg even though it's not a submodule but just a sub-directory in the parent pkg.

Fortunately, gpm seems to still work as the binary gets installed in $GOPATH/bin; however, it'd be nice if we could avoid or suppress errors in output from gpm to avoid confusion.

You can see a basic Godeps file here along with the errors from running gpm: https://gist.github.com/jpfuentes2/ad63ff939a87fdb3d92c

Also, I set $GOPATH and $GOBIN to a temporary directory for the test above and, again, the binary was installed along w/ the pkg.

Potential workarounds:

• Do not issue vcs checkout commands if it's a sub-pkg and it's not a submodule (or equivalent).
• Add a comment in the version placeholder to designate no checkout whereby the vcs will not attempt to checkout the sub-pkg.
• Punt and instruct users to move the sub-pkg to a Makefile or other build script using go get after a gpm run.

I don't know if any of these ideas are good but maybe something sparks.

gpm in Dependencies?

Does gpm install a dependence's gpm file? If not, what are your thoughts on this feature?

I realized the other day that if my dependencies also have specific version requirements, i need gpm to satisfy those. I considered writing a plugin for this, but this seems like something that should be core to gpm, in my opinion.

Thoughts?

Unable to install plugins

I get the following error when attempting to tap the pote/gpm_plugin repo

~ brew tap pote/gpm_plugins
==> Tapping pote/gpm_plugins
Cloning into '/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 10 (delta 3), reused 3 (delta 0), pack-reused 0
Unpacking objects: 100% (10/10), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-all.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-all.rb:6:in `<class:GpmAll>'
Please report this to the pote/gpm_plugins tap!
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-bootstrap.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-bootstrap.rb:6:in `<class:GpmBootstrap>'
Please report this to the pote/gpm_plugins tap!
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-git.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-git.rb:6:in `<class:GpmGit>'
Please report this to the pote/gpm_plugins tap!
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-link.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-link.rb:6:in `<class:GpmLink>'
Please report this to the pote/gpm_plugins tap!
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-local.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-local.rb:6:in `<class:GpmLocal>'
Please report this to the pote/gpm_plugins tap!
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-prebuild.rb
Calling Formula.sha1 is disabled!
Use Formula.sha256 instead.
/usr/local/Homebrew/Library/Taps/pote/homebrew-gpm_plugins/Formula/gpm-prebuild.rb:6:in `<class:GpmPrebuild>'
Please report this to the pote/gpm_plugins tap!
Error: Cannot tap pote/gpm_plugins: invalid syntax in tap!
~

Consider adding GO15VENDOREXPERIMENT support

In 1.5 the go command will read the environment variable GO15VENDOREXPERIMENT. If enabled, it will use the local "/vendor/" folder for dependencies. This replaces the need to manage GOPATH variables as dependencies can be fetched directly into the vendor folder and resolved there when built. This allows the GOPATH to not be modified. The contents of the vendor folder can be ignored.

Will you consider supporting this approach?

Dependency binaries not building?

I have a project with a dependency on github.com/smartystreets/goconvey, but when gpm installs the dependency the goconvey binary is not built. On the flipside, if you use go get github.com/smartystreets/goconvey the binary is created without issue.

So, with that said, is this intentional? And if so, is there a trick to getting the binary created as part of the gpm dependency resolution process?

List of companies using gpm

The rest of the 13Floor crew and me are building a beautiful site for gpm, it'll have documentation, lists of plugins and much more, it'd be fantastic to have a list of companies who are using it.

Do you use gpm and/or know of a company or project that does? Please add them to this issue! If there's enough people interested we should be able to get their logos into the site. ^_^

Avoid race condition on SVN packages

The following code makes sure no issues happen due to concurrently installing packages.

  [[ -e "$install_path/.git/index.lock" ||
     -e "$install_path/.hg/store/lock"  ||
     -e "$install_path/.bzr/checkout/lock" ]] && wait

https://github.com/pote/gpm/blob/master/bin/gpm#L45-L47

We need to add an extra line there for svn. This is low-hanging fruit, if nobody picks it up I'll do it soon. :)

Generate Godeps file

I was thinking about adding a gpm generate command to easily populate a Godeps file in an old project so you don't have to go file by file adding them.

The line:
go list -f '{{join .Deps "\n"}}' | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'

Thoughts?

Consider a different name

I'm looking at packaging gpm (for CentOS) so I can use it to build some go packages.

However, there is already a package named gpm:

 yum info gpm
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.openitc.uk
 * epel: mirror.vorboss.net
 * extras: centos.openitc.uk
 * fasttrack: centos.openitc.uk
 * updates: centos.openitc.uk
Available Packages
Name        : gpm
Arch        : x86_64
Version     : 1.20.7
Release     : 5.el7
Size        : 183 k
Repo        : base/7/x86_64
Summary     : A mouse server for the Linux console
URL         : http://www.nico.schottelius.org/software/gpm/
License     : GPLv2 and GPLv2+ with exceptions and GPLv3+ and Verbatim and Copyright only
Description : Gpm provides mouse support to text-based Linux applications like the
            : Emacs editor and the Midnight Commander file management system.  Gpm
            : also provides console cut-and-paste operations using the mouse and
            : includes a program to allow pop-up menus to appear at the click of a
            : mouse button.

Might you consider using a different name, eg. gopm?

Add Windows support

Handle private repos out-of-the-box

I've been looking for a good way to ship go packages with private dependencies short of Godeps' solution of putting them in the git source tree for the depending project. Would gpm consider a feature like this? Perhaps there's some modifier in the dependencies file that could indicate that it should be fetched over SSH?

Thanks for making gpm! We love it over at VSCO.

I am currently using gpm in part of deployment

I am currently using gpm in part of deployment for my go projects. I have my ci building out binaries before every push.

I often see errors like so

>> Getting package github.com/Shopify/sarama
# cd /Users/myusuf3/go/src/github.com/golang/protobuf; git pull --ff-only
error: Ref refs/remotes/origin/master is at efcaa340c1a788c79e1ca31217d66aa41c405a51 but expected deb4a5e3b15dea23f340a311eea995421845c356
error: Cannot lock the ref 'refs/remotes/origin/master'.
From https://github.com/golang/protobuf
 ! deb4a5e..efcaa34  master     -> origin/master  (unable to update local ref)
package github.com/prometheus/client_model/go
    imports github.com/golang/protobuf/proto: exit status 1
# cd /Users/myusuf3/go/src/golang.org/x/image; git pull --ff-only
fatal: unable to access 'https://go.googlesource.com/image/': Server aborted the SSL handshake
package golang.org/x/image/bmp: exit status 1
# cd /Users/myusuf3/go/src/golang.org/x/tools; git pull --ff-only
fatal: unable to access 'https://go.googlesource.com/tools/': Server aborted the SSL handshake
package github.com/maxbrunsfeld/counterfeiter
    imports github.com/maxbrunsfeld/counterfeiter/generator
    imports github.com/maxbrunsfeld/counterfeiter/model
    imports golang.org/x/tools/go/ast/astutil: exit status 1
# cd /Users/myusuf3/go/src/golang.org/x/net; git pull --ff-only
fatal: unable to access 'https://go.googlesource.com/net/': Server aborted the SSL handshake
package github.com/nlopes/slack
    imports golang.org/x/net/websocket: exit status 1

I need for the gpm to go smoothly and only exit 1 on actual failure ( it continues fine after that, but ci fails due to non zero exit value) and i am unable to understand issue clearly some ideas to help would awesome

Update docs to use `raw.githubusercontent.com` or add `--location` for curl

The new user content domains github started using is causing the curl command to not install anything from the Godeps file.

does not work:

$ curl -s https://raw.github.com/pote/gpm/v1.2.1/bin/gpm | bash

because it is not following the redirect:

$ curl -I  https://raw.github.com/pote/gpm/v1.2.1/bin/gpm
HTTP/1.1 301 Moved Permanently
Date: Mon, 28 Apr 2014 20:01:30 GMT
Server: Apache
Location: https://raw.githubusercontent.com/pote/gpm/v1.2.1/bin/gpm
Accept-Ranges: bytes
Via: 1.1 varnish
Age: 0
X-Served-By: cache-c100-CHI
X-Cache: MISS
X-Cache-Hits: 0
Vary: Accept-Encoding

works:

$ curl --silent --location https://raw.githubusercontent.com/pote/gpm/v1.2.1/bin/gpm | bash
>> Getting package github.com/beego/bee
>> Getting package github.com/astaxie/beego
>> Setting github.com/astaxie/beego to version a0dff9148a739d95632200ee16437d83e8d2017c
>> Setting github.com/beego/bee to version a24bc70e3dbcce159141d0106104b8b7cc617e17
>> All Done

The docs should be updated to use the new domain raw.githubusercontent.com or include the --include arg to curl

Extensions to the Godeps file format?

I'd love to see some extensions to the Godeps format that can accomodate specifying the repository (à la technosophos/gpm-git) and the name of the local package (à la elcuervo/gpm-link and technosophos/gpm-local).

It'd also be cool to support versioning ranges (e.g. in this major version, after this minor version, stable releases after this revision, etc.), though I haven't seen anyone do that yet. Just thinking about the future :)

Here's an idea I was kicking around:

# My name is Godeps-ng

# Local package is specified with a dot. Looks similar to Go's format for
# importing a package into the local namespace
. example.com/michaelmaltese/this-package-name

# Simple dependencies are as before
github.com/mattn/go-sqlite3 71e406b

# Specify repository URI, like gpm-git. I include the extra part after the
# colon to indicate where in the repository the package exists, which
# looks similar to the syntax for the `scp` command and probably
# wouldn't mess up any VCS URIs.
github.com/dotcloud/docker/pkg/namesgenerator c9f216d \
  https://github.com/dotcloud/docker.git:pkg/namesgenerator

Thoughts @pote @technosophos @elcuervo anyone else?

gpm should detect duplicated lines with different sha1 in Godeps file

Currently when I specify some dependency twice in Godeps file (each time with different version) then gpm chooses only one and doesn't report any error.
This can cause a lot of trouble to unwary developer.

gpm should fail in such case, listing dependencies that are duplicated.
I'm opting for failing even when both lines are exactly the same (specify the same dependency version).

Installing dependencies

Since you're using the -d flag on go get (https://github.com/pote/gpm/blob/master/bin/gpm#L49), downloaded packages are not installed by default.

I've seem there is a plugin for "prebuilding" dependencies, and what it does is to call go install on each dependency.

Why not just add another loop around this line (https://github.com/pote/gpm/blob/master/bin/gpm#L69) to call go install for each dependency?

It seems like something you would expect from a dependency manager, instead of having to install a plugin and remember to run another command every time.

Any thoughts?

Asking for github passwords for fully open deps

Surely this is not the intent nor even needed?

Fatal errors installing non-root packages

I'm trying to install the protobuf support with gpm, but the actual go packages aren't in the root of their repo. github.com/golang/protobuf is the repo root, but the actual packages are github.com/golang/protobuf/proto and github.com/golang/protobuf/protoc-gen-go

If I put the actual packages in my Godeps file, I get this error:

>> Getting package github.com/golang/protobuf/proto
>> Getting package github.com/golang/protobuf/protoc-gen-go
# cd .; git clone https://github.com/golang/protobuf /projects/go-ext/src/github.com/golang/protobuf
fatal: destination path '/projects/go-ext/src/github.com/golang/protobuf' already exists and is not an empty directory.
package github.com/golang/protobuf/protoc-gen-go: exit status 128
>> Setting github.com/golang/protobuf/proto to version efd7476481382c195beb33acd8ec2f1527167fb4
>> Setting github.com/golang/protobuf/protoc-gen-go to version efd7476481382c195beb33acd8ec2f1527167fb4
fatal: Unable to create '/projects/go-ext/src/github.com/golang/protobuf/.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
>> Building package github.com/golang/protobuf/proto
>> Building package github.com/golang/protobuf/protoc-gen-go
>> All Done
-e 
Success

If I put the root repo in the file, I get this:

>> Getting package github.com/golang/protobuf
package github.com/golang/protobuf
    imports github.com/golang/protobuf
    imports github.com/golang/protobuf: no buildable Go source files in /projects/go-ext/src/github.com/golang/protobuf
>> Setting github.com/golang/protobuf to version efd7476481382c195beb33acd8ec2f1527167fb4
>> Building package github.com/golang/protobuf
can't load package: package github.com/golang/protobuf: no buildable Go source files in /projects/go-ext/src/github.com/golang/protobuf

EDIT: Despite the fatal error messages in the output, protoc-gen-go does get installed, but it is certainly ugly and unexpected to see fatals in the output.

installation fails if already installed in `src`

$ find .godeps
.godeps
.godeps/bin
.godeps/pkg
.godeps/src

$ cat Godeps 
github.com/golang/glog
github.com/gorilla/mux
golang.org/x/oauth2

$ echo $GOPATH
/Users/tcurdt/..../myproject/.godeps:/Users/tcurdt/..../myproject

$ gpm install
>> Getting package github.com/golang/glog
>> Getting package github.com/gorilla/mux
>> Getting package golang.org/x/oauth2
>> Setting github.com/golang/glog to version 
>> Setting golang.org/x/oauth2 to version 
/usr/local/bin/gpm: line 66: cd: /Users/tcurdt/..../myproject/.godeps/src/github.com/golang/glog: No such file or directory
>> Setting github.com/gorilla/mux to version 
/usr/local/bin/gpm: line 66: cd:  /Users/tcurdt/..../myproject/.godeps/src/golang.org/x/oauth2: No such file or directory
/usr/local/bin/gpm: line 66: cd:  /Users/tcurdt/..../myproject/.godeps/src/github.com/gorilla/mux: No such file or directory
>> Building package github.com/golang/glog
>> Building package github.com/gorilla/mux
>> Building package golang.org/x/oauth2
>> All Done

$ find .godeps
.godeps
.godeps/bin
.godeps/pkg
.godeps/src

Extensions to the Godeps file format?

I'd love to see some extensions to the Godeps format that can accomodate specifying the repository (à la technosophos/gpm-git) and the name of the local package (à la elcuervo/gpm-link and technosophos/gpm-local).

It'd also be cool to support versioning ranges (e.g. in this major version, after this minor version, stable releases after this revision, etc.), though I haven't seen anyone do that yet. Just thinking about the future :)

Here's an idea I was kicking around:

# My name is Godeps-ng
# Local package is specified with a dot. Looks similar to Go's format for importing a package into the local namespace
. example.com/michaelmaltese/this-package-name
# Simple dependencies are as before
github.com/mattn/go-sqlite3 71e406b
# Specify repository URI, like gpm-git. I include the extra part after the colon to indicate where in the repository the package exists, which looks similar to the syntax for the `scp` command and hopefully shouldn't mess up any VCS URIs.
github.com/dotcloud/docker/pkg/namesgenerator c9f216d https://github.com/dotcloud/docker.git:pkg/namesgenerator

Thoughts @pote @technosophos @elcuervo anyone else?

Accept Godeps file parameter

In some cases it is necessary to be able to have different Godeps files - you could have Godeps and Godeps.dev for development/testing dependencies - so gpm needs to accept a file parameter at execution time.

Add -H flag

The -a <package flag adds the latest release from github to the Godeps file.

Sometimes there are no available releases though (in projects that don't use git tags), the -H flag should work alongside with the -a flag and make it add the HEAD sha to the Godeps file instead of the last release.

Clone error when dependencies from the same repo

I get

fatal: Unable to create '/Users/mmazurskiy/IdeaProjects/config-store/.godeps/src/github.com/atlassian/goamz/.git/index.lock': File exists.

with the following Godeps file:

github.com/atlassian/goamz/aws
github.com/atlassian/goamz/s3

This is similar to #35 but probably different issue. Is this a supported usecase o I'm doing something wrong?

POSIX shell support

It would be good to not have bash as a dependency and to use #!/usr/bin/env sh instead for cross platform support.

I've made some headway with this which I'll paste below but it's beyond my shell scripting skills.

#!/usr/bin/env sh

set -eu

## Functions/
usage() {
cat << EOF
SYNOPSIS

    gpm leverages the power of the go get command and the underlying version
    control systems used by it to set your Go dependencies to desired versions,
    thus allowing easily reproducible builds in your Go projects.

    A Godeps file in the root of your Go application is expected containing
    the import paths of your packages and a specific tag or commit hash
    from its version control system, an example Godeps file looks like this:

    $ cat Godeps
    # This is a comment
    github.com/nu7hatch/gotrail         v0.0.2
    github.com/replicon/fast-archiver   v1.02   #This is another comment!
    github.com/nu7hatch/gotrail         2eb79d1f03ab24bacbc32b15b75769880629a865

    gpm has a companion tool, called [gvp](https://github.com/pote/gvp) which
    provides vendoring functionalities, it alters your GOPATH so every project
    has its own isolated dependency directory, its usage is recommended.

USAGE
      $ gpm             # Same as 'install'.
      $ gpm install     # Parses the Godeps file, installs dependencies and sets
                        # them to the appropriate version.
      $ gpm version     # Outputs version information
      $ gpm help        # Prints this message
EOF
}

is_in_use() {
  [ -e "$1/.git/index.lock" -o -e "$1/.hg/store/lock"  -o -e "$1/.bzr/checkout/lock" ]
}

# Iterates over Godep file dependencies and sets
# the specified version on each of them.
set_dependencies() {
  local deps=$(sed 's/#.*//;/^\s*$/d' < $1) || echo ""

  echo "$deps" | while read package version; do
    (
      echo ">> Getting package "$package""
      go get -u -d "$package"
    ) &
  done
  wait

  echo "$deps" | while read package version; do
    (
      local pkg_path=$(echo "$package" | awk -F/ '{print $1"/"$2"/"$3}')
      local install_path="${GOPATH%%:*}/src/${pkg_path%%/...}"
      echo ">> Setting $package to version $version"
      cd $install_path
      is_in_use $install_path && wait

      [ -d .bzr ] && bzr revert   -q -r   "$version"
      [ -d .git ] && git checkout -q      "$version"
      [ -d .hg  ] && hg update    -q      "$version"
      [ -d .svn ] && svn update   -r      "$version"
    ) &
  done
  wait
  echo ">> All Done"
}
## /Functions

## Command Line Parsing
case "${1:-"install"}" in
  "version")
    echo ">> gpm v1.3.1"
    ;;
  "install")
    deps_file="${2:-"Godeps"}"
    [ -r "$deps_file" ] || (echo ">> $deps_file file does not exist." && exit 1)
    (which go > /dev/null) ||
      ( echo ">> Go is currently not installed or in your PATH" && exit 1)
    set_dependencies $deps_file
    ;;
  "help")
    usage
    ;;
  *)
    ## Support for Plugins: if command is unknown search for a gpm-command executable.
    if command -v "gpm-$1" > /dev/null
    then
      plugin=$1 &&
      shift     &&
      gpm-$plugin $@ &&
      exit
    else
      echo ">> No command 'gpm $1'"
      usage && exit 1
    fi
    ;;
esac

generate Godeps

It will be nice if gpm can generate Godeps file.

Disregard

EDIT: Opened in the wrong place, removes irrelevant body.

Add -h usage output.

We should have a -h flag that outputs the usage of the script and all flags.

Switch flags to commands

The -f flag make sense as well as the -H flag, but -a should probably be gpm add and the install command should probably be gpm install. gpm itself should return usage data.