Coder Social home page Coder Social logo

orf / git-workspace Goto Github PK

View Code? Open in Web Editor NEW
274.0 5.0 14.0 2.46 MB

Sync personal and work git repositories from multiple providers :rocket:

Home Page: https://github.com/orf/git-workspace

License: MIT License

Rust 98.65% Shell 0.61% Nix 0.75%

git-workspace's Introduction

git-workspace ๐Ÿš€

Crates.io Actions Status

If your company has a large number of repositories and your work involves jumping between a lot of them then git-workspace can save you some time by:

  • Easily synchronizing your projects directory with Github, Gitlab.com or Gitlab self-hosted ๐Ÿ”ง
  • Keep projects consistently named and under the correct path ๐Ÿ“
  • Automatically set upstreams for forks โšก
  • Move deleted repositories to an archive directory ๐Ÿ’พ
  • Allowing you to access any repository instantly :shipit:
  • Execute git fetch on all projects in parallel :godmode:

This may sound useless, but the "log into your git provider, browse to the project, copy the clone URL, devise a suitable path to clone it" dance can be a big slowdown. The only obvious solution here is to spend more time than you'll ever spend doing this in your whole life on writing a tool in Rust to do it for you.

Table of Contents

Install ๐Ÿ’ฟ

Homebrew (MacOS + Linux)

brew install git-workspace

Nix (MacOS + Linux)

nix-shell

nix-shell -p git-workspace

nix shell (Flakes)

nix shell nixpkgs#git-workspace

home-manager (home.nix)

{
  home.packages = with pkgs; [
    git-workspace
  ];
}

NixOS (configuration.nix)

{
  environment.systemPackages = with pkgs; [
    git-workspace
  ];
}

AUR (ArchLinux)

paru -S git-workspace

Binaries (Windows)

Download the latest release from the github releases page. Extract it and move it to a directory on your PATH.

Cargo

Don't do this, it's pretty slow: cargo install git-workspace

Usage ๐ŸŽท

Git is really annoying and hijacks the --help flag for subcommands. So to get help use git-workspace --help, not git workspace --help.

$ git-workspace --help
git-workspace 1.1.0
Tom Forbes <[email protected]>
Manage and update personal and work git repos from multiple providers

USAGE:
    git-workspace --workspace <workspace> <SUBCOMMAND>

FLAGS:
    -h, --help
            Prints help information

    -V, --version
            Prints version information


OPTIONS:
    -w, --workspace <workspace>
             [env: GIT_WORKSPACE=/Users/tom/PycharmProjects/]


SUBCOMMANDS:
    add                Add a provider to the configuration
    archive            Archive repositories that don't exist in the workspace anymore
    fetch              Fetch new commits for all repositories in the workspace
    help               Prints this message or the help of the given subcommand(s)
    list               List all repositories in the workspace
    lock               Fetch all repositories from configured providers and write the lockfile
    run                Run a git command in all repositories
    switch-and-pull    Pull new commits on the primary branch for all repositories in the workspace
    update             Update the workspace, removing and adding any repositories as needed

Define your workspace

A workspace is the directory that git-workspace will manage for you, filling it with projects cloned from your providers. To configure this just set a GIT_WORKSPACE environment variable that points to an empty directory. For example:

export GIT_WORKSPACE=~/projects

Provider credentials

Both Github and Gitlab require personal access tokens to access their GraphQL endpoints. Create an access token here:

Export these tokens as GITHUB_TOKEN and GITLAB_TOKEN in your shell.

Adding providers

You can use git workspace add to quickly add entries to your workspace.toml:

  • Clone all github repositories for a user or org

    • git workspace add github [USER OR ORG NAME]
  • Exclude specific repositories:

    • git workspace add github [USER OR ORG NAME] --exclude="foo.*bar$" --exclude="(abc|def)"
  • Clone a namespace or user from Gitlab:

    • git workspace add gitlab gitlab-ce/gitlab-services
  • Clone from a self-hosted gitlab/github instance:

    • git workspace add gitlab my-company-group --url=https://internal-gitlab.company.com
    • git workspace add github user-or-org-name --url=https://internal-github.company.com/api/graphql

Multiple configs

Git workspace will read from any workspace*.toml file under your $GIT_WORKSPACE directory.

Updating your workspace

Running git workspace update will:

  1. Fetch all repositories from your providers
  2. Clone any new repositories that are not present locally
  3. Move any deleted repositories to $GIT_WORKSPACE/.archived/ for posterity

Fetching all changes

git workspace fetch will run git fetch on all projects.

Switch projects ๐Ÿ”

git workspace list will output the names of all your projects. You can integrate this with whatever tool you wish to provide a way to quickly search for and select repositories.

Fish, with fzf

The following fish shell snippet gives you a open-project [search-string] command you can use to search for and open projects. It combines the git workspace list command with fzf, and opens the project path with your $EDITOR:

# ~/.config/fish/functions/open-project.fish
function open-project -d "Open a project"
  set filter "$argv"
  set chosen_project (git workspace list | fzf -q "$filter")
  if string length -q -- $chosen_project
     $EDITOR $GIT_WORKSPACE/$chosen_project
     pushd $GIT_WORKSPACE/$chosen_project
  end
end

Zsh, with fzf

function project {
	local filter="$@"
	local chosen_project=$(git workspace list | fzf -q "$filter")
	if [[ -n $chosen_project ]]; then
		pushd "$GIT_WORKSPACE/$chosen_project"
	fi
}

Bash, with fzf

Contributed by a user (@kreyren:github.com):

#!/bin/sh
# shellcheck shell=sh # Written to comply with IEEE Std 1003.1-2017 for standard POSIX environment

###! # WorkSPace (wsp)
###! Switches to specified git-workspace project directory
###! - Requires git and fzf
wsp() {
    # Check for required non-standard commands
    for command in ${FZF:-"fzf"} ${GIT:-"git"}; do
        ${COMMAND:-"command"} -v "$command" || { ${PRINTF:-"printf"} "FATAL: %s\\n" "Command '$command' is not executable"; ${EXIT:-"exit"} 127 ;}
    done
    
    # shellcheck disable=SC2086 # Harmless warning about missing double-quotes that are not expected to allow parsing multiple arguments
    wsp_path="${1:-"${GTT_WORKSPACE:-"$PWD"}/$(${GIT:-"git"} workspace list | ${FZF:-"fzf"} ${fzf_arg:-"-q"} "$@")"}" # Path to the git workspace directory
    
    # Change directory
    ${CD:-"cd"} "$wsp_path" || { printf "FATAL: %s\\n" "Unable to change directory to '$wsp_path'";}
}

Consider using shfmt to optimize the file size.

Contributing ๐Ÿ›

This is my first 'proper' Rust project. If you're experienced with Rust you might puke at the code, but any feedback to help me improve would be greatly appreciated!

If you want to contribute then just go for it. cargo install should get you ready to go. Be warned: there are currently no tests ๐Ÿ’ฃ. I run integration tests with Github Actions, but that's about it. It's on my to-do list, I promise โ„ข๏ธ.

git-workspace's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

git-workspace's Issues

SSH private key

Hey!

I've faced with a problem on git workspace update stage.
Workspace is trying to clone via ssh with standard private key: id_rsa

In my case this way is not working due to another name of private ssh key for git.
I think, it would be nice, if user will have an ability to specify a key.

UPD: Updated description of the Issue

Show error message

I had a token which wasnt authorised for a certain org, and this tool gave only the error:

thread '' panicked at 'Invalid response from GitHub for user [org-name]', src/providers/github.rs:138:36

Underneath the error, the actual response is more helpful:

{"data":{"repositoryOwner":null},"errors":[{"extensions":{"saml_failure":true},"locations":[{"column":9,"line":4}],"message":"Resource protected by organization SAML enforcement. You must grant your Personal Access token access to this organization.","path":["repositoryOwner","repositories"],"type":"FORBIDDEN"}]}

It would be nice if this error could be shown.

Allow specifying individual repos

While it is very useful to manage all repos in an org together, most of my orgs have too many repos, including many I am not interested in.

It would be nice if individual repos can be supported.

Running update with wrong workspace archived everything in that directory

I ran ~/git-workspaces/github> git-workspace --workspace . update , i.e. inside the github sub-directory instead of in ~/git-workspaces. There is no workspace.toml in this github sub-directory, and git-workspace then "archived" the entire tree under that subdirectory, and created an "empty" workspace-lock.toml, i.e. repo = [].

The output doesnt mention the .archive directory, so naturally I panicked initially.

Easily fixed after finding the .archive directory.

However, IMO there should be some check, probably for existence of workspace.toml.

[question] naming

Hi,

@orf

I guess this one has nothing to do with the git worktree official command, right?
(that one handles multiple copies of the same repo, locally)

Whereas your git-workspace handles multiple, independent repositories, at once
It seems to do the same job as myrepos

Installation using cargo fails with "no `strip_ansi_codes` in the root"

I am running Arch Linux and try to install git-workspace. The only option for me seems to be using cargo install git-workspace.

When running the cargo install command compilation of git-workspace v0.8.0 fails with this output:

   Compiling git-workspace v0.8.0
error[E0432]: unresolved import `console::strip_ansi_codes`
 --> /home/micha/.cargo/registry/src/github.com-1ecc6299db9ec823/git-workspace-0.8.0/src/repository.rs:2:15
  |
2 | use console::{strip_ansi_codes, truncate_str};
  |               ^^^^^^^^^^^^^^^^ no `strip_ansi_codes` in the root

For more information about this error, try `rustc --explain E0432`.
error: failed to compile `git-workspace v0.8.0`, intermediate artifacts can be found at `/tmp/cargo-installbR3vGW`

Caused by:
  could not compile `git-workspace` due to previous error

Arbitrary projects

Do you have an idea on how to handle arbitrary projects? For instance, I need to clone this project now in order to work on it, and previously I would do just this:

dev start orf/git-workspace

This would take care of cloning the local repo in the correct location. I imagine this would fit into the concept of a git workspace.

Weird error

I ran git workspace update and got a few of those errors:

Git exited with code 1: Resolving deltas: 100% (454/454), done.git: 'submodule' is not a git command. See 'git --help
'.

Also, super awesome project! I might contribute soon, because it replaces a few scripts I already have in a much better way.

Support GitHub Enterprise Edition

It looks like self-hosted GitLab is supported but GHE is not currently:

git-workspace --workspace . add github my-team-org --url=https://github.internal.company.com
error: Found argument '--url' which wasn't expected, or isn't valid in this context

USAGE:
    git-workspace add github [FLAGS] [OPTIONS] <name>

For more information try --help

Add `status` sub-command

Will list all repo's and show modified/untracked/behind/ahead/stashes/etc. for each repository.

Compilation failure: no `strip_ansi_codes` in the root

Building on mac with the brew default rust

% rustc --version
rustc 1.56.1
% cargo --version
cargo 1.56.0
% cargo install --path .
  Installing git-workspace v0.8.0 (/Users/jayvdb/Documents/GitHub/git-workspace)
    Updating crates.io index
  Downloaded console v0.13.0
  Downloaded 1 crate (22.5 KB) in 0.64s
   Compiling libc v0.2.106
   Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.32
   Compiling unicode-xid v0.2.2
   Compiling memchr v2.4.1
   Compiling syn v1.0.81
   Compiling cfg-if v1.0.0
   Compiling pkg-config v0.3.22
   Compiling lazy_static v1.4.0
   Compiling serde v1.0.130
   Compiling serde_derive v1.0.130
   Compiling failure_derive v0.1.8
   Compiling ryu v1.0.5
   Compiling crossbeam-utils v0.8.5
   Compiling serde_json v1.0.69
   Compiling version_check v0.9.3
   Compiling adler v1.0.2
   Compiling gimli v0.26.1
   Compiling core-foundation-sys v0.8.3
   Compiling crossbeam-epoch v0.9.5
   Compiling rustc-demangle v0.1.21
   Compiling tinyvec_macros v0.1.0
   Compiling void v1.0.2
   Compiling unicode-segmentation v1.8.0
   Compiling bitflags v1.3.2
   Compiling log v0.4.14
   Compiling unicode-width v0.1.9
   Compiling ppv-lite86 v0.2.15
   Compiling ascii v0.9.3
   Compiling byteorder v1.4.3
   Compiling scopeguard v1.1.0
   Compiling either v1.6.1
   Compiling percent-encoding v2.1.0
   Compiling rayon-core v1.9.1
   Compiling matches v0.1.9
   Compiling itoa v0.4.8
   Compiling unicode-bidi v0.3.7
   Compiling regex-syntax v0.6.25
   Compiling once_cell v1.8.0
   Compiling doc-comment v0.3.3
   Compiling remove_dir_all v0.5.3
   Compiling native-tls v0.2.8
   Compiling vec_map v0.8.2
   Compiling ansi_term v0.11.0
   Compiling anyhow v1.0.45
   Compiling strsim v0.8.0
   Compiling fnv v1.0.7
   Compiling number_prefix v0.3.0
   Compiling same-file v1.0.6
   Compiling chunked_transfer v1.4.0
   Compiling base64 v0.13.0
   Compiling fs_extra v1.2.0
   Compiling atomic-counter v1.0.1
   Compiling miniz_oxide v0.4.4
   Compiling memoffset v0.6.4
   Compiling rayon v1.5.1
   Compiling tinyvec v1.5.0
   Compiling unreachable v1.0.0
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling textwrap v0.11.0
   Compiling heck v0.3.3
   Compiling qstring v0.7.2
   Compiling form_urlencoded v1.0.1
   Compiling addr2line v0.17.0
   Compiling walkdir v2.3.2
   Compiling unicode-normalization v0.1.19
   Compiling object v0.27.1
   Compiling combine v3.8.1
   Compiling aho-corasick v0.7.18
   Compiling bstr v0.2.17
   Compiling jobserver v0.1.24
   Compiling getrandom v0.2.3
   Compiling terminal_size v0.1.17
   Compiling num_cpus v1.13.0
   Compiling atty v0.2.14
   Compiling dirs v1.0.5
   Compiling quote v1.0.10
   Compiling crossbeam-channel v0.5.1
   Compiling core-foundation v0.9.2
   Compiling security-framework-sys v2.4.2
   Compiling idna v0.2.3
   Compiling regex v1.5.4
   Compiling rand_core v0.6.3
   Compiling cc v1.0.71
   Compiling console v0.13.0
   Compiling clap v2.33.3
   Compiling security-framework v2.4.2
   Compiling rand_chacha v0.3.1
   Compiling url v2.2.2
   Compiling console v0.15.0
   Compiling globset v0.4.8
   Compiling backtrace v0.3.63
   Compiling libz-sys v1.1.3
   Compiling openssl-sys v0.9.70
   Compiling libssh2-sys v0.2.23
   Compiling libgit2-sys v0.12.24+1.3.0
   Compiling crossbeam-deque v0.8.1
   Compiling rand v0.8.4
   Compiling synstructure v0.12.6
   Compiling tempfile v3.2.0
   Compiling structopt-derive v0.4.18
   Compiling failure v0.1.8
   Compiling structopt v0.3.25
   Compiling graphql-parser v0.2.3
   Compiling pwd v1.3.1
   Compiling indicatif v0.15.0
   Compiling expanduser v1.2.2
   Compiling graphql-introspection-query v0.1.0
   Compiling toml v0.5.8
   Compiling graphql_client_codegen v0.9.0
   Compiling ureq v1.5.5
   Compiling graphql_query_derive v0.9.0
   Compiling graphql_client v0.9.0
   Compiling git2 v0.13.23
   Compiling git-workspace v0.8.0 (/Users/jayvdb/Documents/GitHub/git-workspace)
error[E0432]: unresolved import `console::strip_ansi_codes`
 --> src/repository.rs:2:15
  |
2 | use console::{strip_ansi_codes, truncate_str};
  |               ^^^^^^^^^^^^^^^^ no `strip_ansi_codes` in the root

For more information about this error, try `rustc --explain E0432`.
error: failed to compile `git-workspace v0.8.0 (/Users/jayvdb/Documents/GitHub/git-workspace)`, intermediate artifacts can be found at `/Users/jayvdb/Documents/GitHub/git-workspace/target`

Caused by:
  could not compile `git-workspace` due to previous error

Mirrors and forks

Hi, very nice project ๐Ÿ™‚ !

It fits my workflow perfectly!

...except for a few things ๐Ÿ˜„ :

  • I like to keep my local forks into a "forks" directory instead of my "pawamoy" directory inside the workspace, because it helps distinguish them from my own projects. Would you be willing to implement an option to move forks in a custom directory?
  • I always have two online copies of my repos: one on GitHub and one on GitLab. I have the same username on both (pawamoy), and the projects also have the same name (I also have same orgs with different names, but lets see that later). My local clones are configured to push on both remotes. Would you be willing to add a feature that allows users to do that when cloning projects? Something like telling git-workspace to clone from GitHub but also set remote urls and push urls for both GitHub and another remote like GitLab?

Q: is git-workplace suitable solution for my project?

I have a huge git repository (2.2GB of pure source code) that is composed out of +200 git repositories some of which are submodules of a submodule of a submodule (longest chain is 14) or share a submodule

Is git-workspace a solution that can work with such project to sync all remotes? (designed for 12 remotes including github, gitlab, etc..)

Archive fails if there is an "identical" existing archive

I dont know how I ended up with a new checkout of the archived repo, but when it tries to archive it, it fails because there is already one.

The new clone is clean, while the archived one does have some extra local branches, but everything in the new clone is in the archived one.

...
Archiving 1 repositories
Archiving github/org/repo
Error: Error archiving repositories

Caused by:
    0: Error moving directory /home/jayvdb/git-workspaces/github/org/repo to /home/jayvdb/git-workspaces/.archive/github/org/repo
    1: Directory not empty (os error 39)

Access to groups and subgroups

Cannot access group/subgroup with following problem:

Gitlab group/user mygroup/mysubgroup could not be found. Are you sure you have access?.

With access through API (with the same API token) works fine.

The same problem with only group:

Gitlab group/user mygroup could not be found. Are you sure you have access?

Tested on gitlab provider (gitlab.com and self-hosted instance)

Add Gitea?

[kreyren@leonid:~/Repositories/RiXotStudio]$ git workspace add dotya rixotstudio --url=https://git.dotya.ml/RiXotStudio/RiXotStudio
error: Found argument 'dotya' which wasn't expected, or isn't valid in this context

USAGE:
    git-workspace --workspace <workspace> add [OPTIONS] <SUBCOMMAND>

For more information try --help

[kreyren@leonid:~/Repositories/RiXotStudio]$ git workspace add gitea rixotstudio --url=https://git.dotya.ml/RiXotStudio/RiXotStudio
error: The subcommand 'gitea' wasn't recognized
        Did you mean 'gitlab'?

If you believe you received this message in error, try re-running with 'git-workspace add -- gitea'

USAGE:
    git-workspace --workspace <workspace> add [OPTIONS] <SUBCOMMAND>

For more information try --help

NixOS `home-manager` module documentation

I just submitted this PR that adds a new home-manager module to declaratively set workspaces and create systemd timers to update them.
As soon as it gets (hopefully) merged I think I can rewrite/expand the example in the README.

By the way I'm using this since yesterday and it works great, good job!
Here an example of how the module can be used:

{config, ...}: {
  programs.git-workspace.enable = true;
  services.git-workspace = {
    enable = true;
    frequency = "04:00:00";
    environmentFile = age.secrets.git-workspace-tokens.path; # I use agenix, but one can also directly provide a manually created file
    workspaces = {
      personal.provider = [
        {
          provider = "github";
          name = "aciceri";
          path = "${config.home.homeDirectory}/projects";
          skips_forks = false;
        }
        {
          provider = "github";
          name = "nix-community";
          path = "${config.home.homeDirectory}/nix-community";
        }
      ];
      work.provider = [
        {
          provider = "github";
          name = "company";
          path = "${config.home.homeDirectory}/work" ;
        } 
      ];
    };
  };
}

This generates 2 TOML files for two workspaces (the first one with two providers) and two systemd timers.
I hope I understood how git-workspace was meant to be used correctly and having integrated it well in home-manager but please tell me if I should change something.

Add filtering

Can we design in this issue what way the filtering configuration for git-workspace could be done?

Requirements as I do understand:

  • some wants to define what repositories to "update/sync"
  • filter for repositories that within the scope of record in workspace.yaml (ie: the tool fetch today even repos of people I do follow, however if the name: is "myGithubName" it is desired to fetch only github.com/myGithubName namespace). This would at least allow to filter per GH organizations. (possibly the same per gitlab, gitea, etc..)
  • .filter(|r| r.name .....) we could do even some regex, but it might be better to filter on graph query already
  • how to record rules on configuration

Archive, desire it's own command

use-case: if you comment part of your workspace.yaml (on next update your repos are moved to .archive).

(Note, I possibly hit this because I defined one record in workspace.yaml per my GH organization). So the git workspace updpate did actually have run multiple times on the same path.)

More rational is to

  • perform .archive old repos only when requested
  • add option enable/diisable to source record in workspace.yaml to avoid processing (so you dont have to comment out it)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.