byron / gitoxide

An idiomatic, lean, fast & safe pure Rust implementation of Git

gitoxide is an implementation of git written in Rust for developing future-proof applications which strive for correctness and performance while providing a pleasant and unsurprising developer experience.

gitoxide provides the gix and ein binaries for use on the command-line to allow experimentation with key features like fetch and clone, and to validate the usability and control of the API offered by the gix crate.

gitoxide aspires to be a production-grade server implementation and the ein binary aspires to become the default way to interact with git repositories.

Development Status

The command-line tools as well as the status of each crate is described in the crate status document.

For use in applications, look for the gix crate, which serves as entrypoint to the functionality provided by various lower-level plumbing crates like gix-config.

Feature Discovery

Can gix do what I need it to do?

The above can be hard to answer and this paragraph is here to help with feature discovery.

Look at crate-status.md for a rather exhaustive document that contains both implemented and planned features.

Further, the gix crate documentation with the git2 search term helps to find all currently known git2 equivalent method calls. Please note that this list is definitely not exhaustive yet, but might help if you are coming from git2.

What follows is a high-level list of features and those which are planned:

• clone
• fetch
• blame
• push
• reset
• status
• blob-diff
• merge
• rebase
• commit
• worktree checkout and worktree stream
• reading and writing of objects
• reading and writing of refs
• reading and writing of .git/index
• reading and writing of git configuration
• pathspecs
• revspecs
• .gitignore and .gitattributes

Crates

Follow linked crate name for detailed status. Please note that all crates follow semver as well as the stability guide.

Production Grade

• Stability Tier 1

  • gix-lock

• Stability Tier 2

  • gix-tempfile

Stabilization Candidates

Crates that seem feature complete and need to see some more use before they can be released as 1.0. Documentation is complete and was reviewed at least once.

• gix-mailmap
• gix-chunk
• gix-ref
• gix-config
• gix-config-value
• gix-glob
• gix-actor
• gix-hash

Initial Development

These crates may be missing some features and thus are somewhat incomplete, but what's there is usable to some extent.

• usable (with rough but complete docs, possibly incomplete functionality)
  • gix (⬅ entrypoint)
  • gix-object
  • gix-validate
  • gix-url
  • gix-packetline
  • gix-packetline-blocking
  • gix-transport
  • gix-protocol
  • gix-pack
  • gix-odb
  • gix-commitgraph
  • gix-diff
  • gix-traverse
  • gix-features
  • gix-credentials
  • gix-sec
  • gix-quote
  • gix-discover
  • gix-path
  • gix-attributes
  • gix-ignore
  • gix-pathspec
  • gix-index
  • gix-revision
  • gix-revwalk
  • gix-command
  • gix-prompt
  • gix-refspec
  • gix-fs
  • gix-utils
  • gix-hashtable
  • gix-worktree
  • gix-bitmap
  • gix-negotiate
  • gix-filter
  • gix-worktree-stream
  • gix-archive
  • gix-submodule
  • gix-status
  • gix-worktree-state
  • gitoxide-core
• very early (possibly without any documentation and many rough edges)
  • gix-date
  • gix-dir
• idea (just a name placeholder)
  • gix-note
  • gix-fetchhead
  • gix-lfs
  • gix-rebase
  • gix-sequencer
  • gix-tui
  • gix-tix
  • gix-bundle
  • gix-fsck

Stress Testing

• Verify huge packs
• Explode a pack to disk
• Generate and verify large commit graphs
• Generate huge pack from a lot of loose objects

Stability and MSRV

Our stability guide helps to judge how much churn can be expected when depending on crates in this workspace.

Installation

Download a Binary Release

Using cargo binstall, one is able to fetch binary releases. You can install it via cargo install cargo-binstall, assuming the rust toolchain is present.

Then install gitoxide with cargo binstall gitoxide.

See the releases section for manual installation and various alternative builds that are slimmer or smaller, depending on your needs, for Linux, MacOS and Windows.

Download from Arch-Repository

For Arch Linux you can download gitoxide from community repository:

pacman -S gitoxide

Download from Exherbo Linux Rust repository

For Exherbo Linux you can download gitoxide from the Rust repository:

cave resolve -x repository/rust
cave resolve -x gitoxide

From Source via Cargo

cargo is the Rust package manager which can easily be obtained through rustup. With it, you can build your own binary effortlessly and for your particular CPU for additional performance gains.

The minimum supported Rust version is documented in the CI configuration, the latest stable one will work as well.

There are various build configurations, all of them are documented here. The documentation should also be useful for packagers who need to tune external dependencies.

# A certain way to install `gitoxide` with just Rust and a C compiler installed.
# If there are problems with SSL certificates during clones, try to omit `--locked`.
cargo install gitoxide --locked --no-default-features --features max-pure

# The default installation, 'max', is the fastest, but also needs some libraries available to build successfully.
# Installing these is platform-dependent and thus can't be explained here.
cargo install gitoxide

# For smaller binaries and even faster build times that are traded for a less fancy CLI implementation, use `lean`
# or `lean-termion` respectively.
cargo install gitoxide --locked --no-default-features --features lean

The following installs the latest unpublished release directly from git:

cargo install --git https://github.com/Byron/gitoxide  gitoxide

How to deal with build failures

On some platforms, installation may fail due to lack of tools required by C toolchains. This can generally be avoided by installation with cargo install gitoxide --no-default-features --features max-pure.

What follows is a list of known failures.

• On Fedora, perl needs to be installed for OpenSSL to build properly. This can be done with the following command: dnf install perl (see this issue).

Usage

Once installed, there are two binaries:

• ein
  • high level commands, porcelain, for every-day use, optimized for a pleasant user experience
• gix
  • low level commands, plumbing, for use in more specialized cases and to validate newly written code in real-world scenarios

Project Goals

Project goals can change over time as we learn more, and they can be challenged.

• a pure-rust implementation of git
  • including transport, object database, references, cli and tui
  • a simple command-line interface is provided for the most common git operations, optimized for user experience. A simple-git if you so will.
  • be the go-to implementation for anyone who wants to solve problems around git, and become the alternative to GitPython and libgit2 in the process.
  • become the foundation for a distributed alternative to GitHub, and maybe even for use within GitHub itself
• learn from the best to write the best possible idiomatic Rust
  • libgit2 is a fantastic resource to see what abstractions work, we will use them
  • use Rust's type system to make misuse impossible
• be the best performing implementation
  • use Rust's type system to optimize for work not done without being hard to use
  • make use of parallelism from the get go
  • sparse checkout support from day one
• assure on-disk consistency
  • assure reads never interfere with concurrent writes
  • assure multiple concurrent writes don't cause trouble
• take shortcuts, but not in quality
  • binaries may use anyhow::Error exhaustively, knowing these errors are solely user-facing.
  • libraries use light-weight custom errors implemented using quick-error or thiserror.
  • internationalization is nothing we are concerned with right now.
  • IO errors due to insufficient amount of open file handles don't always lead to operation failure
• Cross platform support, including Windows
  • With the tools and experience available here there is no reason not to support Windows.
  • Windows is tested on CI and failures do prevent releases.

Non-Goals

Project non-goals can change over time as we learn more, and they can be challenged.

• replicate git command functionality perfectly
  • git is git, and there is no reason to not use it. Our path is the one of simplicity to make getting started with git easy.
• be incompatible to git
  • the on-disk format must remain compatible, and we will never contend with it.
• use async IO everywhere
  • for the most part, git operations are heavily reliant on memory mapped IO as well as CPU to decompress data, which doesn't lend itself well to async IO out of the box.
  • Use blocking as well as gix-features::interrupt to bring operations into the async world and to control long running operations.
  • When connecting or streaming over TCP connections, especially when receiving on the server, async seems like a must though, but behind a feature flag.

Contributions

If what you have seen so far sparked your interest to contribute, then let us say: We are happy to have you and help you to get started.

We recommend running just test check-size during the development process to assure CI is green before pushing.

A backlog for work ready to be picked up is available in the Project's Kanban board, which contains instructions on how to pick a task. If it's empty or you have other questions, feel free to start a discussion or reach out to @Byron privately.

For additional details, also take a look at the collaboration guide.

Getting started with Video Tutorials

• Learning Rust with Gitoxide
  • In 17 episodes you can learn all you need to meaningfully contribute to gitoxide.
• Getting into Gitoxide
  • Get an introduction to gitoxide itself which should be a good foundation for any contribution, but isn't a requirement for contributions either.
• Gifting Gitoxide
  • See how PRs are reviewed along with a lot of inner monologue.

Other Media

• Rustacean Station Podcast

Roadmap

Features for 1.0

Provide a CLI to for the most basic user journey:

• initialize a repository
• fetch
  • and update worktree
• clone a repository
  • bare
  • with working tree
• create a commit after adding worktree files
• add a remote
• push
  • create (thin) pack

Ideas for Examples

• gix tool open-remote open the URL of the remote, possibly after applying known transformations to go from ssh to https.
• tix as example implementation of tig, displaying a version of the commit graph, useful for practicing how highly responsive GUIs can be made.
• Something like git-sizer, but leveraging extreme decompression speeds of indexed packs.
• Open up SQL for git using sqlite virtual tables. Check out gitqlite as well. What would an MVP look like? Maybe even something that could ship with gitoxide. See this go implementation as example.
• A truly awesome history rewriter which makes it easy to understand what happened while avoiding all pitfalls. Think BFG, but more awesome, if that's possible.
• gix-tui should learn a lot from fossil-scm regarding the presentation of data. Maybe this can be used for prompts. Probably magit has a lot to offer, too.

Ideas for Spin-Offs

• A system to integrate tightly with gix-lfs to allow a multi-tier architecture so that assets can be stored in git and are accessible quickly from an intranet location (for example by accessing the storage read-only over the network) while changes are pushed immediately by the server to other edge locations, like the cloud or backups. Sparse checkouts along with explorer/finder integrations make it convenient to only work on a small subset of files locally. Clones can contain all configuration somebody would need to work efficiently from their location, and authentication for the git history as well as LFS resources make the system secure. One could imagine encryption support for untrusted locations in the cloud even though more research would have to be done to make it truly secure.
• A syncthing like client/server application. This is to demonstrate how lower-level crates can be combined into custom applications that use only part of git's technology to achieve their very own thing. Watch out for big file support, multi-device cross-syncing, the possibility for untrusted destinations using full-encryption, case-insensitive and sensitive filesystems, and extended file attributes as well as ignore files.
• An event-based database that uses commit messages to store deltas, while occasionally aggregating the actual state in a tree. Of course it's distributed by nature, allowing people to work offline.
  • It's abstracted to completely hide the actual data model behind it, allowing for all kinds of things to be implemented on top.
  • Commits probably need a nanosecond component for the timestamp, which can be added via custom header field.
  • having recording all changes allows for perfect merging, both on the client or on the server, while keeping a natural audit log which makes it useful for mission critical databases in business.
  • Applications
    • Can markdown be used as database so issue-trackers along with meta-data could just be markdown files which are mostly human-editable? Could user interfaces be meta-data aware and just hide the meta-data chunks which are now editable in the GUI itself? Doing this would make conflicts easier to resolve than an sqlite database.
    • A time tracker - simple data, very likely naturally conflict free, and interesting to see it in terms of teams or companies using it with maybe GitHub as Backing for authentication.
      • How about supporting multiple different trackers, as in different remotes?

Shortcomings & Limitations

Please take a look at the SHORTCOMINGS.md file for details.

Credits

• itertools (MIT Licensed)
  • We use the izip! macro in code
• deflate2 (MIT Licensed)
  • We use various abstractions to implement decompression and compression directly on top of the rather low-level miniz_oxide crate

🙏 Special Thanks 🙏

At least for now this section is exclusive to highlight the incredible support that Josh Triplett has provided to me in the form of advice, sponsorship and countless other benefits that were incredibly meaningful. Going full time with gitoxide would hardly have been feasible without his involvement, and I couldn't be more grateful 😌.

License

This project is licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Fun facts

• Originally @Byron was really fascinated by this problem and believes that with gitoxide it will be possible to provide the fastest solution for it.
• @Byron has been absolutely blown away by git from the first time he experienced git more than 13 years ago, and tried to implement it in various shapes and forms multiple times. Now with Rust @Byron finally feels to have found the right tool for the job!

gitoxide's Issues

Complete `receive-pack` and `upload-pack` support

I would be very interested in having the support for Git server operations receive-pack and upload-pack. I see this mentioned in the README but there's no timeline.

Refs: rust-lang/git2-rs#591, libgit2/libgit2#5605

improve ancestor traversal

• resolve tag objects (recursively, safely)

gitoxide/git-odb/src/traverse.rs

Line 66 in 405dd9d

if let Some(commit) = obj.as_commit() {

• be smarter about .seen - it's never cleared even though maybe there is a way to conclude that it doesn't have to be kept (or doesn't have to be that big
  • Even with huge histories we wouldn't actually keep more than 250000k object ids, which is probably no more than 10 or 20MB effectively.

macOS cargo build is broken (default/max and max-termion)

Hi there,

I don't seem to be able to install gitoxide via cargo on macOS Catalina, despite the builds passing. I get the same errors when running the default (max) and max-termion feature:

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:10:26
   |
10 |     #[clap(long, short = "t")]
   |                          ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:17:26
   |
17 |     #[clap(long, short = "v")]
   |                          ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:33:17
   |
33 |         short = "f",
   |                 ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:49:30
   |
49 |         #[clap(long, short = "p")]
   |                              ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:55:30
   |
55 |         #[clap(long, short = "r")]
   |                              ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:76:30
   |
76 |         #[clap(long, short = "p")]
   |                              ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
  --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:96:21
   |
96 |             short = "i",
   |                     ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
   --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:105:30
    |
105 |         #[clap(long, short = "p")]
    |                              ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
   --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:131:21
    |
131 |             short = "c",
    |                     ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
   --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:158:30
    |
158 |         #[clap(long, short = "s")]
    |                              ^^^ expected `char`, found `&str`

error[E0308]: mismatched types
   --> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:163:21
    |
163 |             short = "a",
    |                     ^^^ expected `char`, found `&str`

error: aborting due to 11 previous errors

For more information about this error, try `rustc --explain E0308`.
error: could not compile `gitoxide`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.1`, intermediate artifacts can be found at `/var/folders/kt/mnhg_d0d261668l69cs_m2vh0000gn/T/cargo-installJ5NecG`

Let me know if you need any more info.

Remove need for Polonius borrow checker in `git_odb::compound::Db::locate(…)`

Currently there is branching code in git_odb::compound::Db::locate(…) to workaround a borrowcheck shortcoming using the upcoming Polonius borrow checker.

gitoxide/git-odb/src/compound/locate.rs

Lines 32 to 42 in b317200

	// See 8c5bd095539042d7db0e611460803cdbf172beb0 for a commit that adds polonius and makes the proper version compile.
	// See https://stackoverflow.com/questions/63906425/nll-limitation-how-to-work-around-cannot-borrow-buf-as-mutable-more-than?noredirect=1#comment113007288_63906425
	// More see below! Of course we don't want to do the lookup twice… but have to until this is fixed or we compile nightly.
	#[cfg(not(feature = "polonius"))]
	if alternate.locate(id, buffer)?.is_some() {
	return alternate.locate(id, buffer);
	}
	#[cfg(feature = "polonius")]
	if let Some(object) = alternate.locate(id, buffer)? {
	return Ok(Some(object));
	}

Without the workaround, the lookup costs are doubled.

Using Polonius solves the problem but comes at a cost.

• It requires the nightly compiler (forcing gitoxide to be compiled with nightly if that code-path is chosen).
• The borrow checker performance is reduced and can 'explode' for certain crates like tinyvec.
• It's unclear when Polonius will be stable.

Here is @joshtriplett thoughts on how to resolve this:

When first initializing the Db and looking up all the alternates, rather than recursing, collect all of the alternates as a top-level Vec of (Packs, Loose), so that alternates themselves never have alternates. That way, you don't have to recurse, and there's only one level of alternates. Then, in the loop over alternates, rather than recursively calling locate, just inline the same code that looks in packs and loose, but without the check for alternates.
I think the simplest way to do that would be to have a version of at (at_no_alternates) that ignores alternates entirely, and then have the top-level at collect a deduplicated list of alternates and call at_no_alternates on each one.

Additional Information

Also have a look at how this is currently done for looking up objects in packs. The gist is to get an index of where the object is found first, bypassing the need for borrowing an output buffer, and fill the buffer if an index was found.

gitoxide/git-odb/src/compound/locate.rs

Lines 28 to 30 in b317200

	if let Some(object) = self.loose.locate(id)? {
	return Ok(Some(compound::Object::Loose(object)));
	}

Shallow clone support

I see this commit dc7908f
You don't plan to support shallow clones ?

Organized Repos End with `.git`

Hey! I love the idea of the organize command to clean up my messy ~/Code folder.

I ran it this morning and have found that all of the moved directories now end in .git. Here's an example of the directory with all my work repositories in it:

Repo List

drwxr-xr-x - alafroscia  2 Oct  2020 AIExpedition.git
drwxr-xr-x - alafroscia  4 Dec  2019 boxes-interview-app.git
drwxr-xr-x - alafroscia 10 Jan  2020 builder.git
drwxr-xr-x - alafroscia 11 Feb  2020 cartridge_hooks.git
drwxr-xr-x - alafroscia  9 Mar 14:21 dependabot-clubhouse-actions.git
drwxr-xr-x - alafroscia 11 Jan 08:53 docker-images.git
drwxr-xr-x - alafroscia 20 May  2020 ember-bar-chart.git
drwxr-xr-x - alafroscia  5 Dec  2019 ember-cli-deploy-github-deployment-status.git
drwxr-xr-x - alafroscia 24 Jan  2020 ember-cli-deploy-gitlab-scm-data-generator.git
drwxr-xr-x - alafroscia 23 Oct  2019 ember-cli-mirage.git
drwxr-xr-x - alafroscia  2 Feb 17:09 ember-cli-sentry.git
drwxr-xr-x - alafroscia  1 Mar 13:57 ember-data-json-api-bulk-ext.git
drwxr-xr-x - alafroscia  4 Sep  2019 ember-examples.git
drwxr-xr-x - alafroscia 20 Sep  2019 ember-pop-over.git
drwxr-xr-x - alafroscia 25 Jan  2020 ember-radio-button.git
drwxr-xr-x - alafroscia 27 Apr  2020 eslint-plugin-no-wildcard-postmessage.git
drwxr-xr-x - alafroscia 11 Mar 16:02 fluid-docs-investigation.git
drwxr-xr-x - alafroscia 13 Nov  2019 fluid-tailwind-playground.git
drwxr-xr-x - alafroscia 18 Sep  2019 fluid-wc.git
drwxr-xr-x - alafroscia  8 May 13:22 front-end.git
drwxr-xr-x - alafroscia  8 Apr  2020 fullstack-interview-stats-table.git
drwxr-xr-x - alafroscia 11 Feb  2020 gitd
drwxr-xr-x - alafroscia 11 Feb  2020 gitd.git
drwxr-xr-x - alafroscia  6 May 16:34 lint-config.git
drwxr-xr-x - alafroscia 28 Apr 16:15 movableink.git
drwxr-xr-x - alafroscia 25 Feb 16:13 page-objects.git
drwxr-xr-x - alafroscia 29 Jun  2020 provisioning
drwxr-xr-x - alafroscia  6 Sep  2019 rfcs.git
drwxr-xr-x - alafroscia 28 Oct  2020 studio-app-tools.git
drwxr-xr-x - alafroscia 28 Oct  2020 studio-apps.git
drwxr-xr-x - alafroscia 29 Mar 15:14 studio-framework.git
drwxr-xr-x - alafroscia 28 Oct  2020 studio-package-proxy.git
drwxr-xr-x - alafroscia 20 Jan 11:53 studio-packaging-service.git
drwxr-xr-x - alafroscia  1 Apr 10:02 tailwind-config.git
drwxr-xr-x - alafroscia 14 Apr 15:18 template-lint-plugin.git
drwxr-xr-x - alafroscia 19 Nov  2020 terraform.git

Is this intentional? At least in my case, this was unexpected. I ran the dry run first but I guess didn't notice that .git was going to be added to all of these.

Adjust license to include Apache2 and add contribution guidelines similar to Rust-Analyzer

Right now is not a good time to contribute as a few parts are missing:

• I still want to have a dual license like many popular Rust projects do, also offering Apache 2
• There should be well thought-out contribution guidelines that learn from the best, I heard Rust Analyzer is a great example

Name collision with `glib2`

I'm not really sure what can be done, maybe add a warning, but gio collides with an executable from glib2, I don't know for other linux distributions, but on Arch it's pretty much impossible to not install glib2.

[TRACKING] Async Transport and Protocol

It becomes clear that in order to be more useful for application developers, it's best to support async operation in the transport and protocol layers. This issue tracks the overall progress in making this conversion from blocking code to non-blocking/async code.

• git-packetline
• git-transport
• git-protocol
• gitoxide-core + gitoxide
• create discussion in blocking to figure out if we can have blocking::spawn(future) to avoid clumsy code and overhead.

Potential improvements

• get rid of unsafe pointer magic WithSidebands (cost: high)
  • What needs to be done is to transform the &mut StreamingPeekableIter into a child future, and when exhausted, it must be transformed back into the &mut _ that created it. That way, only a single mutable reference to said Iter is present at any time. Unfortunately the generated futures (using async) don't support that as we would have to keep both the future and the parent that created it inside of our own struct. Instead of hiding this using pointers, one could implement the magical part by hand, a custom future, which happily dissolves into its mutable parent iter ref. That would be quite some work though.

Requirements

• git-transport via quinn - how would that work? Maybe async http isn't even necessary?
• must support tokio 1.0 (quinn needs it).
  • surf via hyper seems to be locked to tokio 0.2 (even though async-std supports tokio 1.0 as well.

Research

• maybe-async - conditional compilation of both sync and async APIs
  • Useful to keep the sync version which compiles quickly thanks to libcurl used under the hood. This might allow to write the async version in smaller increments.
• async-http(s) client
  • tokio
  • hyper::client - probably too low level
  • reqwest
  • non-tokio
    • surf - **supports multiple backends, some of which are tokio, like hyper which relies on their traits interestingly.
• Abstracting over the runtime can be done by…knowing which one to use. Sync and async selection can be done with maybe-async.
• tokio
  • comes with its own compatibility layer to adapt to futures::io related traits.
  • has its own non-blocking process implementation.

Maybe_async

As it's async by default which requires more dependencies than sync code it won't be used in crates to avoid pulling in unnecessary dependencies for builds that don't use it.

This also means definitive duplicates here and there for IO code that otherwise could be deduplicated with maybe_async.

However, tests can and should benefit from maybe_async as minimal dependencies are less important there. Thus we deduplicate tests with it to assure there is no drift in code. This rightfully assumes decent test coverage.

Related

• Can maybe_async get an is_async feature toggle?
  • No need
• See if futures-lite::BlockOn can be used instead by now.

Out of Scope

• async server

Archive

• Figure out how git-transport could be used with quinn and other custom transports.
  • quinn streams support both tokio and futures AsyncReadWrite traits, so going with the more general trait at first should be good. Probably it's a good idea to have a redirection for traits so replacing them with the tokio version is easier should it ever be required.
• Navigate the crate ecosystem and find an http client that can work.
  • Surf with lib-curl as backend seems to do it, as it uses future::io traits which are actually most compatible.
• Realize that what's really needed is a 'custom async transport' and not do all of the following (as originally intended) unless this comes up again.
  • async client
    • maybe::async transport
      • surf based http backend with feature toggles to allow backend selection. This allows choosing the runtime.
      • assure enough tests for these new feature toggles.
    • maybe::async protocol
    • maybe::async gixp clone path - this represents the application and is a good example on how the API feels, alongside the blocking implementation.
• Research what it would mean to have an async transport trait and how it could fit
  • goal put everything that's currently availabe into client::blocking and start working with maybe_async to pull the git::Connection into async land, step by step, also using maybe_async. This has a feature-toggle-able ripple effect and hides all other blocking implementations (as the trait is either blocking or async) until these are ported. Generally the git-transport crate will only support one mode at a time.
• git-packetline ✓
  • the grand refactor - it always felt a little off
  • feature toggles for 'blocking' mode, with default to async operation like in git-transport/protocol
  • refactor packet line provider
  • feature toggle for turning on async-io - otherwise blocking io MUST pull in futures-io for no reason. This will propagate but it's OK.
  • async encode needed for Writer
  • async low level line writer (new)
  • async higher level line writer
  • try to unify encode tests using maybe_async as test-utility only
  • async encode
  • immutable packet line IO
  • async packet line provider (easy mode)
  • async sideband reader (no easy mode)
    • figure out how to use easy mode packet line provider from no-easy-mode (aka async-futures from poll based
• git-transport
  • async test case - empty, but at least we can compile something
  • async capabilities
  • remove maybe_async from library code and add async-client trait behind feature toggle
  • move everything without IO ties into shared space
  • Capabilities::from_lines_with_version_detection(…)
  • Handshake V1 tests
  • decode an actual pack from an async-read - show how to translate it into the blocking world to combine async and sync
  • TransportExt trait
  • Handshake V2 tests
  • Show how to read a pack while keeping everything async
    • unblock handshake v2 test when traversing entries.
• git-protocol
  • separate common (no feature toggles needed) tests and those who need feature toggles
  • git-credentials - keep it blocking as async programs aren't really a thing and the blocking crate exists.
  • start out moving everything into 'blocking' module to be able to take on asyncification step-by-step
  • async arguments and command
  • Refs parsing in async
  • Response in async
  • async Delegate
  • use maybe_async and disallow --all-features as it would misconfigure maybe-async
  • async fetch(…)
  • remove #[allow(dead_code)] in async mode
  • crate-features.md update
• refactor
  • disallow all client features enabled in git-transport
  • disallow all client features enabled in git-packetline
• gitoxide-core + gitoxide
  • feature toggle for blocking client, none set means no networking
  • feature toggle for async client
  • don't let blocking client take precedence in gitoxide-core
  • async transport creation
  • async LsRemote delegate impl + journey test
  • async PackReceive delegate impl
  • Is there an easy way to unblock the pack writing?
  • journey test for async client receiving a pack from a git daemon using async io + fix TODO

Ensure robustness when killed in the middle

git is somewhat robust when killed in the middle. libgit2 seems somewhat less so. I'd love to see gitoxide be rock-solid on that front.

When doing things like clone, fetch, and similar, it's important to deal with network failure, ctrl-c, systems being powered off, and other sources of abrupt interruption. It's important that this never leave the repository in an inconsistent state (e.g. references to objects that don't exist, corrupted packs or indexes that git will choke on, and similar). This includes things like putting things in temporary locations and atomic-renaming them in place, or ensuring that objects are put in place before references to them.

Question: Avoid unnecessary IO calls - use shallow/sparse technique from VFSForGit

I just found this project today and have a question does gitoxide already have the improvements suggested and made in VFS for git?

pack-generation MVP

gen-pack like plumbing command

Generate a pack using some selection of commits or possibly objects. Drives different kinds of iteration as well as ways of building a pack.

Progress

• sort out todos in test
• counting performance
  • things to try is intrusive collections and maybe the caches-rs frequency based LRU
• gixp pack-create
  • use Easy*
  • allow creating thin packs (really just passing on a flag from options)
  • make cache sizes configurable for easier performance tuning/testing
  • See if ahash can speed up single-threaded counting significantly enough (no, maybe 10%)
  • What's so slow with counting? Tree-traversal and the object looks-up required for that. However, what's really slow is accessing the hashmap - it needs its own key generator similar to this. Even that doesn't bring more than say 15%. The actual object is collects is way bigger than our current one, which may help us when trying to focus on copying packs, without creating new deltas. So delta-handling might be an entirely different story for reasons of memory consumption. However, how can this get faster? Also note that git has a pack cache but it's really simple, without frequency analysis or anything fancy.
  • non-deterministic counting somehow ends up with three times more objects than there are
  • a separate argument to control the amount of threads while counting. I saw only 4x speedup when using 10x cores, dashmap doesn't scale perfectly unfortunately. (there seems to be no way to reasonably speed this up across cores), getting best single-core performance seems key but even there it's not getting faster.
  • try hash_hasher for single-threaded counting to see how it performs compared to the standard hash-set with hasher override.
    • see if their build-hasher implementation can be used in dashmap for the same purpose.
  • Byron/prodash#11 - prodash should be faster as with these numbers, prodash will cause slowdown.
  • try leapfrog as dashmap replacement for more performance, once it's running on stable.
  • ⚠️ Make use of reachability bitmaps
  • deltification
    • research: check of git-fastimport builds quick packs, see how deltification works there - delta.c is just 500 lines
    • build diff-deltas based on the result when comparing two buffers
    • see how deltification fits into the machinery to find/try comparing two buffers and find good ones
  • validate object replacement is correctly implemented - right now it's forced to be ignored during pack-gen, see ignore_replacements . It's correct to ignore replacements during reachability traversal, i.e. pack
  • obtain a sort key to help with determining bases
  • create an index file from the pack, for now just by validating it aka pack-index-from-data.
  • journey tests

Command-lines

• create a full pack fast, like clone

cargo build  --release --no-default-features --features max,cache-efficiency-debug --bin gix && /usr/bin/time -lp ./target/release/gix -v free pack create -r  tests/fixtures/repos/rust.git --statistics  --thin -e tree-traversal --pack-cache-size-mb 200 --object-cache-size-mb 100  HEAD --nondeterministic-count

• create a partial pack for fetches

cargo build  --release --no-default-features --features max,cache-efficiency-debug --bin gix && /usr/bin/time -lp ./target/release/gix -v free pack create -r  tests/fixtures/repos/rust.git --statistics  --thin -e tree-diff --pack-cache-size-mb 400 --object-cache-size-mb 100  < tests/fixtures/repos/rust.git/first-thousand.commits

• create a pack with git

echo HEAD | /usr/bin/time -lp  git -C .  pack-objects --all-progress --stdout --revs >/dev/null

Out of scope

• a prototype of a server side sending a pack with only the objects the other side doesn't have.
• reuse existing deltas - doing so would allow to save a lot of time and avoids the need for implementing actual delta compression. One would only have to get the order of entries right to assure consistency. Without that it's not really usable.
• a way to write an index file at the same time, ideally so that it's entirely separate there there isn't always a need
• A gixp subcommand to make the functionality available for stress testing and performance testing
  *Machinery to produce object deltas.

User Stories

I have a system image, which contains potentially 10k to 100k blobs taking up a total of 100M-1G, many of them binary (such as executables and libraries), and I don't know whether the server has any of those blobs or not. I want to turn that into a standalone commit (typically with no parents), push that commit as a compressed pack while (easier) transferring no blobs or trees the server already has, (medium) delta-compressing blobs reasonably against each other, and (hard) delta-compressing any blobs or trees vs the most similar ones the server does have to make a thin-pack. If the server doesn't have anything useful I want to recognize that quickly and just push a reasonably compressed full pack. Metrics I care about: the server spending as little memory as possible incorporating the pack into its repository (to be immediately usable), the transfer being limited only by bandwidth on either fast (100Mbps) or slow (1Mbps) connections to the server, and getting decent compression and delta-compression to help with the slow-connection case.

The user is working in a git repository, such as the Linux kernel repository. As fast as possible, I want to figure out the changes from their most recent commit (ignoring the index), create a new commit with their current commit as a parent, and then push that commit to the server (along with whatever the server doesn't have). Same easy/medium/hard as above, same metrics as above.

Interesting

• git garbage collection and cruft-files (github article)

Archive

Progress

• stream pack entries (base objects only) from an iterator of input objects

• write pack entries to a V2 pack

• clone-like - the other side has no objects

  • write all objects based on unique objects visible in trees using tree-

• fetch-like - the other side has some objects

  • write only changed objects based on another party which has some of our objects, but not all, employing tree-diff capabilities.

• Handle tag objects - their pointee must be added to the set as well.

• Make it so that the total amount of objects to be written is known in advance

  • if the header of the pack wouldn't contain the amount of objects or it would just be bogus, this wouldn't be necessary and allow for a much fast start of the operation. Does git allow invalid counts?

• Statistics about re-used objects and repacked ones, maybe more. This is critical to eventually avoid repacking most.

  • for counts (note about 4x more decoded objects of diffs compared to traversals)
  • for entries
  • pack-create program writes properly named pack files and outputs statics with --statistics

• restore chunk-ordering to allow multi-threaded generators to yield the same pack all the time (chunks may be out of order)

• figure out why the pack hash in parallel mode is still not reproducible

• write object entries directly, without forcing to copy their compressed data into an output::Entry

  • Let's keep copying it as it simplifies the writer and avoids it having to do pack lookups on a single thread. This also means that more memory is used while chunks of objects are in flight and that more memory allocation is done. Probably worth it.

• re-use delta objects as well and write their offsets correctly.

• gixp pack-create

  • figure out why git with one thread is faster when enumerating objects than us with 4 :D (with the Rust repo in particular)
    • dashmap::insert() costs 41% of the runtime apparently. A single-threaded counter might be beneficial for using non-threadsafe types.
    • Using a much bigger pack cache helps a lot with 60s lower counting time, making us just 20s slower than git itself.
    • pack-locations are looked up in single-threaded mode, but that's fast and not doing so doesn't significantly speed things up
    • ⚠️ Note that thus far we were unable to improve counting performance
  • can't pack the Rust repository as it can't find an object that it apparently traverses and that does not exist ( 221483ebaf45df5c956adffee2d4024e7c3b96b8, 6c4f4e1990b76be8a07bde1956d2e3452fd55ee4, 7bda1161a37ff51f254ff0a7862abe6dc54fdb36 ) (it's many objects actually with non-determinisic counting)
  • find a way to not have to collect input objects beforehand, it's something about error handling and types here rather than technical requirements.
  • a list of objects from stdin (like created by git rev-list) (AsIs traversal is default here)
  • a set of tips for internal rev-list traversal of the commit graph with settings for traversal or diff based expansion, with AsIs being an option too
  • tui support
  • A way to place and control object data caches for 4x and more speedups of tree-diffs
  • servo/uluru#22 - when landed, can bring back 8d49976 for reduction of allocation pressure.

• #167

• #170

Performance Opportunities

• Need to decompress entries to find their length - it would save a lot of time if we would know where the next entry begins.
  • The index has that information but only if we build a sorted vec of all offsets. How would that fit in? Maybe a high-performance trait with more complex call signatures to allow building caches?
  • No, this doesn't make a difference at all interestingly, maybe 1.5s of 48 at most. Implemented in ad6d007
    

    gitoxide/git-odb/src/linked/find.rs

    Line 52 in 34b6a2e

    .decompress_entry(&entry, buf)

Notes

packs in general

• on demand loading of packs. Same story for libgit2
• alternates should be expanded to a list of object repositories that don't know alternates. Related to #66 .
• As they keep the amount of stored objects in the header, immediate streaming is limited by knowing that in time.

  gitoxide/git-odb/src/pack/data/init.rs

  Lines 32 to 33 in ad04ad3

  	let (kind, num_objects) =
  	data::parse::header(&data[..12].try_into().expect("enough data after previous check"))?;

  . Streaming can only start once all objects to send have been discovered.

Pack generation

• They use a list of objects to handle along with workstealing among threads.
  • These threads regularly pause their work to allow the stealing to happen safely which might be why that doesn't scale?(actually that one does seem to scale at least with the amount of core I have, it's pack resolution that doesn't scale)
• probably a good idea to not try to find deltas for 'big' objects and make the threshold configurable.

gixp pack-create does not terminate on ctrl-c when reading from stdin

Repro steps:

• ./gixp pack-create > /dev/null

The global handler (from git-features) will respond correctly but the process will refuse to terminate until EOF (ctrl-d) is provided.

I believe the "hanging" behavior is attributed to .lines() call in gitoxide-core/pack/create.rs:L89 as that call will continue to read from the buffer until EOF.

Not sure what the best solution is here, since the global handler implies it will terminate but it never actually does, but on the other hand this isn't quite a user facing tool, so they might realize they just need to send an EOF to stop it.

git-url parsing

In the git-url crate, implement parsing of all URL types that git supports, as documented here: https://www.git-scm.com/docs/git-clone#_git_urls

The implementation should be practical, and does require tests for each case.

nom can be used, and if used, the parser should not support streaming.
Please feel free to add your author name into the crate list.

It's preferred to have a WIP PR and push all results into that one as I must reserve the right to pick up what's there in case development stalls to avoid blocking overall progress.

`gix-ref`: ref-table backend (research)

It's not in mainline yet, but definitely kind of on the way and available.

• an integration to mainline - superseded
• a standalone implementation
• the documentation in reftable.txt in the git repository

I thought I saw it show up once in the mainline git repository, but it's definitely not there anymore.

License issue: docs copied from git into MIT-licensed project

069b312 copied the documentation from git into the project, but the overall project license is still listed as MIT. (And in any case, you don't want to have to keep updating them from git.git.)

[gixp pack-receive] The first proper fetch to a bare repository

Do what's needed to fetch as good as git does (on a bare repository, one without a working tree). This particularly includes proper ref handling as well as safety in the light of concurrent repository access.

Tasks

• fix gitoxide interrupt and signal handling
  • now it will work for CLIs and servers alike with fine-grained control and no global state (unless the application wants it)
• git-tempfile (based on git
• git-lock - a crate providing git-style lock files.
• git-refs - write loose refs and handle the git reflog, temp files, lock files, packed-refs and namespaces
  • publish latest release (and everything else, too)
• git-pack - assure packs are written safely, that is won't interfere with multi-packs or other pack writers writing the very same pack. Check how locking works.
  • #153
  • thin pack support in bundle writer
• gix clone
  • turn gixp pack-receive into gixp clone creating an empty repository (for lack of index handling/checkout) and cloning the first pack.
• gix fetch
  • #181
  • A tool to fetch into an existing repository correctly, creating a new pack and writing refs using transactions (for now without hook execution)
  • investigate fetch negotiation and see how much work is truly needed there. If logic is involved, make it readily reusable in via git-protocol.
• git-repository
  • Is there a way to bring transport/protocol related functionality to git-repository to greatly simplifying doing ref-listings and fetches?

Archive

Research

Research

Reflog Handling

• entirely disabled in bare repos
• forward iterators could be bstr::lines()
• reverse-iterators could be bstr::SplitReverse with a VecDeque for refilling a read buffer from the end of a file with seeks.
• line parsing is here
• expiry is done by rewriting the entire file based on a filter, writing is literally here

Refs Writing

• You can turn a symbolic ref into a peeled one (i.e. detach a HEAD) with transactions but you cannot turn it back into a symbolic one with that. All that happens directly and outside of transactions.
• Writing symbolic references like HEAD splits the ref update transparently and across any amount of refs.
• You cannot delete ref logs using REF_LOG_ONLY but they are deleted with the owning reference.
• ref transactions
  • there is a transaction hook which gets all transaction data without flags, that is old and new oid and refname, along with the 'action' indicating what happened to the transaction.
  • probably it should be possible to introspect transactions as they are executing, but theoretically this can also happen outside of the method itself.
• git file lock
  • it looks like they are creating a tempfile with a specified name for locks (exclusive and all using atomic FS ops) which can then potentially be written in the same moment. Definitely good for loose refs that don't exist.
• loose refs writing intricately knows packed refs, which makes sense in order to keep them consistent.

File Locking

• investigate tempfile to conclude that it's certainly great as reference but won't be exactly what git does. Let's see if it's needed after all to do it exactly like that. Git definitely sets up signal handlers to delete tempfiles so probably these will have to be threadsafe or interned objects.
• If directories are involved, use raceproof file creation
• lockfile.c holds the entire blocking implementation, including backoff. Looks like that's git-lock.

Reflogs

• The file is read line by line and entries are handled on the fly using iterators, easiest to use bstr::lines() there.
• reverse iterators use a buffer of 1024 bytes to seek lines backwards
• parsing is here
• for expiry the file is rewritten based on iteration
• for new reflogs, these are appended (only)

Refs Writing

• git file lock
  • cargo uses flock for comparison with different semantics.
  • fslock seems a bit newer and has a few tests
  • fs2 does not compile anymore and seems unmaintained for years now. Can do more than we need, too.
  • file-lock is posix only but uses fcntl under the hood.

Signal-Hook

• The use of mutexes is unsafe as the current thread might be interrupted while holding the mutex. When trying to obtain a lock in the handler the thread will inevitably deadlock.
• Memory allocation and deallocation is not allowed! So inside a handler we have to do what we do and call std::mem::forget to implement it correctly.

Done Tasks

• prodash
  • replace usage of ctrlc that starts yet another thread with the signal-hook iterator to process pending events from time to time as part fo the ticker thread. Saves a thread and enables proper handler chaining.
• git-features
  • Replace ctrlc usage with signal-hook (i.e. current atexit handler for interrupts)
  • don't use stdout in interrupt handler as it does use a mutex under the hood. Instead allow aborting after the second interrupt in case the application is not responding. It would be great to have a lock-free version of stderr though… .
  • Integrate 'git-tempfile' behind feature toggle to allow interrupt handlers to be tempfile handler aware and not interfere.
  • replace existing usage of git_features::interrupt::is_interrupted() with versions of it that are local to the method or function.
  • move git-features::interrupt into git-repository as this kind of utility is for application usage only. There the git-tempfile integration makes sense, too.
• git-tempfile
  • registered tempfile support to allow deletion on exit (and other signals). Use dashmap as storage.
  • Make sure pid is recorded to assure forking works as expected.
  • docs
  • fix windows build
  • a test validating default handlers are installed
  • release
  • race-proof creation of directories leading to the tempfile
  • a way to use the above for actual tempfiles
  • race-proof deletion of empty directories that conflict with the filename
  • a way to use the above for actual tempfiles
  • differentiate between closed and writable tempfiles in the typesystem to make choice permanent
  • a way to not install any handlers so that git-repository interrupt can run the tempfile removal itself right before aborting.
  • Make with_mut less cumbersome to use by assuming the interrupt handler will indeed abort.
• git-lock - a crate providing git-style lock files.
  • lock file for update
  • marker for holding a lock
  • exponential backoff
  • the above with randomization
  • actual retries with blocking sleep
  • test for the above
• git-refs
  • sketch transaction type
  • figure out whether or not to 'extend' the API to include changes from Symbolic refs to peeled ones in transactions
  • git signature parsing code is shared and moved to git-actor
  • git-object uses git-actor
  • git-object: unify nom error handling everywhere (to reuse the nom error handling machinery instead of re-inventing it)
  • git-object can use verbose errors and () - unit errors per feature toggle.
  • parse ref log line
  • reflog forward iteration
  • reflog backward iteration
  • file reflog writing
  • git-tempfile close (Handler -> Handle)
  • git-lock File close and Marker persist
  • an API to access ref logs for a reference
  • create single symbolic ref without reflog
  • split refs and reusable edit preprocessing
  • delete refs with reflog handling
  • handle parent links for 'old' oid in the log of parent refs
  • handle parent links for error messages of reference names (for lock errors at least)
  • Figure out how to deal with 'previous-value' ambiguity with create-or-update modes.
  • git-lock commit() is recoverable
  • commit()'ing onto empty directories can delete the directory in git-ref
  • internal reflog writing or appending for locked refs
  • persisting lock file onto an empty directory deletes the empty directory and tries again
  • create or update refs with reflog handling
  • research different mmap implementation but ultimately stick to fast-and-simple filebuffer
  • packed-refs iteration - important for being able to read all refs during packfile negotiation
  • iter packed refs from separately loaded buffer
  • iter loose refs with prefix
  • packed-refs lookup with binary search (full-paths)
  • packed-refs lookup with binary search (partial-paths), following lookup rules
  • re-add perf test of sorts, see script to generate big pack file
    • ~6.2mio/s in iteration and 720k/s for lookups/finds using full paths
  • use binary search to find start point for packed prefix iteration
  • iterate all refs (including packed ones)
  • the above, with prefix filtering
  • find_one uses packed-refs if available (use appropriate strategy for reading in full or mapping)
  • remove and test remaining todos
  • packed-refs writing and integration with transaction (must be) - deletions have to be propagated, updates only go to refs (I think, check)
  • #138
  • #139
  • #140
  • #152
  • Make sure broken/invalid loose refs don't break ref iteration and have a way to find them

`gix tool organize` does not support aliased URLs

There's a somewhat lesser known feature in git where you can use aliases for your remote URLs by putting something like

[url "ssh://[email protected]/"]
    insteadOf = "gh:"

And then use git clone gh:Byron/gitoxide instead of typing out the entire URL.

As of 0.9, gitoxide fails to parse those remotes:

~/scratch
❯ git clone gh:Byron/gitoxide
Cloning into 'gitoxide'...
remote: Enumerating objects: 24757, done.
remote: Counting objects: 100% (5485/5485), done.
remote: Compressing objects: 100% (2069/2069), done.
remote: Total 24757 (delta 3133), reused 5447 (delta 3101), pack-reused 19272
Receiving objects: 100% (24757/24757), 5.50 MiB | 7.97 MiB/s, done.
Resolving deltas: 100% (14407/14407), done.

~/scratch
❯ gix t organize             
 11:17:42 organize Error when handling directory "./gitoxide": Remote URLs must have host names: file://gh:Byron/gitoxide
Error: Failed to handle 1 repositories

I'm not sure if gitoxide can parse .gitconfig files right now, so this might be tricky to handle correctly...

`git_odb::compound::Db::locate()` doesn't work consistently

Write unit test to reproduce.

Manual testing with this sample program it turned out that loose objects can be located, but not objects that are packed.

Here is how it can be invoked:

git rev-list --objects --no-object-names HEAD | cargo eval   ./object-access.rs

I recommend installing cargo-eval using cargo install --git https://github.com/reitermarkus/cargo-eval to get the latest greatest.

CHANGELOG.md not updated

I don't know how your release workflow works, but the CHANGELOG.md hasn't been updated since 0.4.1, so i have no idea what's new ;)

Benchmark alternate malloc implementations (jemalloc, scudo)

Given that gitoxide is allocating memory and working across many threads, it would be worthwhile to benchmark using alternate memory allocators.

It'd be trivial to test jemalloc: just use the jemallocator crate from the top-level gitoxide bin crate, following the instructions in its README.

The scudo allocator has even more performance, as well as some resilience against some kinds of security-related bugs. See https://expertmiami.blogspot.com/2019/05/what-is-scudo-hardened-allocator_10.html for performance information. However, it isn't packaged for Rust yet, so it won't be quite as easy to try.

[git-protocol]: review 'fetch' implementation and make public what's useful for those who only use git-transport

Currently a few portions of the API, like refs parsing, are crate-only. By now it's clear these are useful and after some cleanup should be made public to allow users of git-transport alone to use them without relying on the existing fetch implementation.

Fix compile warnings with Rust 1.51

Most of them seem to point at git-config. When seeing unwraps() I recommend making them into '.expect(…)in case it's more like an assertion, or to transform them into errors to be able to use?` instead.

Everything unrelated to git-config I will fix, please let me know @edward-shen if I should use this opportunity to take a first and more thorough look myself and fix them as I go along.

warning: docs for function which may panic missing `# Panics` section
  --> git-config/src/file.rs:70:5
   |
70 | /     pub fn pop(&mut self) -> Option<(Key, Cow<'event, [u8]>)> {
71 | |         let mut values = vec![];
72 | |         // events are popped in reverse order
73 | |         while let Some(e) = self.section.0.pop() {
...  |
93 | |         None
94 | |     }
   | |_____^
   |
note: the lint level is defined here
  --> git-config/src/lib.rs:3:9
   |
3  | #![warn(clippy::pedantic, clippy::nursery)]
   |         ^^^^^^^^^^^^^^^^
   = note: `#[warn(clippy::missing_panics_doc)]` implied by `#[warn(clippy::pedantic)]`
note: first possible panic found here
  --> git-config/src/file.rs:82:55
   |
82 |                         return Some((k, normalize_cow(values.pop().unwrap())));
   |                                                       ^^^^^^^^^^^^^^^^^^^^^
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:242:5
    |
242 | /     pub fn value(&self, key: &Key) -> Option<Cow<'event, [u8]>> {
243 | |         let range = self.get_value_range_by_key(key);
244 | |         if range.is_empty() {
245 | |             return None;
...   |
271 | |         )))
272 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:253:22
    |
253 |                 _ => unreachable!(),
    |                      ^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
    = note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:288:5
    |
288 | /     pub fn values(&self, key: &Key) -> Vec<Cow<'event, [u8]>> {
289 | |         let mut values = vec![];
290 | |         let mut found_key = false;
291 | |         let mut partial_value = None;
...   |
316 | |         values
317 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:309:21
    |
309 |                     partial_value.as_mut().unwrap().extend(&**v);
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:620:5
    |
620 | /     pub fn section<'lookup>(
621 | |         &mut self,
622 | |         section_name: &'lookup str,
623 | |         subsection_name: Option<&'lookup str>,
...   |
626 | |         Ok(self.sections.get(section_ids.last().unwrap()).unwrap())
627 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:626:12
    |
626 |         Ok(self.sections.get(section_ids.last().unwrap()).unwrap())
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:635:5
    |
635 | /     pub fn section_mut<'lookup>(
636 | |         &mut self,
637 | |         section_name: &'lookup str,
638 | |         subsection_name: Option<&'lookup str>,
...   |
644 | |         ))
645 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:643:13
    |
643 |             self.sections.get_mut(section_ids.last().unwrap()).unwrap(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:762:5
    |
762 | /     pub fn remove_section<'lookup>(
763 | |         &mut self,
764 | |         section_name: &'lookup str,
765 | |         subsection_name: impl Into<Option<&'lookup str>>,
...   |
773 | |         self.sections.remove(&id)
774 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:772:21
    |
772 |             .remove(self.section_order.iter().position(|v| *v == id).unwrap());
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:811:5
    |
811 | /     pub fn rename_section<'lookup>(
812 | |         &mut self,
813 | |         section_name: &'lookup str,
814 | |         subsection_name: impl Into<Option<&'lookup str>>,
...   |
824 | |         Ok(())
825 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:819:18
    |
819 |         let id = id.last().unwrap();
    |                  ^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
   --> git-config/src/file.rs:919:5
    |
919 | /     pub fn get(&self) -> Result<Vec<Cow<'_, [u8]>>, GitConfigError> {
920 | |         let mut found_key = false;
921 | |         let mut values = vec![];
922 | |         let mut partial_value = None;
...   |
955 | |         Ok(values)
956 | |     }
    | |_____^
    |
note: first possible panic found here
   --> git-config/src/file.rs:931:27
    |
931 |             for event in &self.section.get(section_id).unwrap().0[offset..offset + size] {
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:998:5
     |
998  | /     pub fn set_value<'a: 'event>(&mut self, index: usize, input: Cow<'a, [u8]>) {
999  | |         let EntryData {
1000 | |             section_id,
1001 | |             offset_index,
...    |
1010 | |         );
1011 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1006:13
     |
1006 |             self.section.get_mut(&section_id).unwrap(),
     |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1021:5
     |
1021 | /     pub fn set_values<'a: 'event>(&mut self, input: impl Iterator<Item = Cow<'a, [u8]>>) {
1022 | |         for (
1023 | |             EntryData {
1024 | |                 section_id,
...    |
1038 | |         }
1039 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1033:17
     |
1033 |                 self.section.get_mut(section_id).unwrap(),
     |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1051:5
     |
1051 | /     pub fn set_owned_values_all(&mut self, input: &[u8]) {
1052 | |         for EntryData {
1053 | |             section_id,
1054 | |             offset_index,
...    |
1065 | |         }
1066 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1060:17
     |
1060 |                 self.section.get_mut(section_id).unwrap(),
     |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1074:5
     |
1074 | /     pub fn set_values_all<'a: 'event>(&mut self, input: &'a [u8]) {
1075 | |         for EntryData {
1076 | |             section_id,
1077 | |             offset_index,
...    |
1088 | |         }
1089 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1083:17
     |
1083 |                 self.section.get_mut(section_id).unwrap(),
     |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1114:5
     |
1114 | /     pub fn delete(&mut self, index: usize) {
1115 | |         let EntryData {
1116 | |             section_id,
1117 | |             offset_index,
...    |
1125 | |         }
1126 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1121:13
     |
1121 |             self.section.get_mut(section_id).unwrap().0.drain(offset..offset + size);
     |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1130:5
     |
1130 | /     pub fn delete_all(&mut self) {
1131 | |         for EntryData {
1132 | |             section_id,
1133 | |             offset_index,
...    |
1142 | |         self.indices_and_sizes.clear();
1143 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1138:17
     |
1138 |                 self.section.get_mut(section_id).unwrap().0.drain(offset..offset + size);
     |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1195:5
     |
1195 | /     pub fn get_raw_value<'lookup>(
1196 | |         &self,
1197 | |         section_name: &'lookup str,
1198 | |         subsection_name: Option<&'lookup str>,
...    |
1215 | |         Err(GitConfigError::KeyDoesNotExist)
1216 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1210:30
     |
1210 |             if let Some(v) = self.sections.get(section_id).unwrap().value(&key) {
     |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1228:5
     |
1228 | /     pub fn get_raw_value_mut<'lookup>(
1229 | |         &mut self,
1230 | |         section_name: &'lookup str,
1231 | |         subsection_name: Option<&'lookup str>,
...    |
1272 | |         Err(GitConfigError::KeyDoesNotExist)
1273 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1242:31
     |
1242 |             for (i, event) in self.sections.get(section_id).unwrap().0.iter().enumerate() {
     |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1315:5
     |
1315 | /     pub fn get_raw_multi_value<'lookup>(
1316 | |         &self,
1317 | |         section_name: &'lookup str,
1318 | |         subsection_name: Option<&'lookup str>,
...    |
1337 | |         }
1338 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1324:17
     |
1324 | /                 self.sections
1325 | |                     .get(&section_id)
1326 | |                     .unwrap()
     | |_____________________________^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: docs for function which may panic missing `# Panics` section
    --> git-config/src/file.rs:1395:5
     |
1395 | /     pub fn get_raw_multi_value_mut<'lookup>(
1396 | |         &mut self,
1397 | |         section_name: &'lookup str,
1398 | |         subsection_name: Option<&'lookup str>,
...    |
1448 | |         }
1449 | |     }
     | |_____^
     |
note: first possible panic found here
    --> git-config/src/file.rs:1413:31
     |
1413 |             for (i, event) in self.sections.get(section_id).unwrap().0.iter().enumerate() {
     |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
    --> git-config/src/file.rs:1773:1
     |
1773 | impl<'a> Into<Vec<u8>> for GitConfig<'a> {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(clippy::from_over_into)]` on by default
     = help: consider to implement `From` instead
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
    --> git-config/src/file.rs:1779:1
     |
1779 | impl<'a> Into<Vec<u8>> for &GitConfig<'a> {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |
     = help: consider to implement `From` instead
     = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:130:1
    |
130 | impl Into<Vec<u8>> for Event<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:136:1
    |
136 | impl Into<Vec<u8>> for &Event<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:350:1
    |
350 | impl Into<Vec<u8>> for ParsedSectionHeader<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:356:1
    |
356 | impl Into<Vec<u8>> for &ParsedSectionHeader<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:362:1
    |
362 | impl<'a> Into<Event<'a>> for ParsedSectionHeader<'a> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:415:1
    |
415 | impl Into<Vec<u8>> for ParsedComment<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/parser.rs:421:1
    |
421 | impl Into<Vec<u8>> for &ParsedComment<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:224:1
    |
224 | impl Into<Vec<u8>> for Value<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:230:1
    |
230 | impl Into<Vec<u8>> for &Value<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:376:1
    |
376 | impl Into<bool> for Boolean<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:385:1
    |
385 | impl<'a, 'b: 'a> Into<&'a [u8]> for &'b Boolean<'a> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:394:1
    |
394 | impl Into<Vec<u8>> for Boolean<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:400:1
    |
400 | impl Into<Vec<u8>> for &Boolean<'_> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:492:1
    |
492 | impl<'a, 'b: 'a> Into<&'a [u8]> for &'b TrueVariant<'a> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:617:1
    |
617 | impl Into<Vec<u8>> for Integer {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:623:1
    |
623 | impl Into<Vec<u8>> for &Integer {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:839:1
    |
839 | impl Into<Vec<u8>> for Color {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
   --> git-config/src/values.rs:845:1
    |
845 | impl Into<Vec<u8>> for &Color {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider to implement `From` instead
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into

[...]

MacOS build via `cargo` is broken

cargo install gitoxide yields:

...
   Compiling crosstermion v0.3.2
   Compiling prodash v9.0.0
error[E0425]: cannot find function `select_all` in module `futures_util::stream`
   --> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:162:48
    |
162 |         let mut events = futures_util::stream::select_all(vec![
    |                                                ^^^^^^^^^^ not found in `futures_util::stream`

error[E0599]: no method named `boxed` found for struct `futures_util::stream::Map<impl futures_core::Stream, [closure@/Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:44: 163:59]>` in the current scope
   --> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:61
    |
163 |             ticker(duration_per_frame).map(|_| Event::Tick).boxed(),
    |                                                             ^^^^^ method not found in `futures_util::stream::Map<impl futures_core::Stream, [closure@/Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:44: 163:59]>`
    |
    = help: items from traits can only be used if the trait is in scope
    = note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
            candidate #1: `use futures_lite::future::FutureExt;`
            candidate #2: `use futures_lite::stream::StreamExt;`
            candidate #3: `use futures_lite::FutureExt;`
            candidate #4: `use futures_lite::StreamExt;`

error[E0599]: no method named `boxed` found for struct `futures_util::stream::Map<impl futures_core::Stream, fn(crosstermion::input::Key) -> render::tui::engine::Event {render::tui::engine::Event::Input}>` in the current scope
   --> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:164:43
    |
164 |             key_receive.map(Event::Input).boxed(),
    |                                           ^^^^^ method not found in `futures_util::stream::Map<impl futures_core::Stream, fn(crosstermion::input::Key) -> render::tui::engine::Event {render::tui::engine::Event::Input}>`
    |
    = help: items from traits can only be used if the trait is in scope
    = note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
            candidate #1: `use futures_lite::future::FutureExt;`
            candidate #2: `use futures_lite::stream::StreamExt;`
            candidate #3: `use futures_lite::FutureExt;`
            candidate #4: `use futures_lite::StreamExt;`

error[E0599]: no method named `boxed` found for type parameter `impl futures_core::Stream<Item = Event> + Send` in the current scope
   --> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:165:20
    |
165 |             events.boxed(),
    |                    ^^^^^ method not found in `impl futures_core::Stream<Item = Event> + Send`
    |
    = help: items from traits can only be used if the trait is in scope
    = note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
            candidate #1: `use futures_lite::future::FutureExt;`
            candidate #2: `use futures_lite::stream::StreamExt;`
            candidate #3: `use futures_lite::FutureExt;`
            candidate #4: `use futures_lite::StreamExt;`

error: aborting due to 4 previous errors

Some errors have detailed explanations: E0425, E0599.
For more information about an error, try `rustc --explain E0425`.
error: could not compile `prodash`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.0`, intermediate artifacts can be found at `/var/folders/2h/z_3xs03s58sb4qc9c1x8nj1w0000gq/T/cargo-installZn5MqM`

Caused by:
  build failed

OSX 10.15.6
cargo 1.46.0 (149022b1d 2020-07-17)
rustc 1.46.0 (04488afe3 2020-08-24)

git-odb docs don't build correctly

Has to do with zlib-ng not building on docs.rs, see the logs:

https://docs.rs/crate/git-odb/0.10.0/builds/373703

[gix] Panicks in commands don't propagate, program hangs

gitoxide/src/shared.rs

Lines 110 to 111 in ea0ecc2

	// LIMITATION: This will hang if the thread panics as no message is send and the renderer thread will wait forever.
	// `catch_unwind` can't be used as a parking lot mutex is not unwind safe, coming from prodash.

It's known but apparently not super easy to fix given the way things currently work. This issue is verified to exist in verbose/not-quite mode, but is probably also present with the --progress TUI.

git-odb depends on prodash through git-features

It should be possible to pull in a lower-level crate like git-odb without pulling in prodash, or similar high-level output crates.

(This is not in any way urgent, but it'd be nice to reduce the dependencies of applications building on git-odb and similar.)

Empty tree gives a parse error

Tree parsing is defined as all_consuming(many1(parse_entry)), which rejects the (valid) empty tree. I'm pretty sure that should be many0.

Contributing to pack-send

Hi, I wish to contribute to a feature in this repo, and pack-send sounds straightforward. Should I try to work on this if it's already not taken?

[performance+memory] Beating git in `index-pack` (as used for clones and fetches) ✅ 🚀

git index-pack is streaming a pack and creates an index from it. The difficulty arises from having to decompress every entry in the pack stream, which can be composed of many small objects. These are placed in some sort of index to accelerate the next stage that is all about resolving the deltas in order to produce a SHA1. Per pack entry, the SHA1, pack offset and CRC32 are written into the index file to complete the operation.

The indexing phase in inherently single-threaded with little potential for improvements, whereas the resolving phase is fully multithreaded and entirely lock free. The first phase could be improved by writing the pack file in parallel - right now it happens after reading it (the pack file is used later for lookup to not hold everything in memory). However, IO doesn't appear to be the bottleneck at all.

Compared to gitoxide, git is considerably faster when creating the index, averaging 54MB/s of reading uncompressed bytes. gitoxide clocks in at about 45MB/s 50MB/s, and slows down considerably during the end. Part of that slowdown might be attributed to this issue with resetting miniz_oxide's decompressor.

Luckily gitoxide is way faster when resolving deltas, which already gives it a good first place in the race, with some room for more if it manages to get as fast as git when decompressing and indexing objects.

The picture below shows the fastest git run I could produce, probably with everything being properly cached:

Without cache, it seems to look different:

The fastest gitoxide runs, which are pretty comparable in the amount of work done, as they also write out the pack and the index. The only difference is that they use the packfile directly instead of reading it from stdin, it's streamed nonetheless though, and merely an oversight.

Memory consumption of git hovers consistently around 650MB (for the kernel pack), and is lower higher than the 1.2GB 750MB 580MB that gitoxide uses. However, gitoxide can temporarily use more memory as it keeps intermediate decompressed objects per thread, whose maximum sizes depend on the amount of children and the base size. So I have seen this go up to 850MB for small fractions of time because of that.

`git-refs` MVP

When writing any tool it's obvious that we really, really want to be able to resolve refs to objects.
We need to add the concept of stores right away as there are various types of them.

In typical repositories this requires the following:

• factor zlib code into git-features, allowing different backends to be swapped with feature toggles.
• split git-odb into stores and packs.
• A loose file ref store with the ability to
  • find a single ref by partial name with algorithm of git
  • initialize based on .git dir
  • and a loose reference type to
    • resolve targets step by step
    • peel it until an object is found
  • do validation
  • basic docs
• git-repository
  • discover(…) to find git repos easily
• Update existing tools to use the new capability
  • experiments
    • traversal
    • diffing
    • object-access
  • gixp
    • organize
  • gix
    • hours
  • gitoxide-core
• better git-repository docs

Out of scope

• reftable backend, which will be an addition to the existing files and packed-refs format.
• packed refs (handle peeled, fully peeled, sorted features)

Research

• parse a loose ref
• for parsing FETCH_HEAD and MERGE_HEAD
• instructions with a lot of information on how refs are seen in git.
• There are various backends - files, packed, and an in-memory cache.

Improve ergonomics around owned::Id and borrowed::Id

Benchmark it and different alternatives like:

• using Cow
• using owned::Ids everywhere

Be sure to put Sha256/32 bytes in there right away to have a more faithful prediction of the future and prevent optimizations that currently are definitely at play.

The outcome is to find the tradeoff between ergonomics and performance and apply the change to the codebase.
Also: Select one name for the type and use it everywhere unchanged, right now some crates use it as Digest.

Why do you repeat star the `gitoxide` 😂

`git-ref`: packed-refs backend for files store MVP

As a follow-up of #90

• research - it seems clear how it works and how it should be implemented competitively. They are zero copy, too :D.
• parse header line
• parse records
• lookup records

Research

• mmap can be a problem on windows - rather read it all into memory and use reftable if that's too much at some point.
• big files of 150k packed-refs (see this script to generate it
• git essentially parses each line on the fly and calls the owning structure rightfully 'snapshot'. Most if not all operations are done directly on these. The faster newlines are detected the better, validation happens too.
• the find implementation is a binary search over snapshot records.
• Not all refs are packed, symbolic refs for one are missing

[performance] beating git in `git verify-pack` ✅ 🚀

When trying to beat git verify-pack

Attempt 1

I remembered timings on a cold cache that indicated something around 5:50min for git to run a verify pack on the linux kernel pack. However, turns out that if the environment is a little more controlled, git is still considerably faster than us despite using an LRU cache and despite using multiple cores quite efficiently.

Observation

Git uses a streaming pack approach which is optimized to apply objects inversely. It works by

• decompressing all deltas
• applying all deltas that depend on a base, recursively (and thus avoiding to have to decompress deltas multiple times)

We work using a memory mapped file which is optimized for random access, but won't be very fast for this kind of workload.

How to fix

Wait until we have implemented a streaming pack as well and try again, having the same algorithmical benefits possibly faired with more efficient memory handling.
Git for some reason limits the application to 3 threads, even though we do benefit from having more threads so could be faster just because of this.
The streaming (indexing) phase of reading a pack can be parallelised in case we have a pack on disk, and it should be easy to implement if the index datastructure itself is threadsafe (but might not be worth the complexity or memory overhead, let's see).

git pack generation

As promised on Reddit, here's an outline of the parallel pack generation use case:

• I have a set of objects in the repository, identified via a list of object roots.
• I have an optional set of potential thin-pack bases, which the other end has; the objects that go in the pack are all objects reachable from the roots and not reachable from the bases, and the objects usable for deltas are all objects reachable from either the roots or the bases.
• I'd like to generate a pack (or thin-pack if any bases are specified), and stream that pack either to disk or over a network connection.
• Sometimes that connection or disk will be slow, other times it'll be absurdly fast. I'd like to have some reasonable control over the tradeoffs between pack generation speed and pack size.
• It would be nice to handle generating a pack that includes some objects that aren't in a repository, without first having to add the objects to the repository. (For instance, blobs and trees generated from a directory or extracted from a tarball.) Not a hard requirement, but helpful.
• Massive bonus points if git-oxide could start streaming the pack almost immediately, and adaptively do as well at compression as you can in the time until the next bits are needed to send over the wire, to come as close as possible to saturating the available network speed. (The pack can later be repacked for space, taking more time to do so more effectively.)

I'd love to test this out, and I'd be happy to do so on the biggest machines I can throw at it, though I'd also like it to work well in the 2-8 CPU case.

Support static pages

It'd be nice if static pages were supported because we want to rewrite our thing in Rust and for security reasons we recommend not running complex scripts and programs on the HTTP server, which means we recommend configuring all git repos with the so-called "dumb" protocol.

Our current thing currently just uses git directly and clones/fetches everything into a single cache repo, which means we can't parallelize it, so we're looking into alternatives for that, whether by using something like gitoxide or having multiple temporary caches and joining them together after fetching. However, having static pages support is a must.

Drop git::Protocol::V1 support once V2 is the default in upstream git

Supporting it adds quite some complication to our codebase, namely:

• peek-line support in packet-line crate is only needed for V1, and adds a lot of tests and complication to deal with it properly. Refactoring is harder than it should due to the nature of the 'self-borrowing' of lines, which is also the reason the packetline Provider is not an iterator.
• all V1 logic/special cases in transport layer
• all V1 logic/special cases in protocol layer

Cargo Install fails

By running cargo install gitoxide I get following crash while compiling.

❯ cargo install gitoxide
    Updating crates.io index
  Installing gitoxide v0.4.3
   Compiling libc v0.2.81
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.54
   Compiling version_check v0.9.2
   Compiling cfg-if v0.1.10
   Compiling serde_derive v1.0.118
   Compiling serde v1.0.118
   Compiling autocfg v1.0.1
   Compiling log v0.4.11
   Compiling memchr v2.3.4
   Compiling proc-macro-hack v0.5.19
   Compiling bitflags v1.2.1
   Compiling futures-core v0.3.8
   Compiling scopeguard v1.1.0
   Compiling crunchy v0.2.2
   Compiling lazy_static v1.4.0
   Compiling getrandom v0.2.0
   Compiling tiny-keccak v2.0.2
   Compiling cfg-if v1.0.0
   Compiling unicode-width v0.1.8
   Compiling futures-task v0.3.8
   Compiling pin-utils v0.1.0
   Compiling cc v1.0.66
   Compiling pkg-config v0.3.19
   Compiling cache-padded v1.1.1
   Compiling quick-error v2.0.0
   Compiling futures-io v0.3.8
   Compiling fastrand v1.4.0
   Compiling parking v2.0.0
   Compiling typenum v1.12.0
   Compiling pin-project-lite v0.1.11
   Compiling unicode-segmentation v1.7.1
   Compiling cassowary v0.3.0
   Compiling const_fn v0.4.4
   Compiling waker-fn v1.1.0
   Compiling radium v0.5.3
   Compiling futures-sink v0.3.8
   Compiling event-listener v2.5.1
   Compiling build_const v0.2.1
   Compiling tinyvec_macros v0.1.0
   Compiling once_cell v1.5.2
   Compiling matches v0.1.8
   Compiling vec-arena v1.0.0
   Compiling getrandom v0.1.15
   Compiling ansi_term v0.12.1
   Compiling human_format v1.0.3
   Compiling bytesize v1.0.1
   Compiling nix v0.18.0
   Compiling tap v1.0.0
   Compiling maybe-uninit v2.0.0
   Compiling funty v1.0.1
   Compiling wyz v0.2.0
   Compiling hex v0.4.2
   Compiling humantime v2.0.1
   Compiling sha1 v0.6.0
   Compiling percent-encoding v2.1.0
   Compiling itoa v0.4.6
   Compiling curl v0.4.34
   Compiling opaque-debug v0.3.0
   Compiling ppv-lite86 v0.2.10
   Compiling cpuid-bool v0.1.2
   Compiling byteorder v1.3.4
   Compiling bytes v0.5.6
   Compiling same-file v1.0.6
   Compiling home v0.5.3
   Compiling ryu v1.0.5
   Compiling adler v0.2.3
   Compiling arrayvec v0.5.2
   Compiling base64 v0.12.3
   Compiling remove_dir_all v0.5.3
   Compiling anyhow v1.0.35
   Compiling serde_json v1.0.60
   Compiling hashbrown v0.9.1
   Compiling quick-error v1.2.3
   Compiling termcolor v1.1.2
   Compiling strsim v0.10.0
   Compiling vec_map v0.8.2
   Compiling os_str_bytes v2.4.0
   Compiling lock_api v0.3.4
   Compiling lock_api v0.4.2
   Compiling instant v0.1.9
   Compiling textwrap v0.12.1
   Compiling concurrent-queue v1.2.2
   Compiling git-repository v0.4.0
   Compiling standback v0.2.13
   Compiling time v0.2.23
   Compiling generic-array v0.14.4
   Compiling nom v6.0.1
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling futures-channel v0.3.8
   Compiling tinyvec v1.1.0
   Compiling num-traits v0.2.14
   Compiling crossbeam-utils v0.7.2
   Compiling miniz_oxide v0.4.3
   Compiling indexmap v1.6.1
   Compiling unicode-bidi v0.3.4
   Compiling crc v1.8.1
   Compiling form_urlencoded v1.0.0
   Compiling walkdir v2.3.1
   Compiling humantime v1.3.0
   Compiling uluru v1.0.0
   Compiling heck v0.3.1
   Compiling async-channel v1.5.1
   Compiling libz-sys v1.1.2
   Compiling curl-sys v0.4.39+curl-7.74.0
   Compiling futures-lite v1.11.2
   Compiling bitvec v0.19.4
   Compiling quote v1.0.7
   Compiling unicode-normalization v0.1.16
   Compiling mio v0.7.6
   Compiling signal-hook-registry v1.2.2
   Compiling polling v2.0.2
   Compiling num_cpus v1.13.0
   Compiling nb-connect v1.0.2
   Compiling socket2 v0.3.17
   Compiling filebuffer v0.4.0
   Compiling atty v0.2.14
   Compiling crossbeam-channel v0.4.4
   Compiling rand_core v0.5.1
   Compiling async-io v1.3.1
   Compiling signal-hook v0.1.16
   Compiling env_logger v0.7.1
   Compiling idna v0.2.0
   Compiling btoi v0.4.2
   Compiling const-random-macro v0.1.13
   Compiling rand_chacha v0.2.2
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling sha-1 v0.9.2
   Compiling rand v0.7.3
   Compiling const-random v0.1.13
   Compiling url v2.2.0
   Compiling ahash v0.3.8
   Compiling dashmap v3.11.10
   Compiling tempfile v3.1.0
   Compiling ctrlc v3.1.7
   Compiling pin-project-internal v1.0.2
   Compiling time-macros-impl v0.1.1
   Compiling thiserror-impl v1.0.22
   Compiling clap_derive v3.0.0-beta.2
   Compiling time-macros v0.1.1
   Compiling thiserror v1.0.22
   Compiling pin-project v1.0.2
   Compiling futures-util v0.3.8
   Compiling clap v3.0.0-beta.2
   Compiling smallvec v1.5.1
   Compiling bstr v0.2.14
   Compiling parking_lot_core v0.7.2
   Compiling parking_lot_core v0.8.1
   Compiling parking_lot v0.10.2
   Compiling parking_lot v0.11.1
   Compiling git-ref v0.4.0
   Compiling git-packetline v0.2.1
   Compiling git-url v0.1.0
   Compiling git-object v0.4.0
   Compiling crossterm v0.17.7
   Compiling tui v0.10.0
   Compiling tui-react v0.10.1
   Compiling crosstermion v0.3.2
   Compiling prodash v10.0.2
   Compiling git-features v0.6.0
   Compiling git-features v0.7.0
   Compiling git-odb v0.4.2
   Compiling git-transport v0.2.1
   Compiling git-protocol v0.1.1
   Compiling gitoxide-core v0.4.1
error[E0433]: failed to resolve: could not find `DecodeEntryLRU` in `cache`
   --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/explode.rs:221:22
    |
221 |         pack::cache::DecodeEntryLRU::default,
    |                      ^^^^^^^^^^^^^^ could not find `DecodeEntryLRU` in `cache`

error[E0433]: failed to resolve: could not find `DecodeEntryLRU` in `cache`
   --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:145:53
    |
145 |                     EitherCache::Right(pack::cache::DecodeEntryLRU::default())
    |                                                     ^^^^^^^^^^^^^^ could not find `DecodeEntryLRU` in `cache`

error[E0412]: cannot find type `DecodeEntryNoop` in module `pack::cache`
  --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:76:23
   |
76 |     Left(pack::cache::DecodeEntryNoop),
   |                       ^^^^^^^^^^^^^^^ help: a trait with a similar name exists: `DecodeEntry`
   |
  ::: /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/git-odb-0.4.2/src/pack/cache.rs:4:1
   |
4  | pub trait DecodeEntry {
   | --------------------- similarly named trait `DecodeEntry` defined here

error[E0412]: cannot find type `DecodeEntryLRU` in module `pack::cache`
  --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:77:24
   |
77 |     Right(pack::cache::DecodeEntryLRU),
   |                        ^^^^^^^^^^^^^^ help: a trait with a similar name exists: `DecodeEntry`
   |
  ::: /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/git-odb-0.4.2/src/pack/cache.rs:4:1
   |
4  | pub trait DecodeEntry {
   | --------------------- similarly named trait `DecodeEntry` defined here

error[E0425]: cannot find value `DecodeEntryNoop` in module `pack::cache`
   --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:143:52
    |
143 |                     EitherCache::Left(pack::cache::DecodeEntryNoop)
    |                                                    ^^^^^^^^^^^^^^^ not found in `pack::cache`

error[E0061]: this function takes 3 arguments but 4 arguments were supplied
   --> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:149:17
    |
149 |             idx.verify_integrity(
    |                 ^^^^^^^^^^^^^^^^ expected 3 arguments
150 |                 pack.as_ref().map(|p| (p, mode, algorithm.into())),
    |                 --------------------------------------------------
151 |                 thread_limit,
    |                 ------------
152 |                 progress,
    |                 --------
153 |                 cache,
    |                 ----- supplied 4 arguments

error: aborting due to 6 previous errors

Some errors have detailed explanations: E0061, E0412, E0425, E0433.
For more information about an error, try `rustc --explain E0061`.
error: could not compile `gitoxide-core`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.3`, intermediate artifacts can be found at `/var/folders/pz/1bgtc0456j33k6w2w_ggxm5m0000gn/T/cargo-install7bD95K`

Caused by:
  build failed

I am on macOS 11.1 (Beta).

Radicle

Hey @Byron,

I met with @xla and @kim to discuss gitoxide and our needs in radicle-link.

We landed in a high-level priority list as follows:

• Remote capabilities would be our highest priority, being able to fetch changes from a peer
  in our case and the peer responding with the changeset.
• The Ref DB would be our next highest priority.
• Followed by local capabilities, we're happy to use libgit2 here during a transitionary period.

Remote capabilities

Looking through the checklist from the README, we saw that one part of the fetching story is
complete but we would need pack-send which is unchecked. As I mentioned above, the fetch exchange
would be top of our list. Maybe you could flesh out what's complete here, what's missing, and where
we could start looking in the code-base to understand this better.

To elaborate on our needs in this area, we need to implement our own transport
layer, so we'd be interested in what the gitoxide API looks like here.

Reference DB

We will have a lot of refs, which we constantly read and update. Eventually, we will want a
"real" database, but if we keep using libgit2, we would have to teach it to use a custom backend. As
a middleground, if gitoxide supports "packed refs" (that's a single file containing all the refs),
this would already be an improvement (that's assuming libgit2 loads the refs lazily).

Local Needs

Reference Handling

For reference handling, we need to be able to read references, perform commits, and compute merge
basing. This functionality is necessary for our handling of documents on the network.

Git Config

We use the git config in a couple of places currently. One way is to track remotes and the other is
for configuring working-copies for fetching changes. Since we can rely on libgit here, it's lower
priority.

Notes and Questions

The V2 implementation

Because in v2, the server doesn't send all the refs unsolicited, but the client requests them
explicitly, and can filter by prefix. It is also, afaiu, that the "ref-in-want" feature would allow
us to request refs across namespaces, without jumping through hoops to make the server ref
advertisement match what we are likely going to fetch (we can compute all refs we want to fetch up
front).

Somewhat relatedly, libgit2 currently insists to only update (a subset of) the refs the server
advertised, at the exact oids the server advertised -- what we would want is a bit more control over
this:

Say we have refs/remotes/XYZ/rad/foo, for which the remote peer advertises that its tip is
abcde. We know, however, that XYZ signed a tip xyz09 which is an ancestor of abcde. We'd like
to ask the remote end for xyz09, and if that object is in the packfile, we'd update
refs/remotes/XYZ/rad/foo locally to this. Currently, if the remote end is ahead, we would not
update the ref at all.

Questions

1. We saw a check box for "Write all data types" and we were wondering what this meant?
2. The checklist is useful to understand at a high level what you've completed but I think we'd like
   to get a better understanding of what's more complete, what's still in progress, and what hasn't
   been started.

We're excited about the prospect of using gitoxide and being able to contribute to it, however,
we've only got so much person-power at the moment. So this is a good first step in figuring out how
much work is involved. After that, we can work out how much time we delegate to helping out on the
project.

Make git::Connection so versatile that it can fit more use-cases

Discussed in #110

^{Originally posted by kim June 25, 2021}
It turns out that I needed to define my own Transport impl, because:

• I need to pass Extra Parameters in the header
• I want to abort the handshake if the server doesn’t support v2
• I wasn’t sure how the to_url and is_stateful methods are used, but morally they need to return different things than the default Connection for my case

To do that, I needed to make the capabilities parsing public. While that is probably always needed for custom transports, I’m wondering if the interface could be generalised such that more use cases can just reuse a parametric Connection. For example, the delegate pattern could be employed for the handshake, and extra parameters could be stored in Connection.

I’d be happy to propose a patch, but wanted to gauge first if there’s interest, or maybe different plans already.

Windows builds is broken

Edit: oops, opened this without content. hang on a second.

This is kind of a tracking issue for the currently broken windows build, as mentioned in the latest release.

I opened this because i needed a space to leave my findings, specifically this issue: actions/runner-images#1143. I never worked with cross before, but perhaps it's an option to run this workflow on a linux runner?

Smart-GC - build a big pack from smaller ones (incrementally) as file handles get scarce

It's too easy to get bitten by git pull not working anymore as git cannot perform its operations anymore. This is because each fetch MAY create a new pack in libgit2. And even if not, and packs are exploded into loose objects, usually 10_000 of these warrant the creation of a new pack. It's just a matter of time until there are too many of them for the git-repository to memory map all of these for fast access.

Gitoxide should be better and either GC automatically once files cannot be opened anymore due to insufficient handles, or track its own usage to know when they are about to be tight and do an incremental GC to combine a few packs into one, fast.

Another avenue would be try to map only packs that actually have objects we are interested in, avoiding to map all by default, if that's even possible or viable.

Related to rust-lang/docs.rs#975 , and probably many more.

Benchmark switching to anyhow

anyhow::Error is smaller than Box<dyn Error>; it pulls some tricks to store itself in one pointer rather than two. It's also more convenient to use; for instance, anyhow::Result<T> is an alias for Result<T, anyhow::Error>, and anyhow::Error already implies Send and Sync so you don't have to specify them. And anyhow will make it easier to provide additional context on errors.

You might consider switching code that currently uses Box<dyn Error> (or similar), and benchmarking to see if it has any performance impact. (Even if it doesn't provide a noticeable improvement, if it doesn't cause a regression it'd probably still be worth switching.)

Config Parse/Read/Write Discussion

Hi! I'm a developer that's been tasked to implement GitOxide into a WASM stack in an effort to clone a project as fast as possible in the browser.

As part of that work, I'm looking to introduce a lot of new functionality into the project to get clone working. As an initial task, I'm hoping to get the git-config package able to read and write config files. I've got a (very) WIP port of config parsing logic from isomorphic-git over on my fork.

I won't bother asking for a code review right now, I'm brand new to Rust and have a colleague taking a look at the code right now to give me some pointers on how I can improve my work

That said, I am unsure of what direction we want to go in with regards to data organization. The current output is a pretty unmanageable tangled vector of Sections and Entries alike in a single struct

I've noticed that the current git-config file has both Section and Entry types that look to contain all of the relevant properties, so I'll be refactoring my current code to use that data structure.

Outside of that, I was hoping you could give me some kind of direction when it comes to the different mods in the git-config/lib.rs file:

https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L57-L72

https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L131-L217

I'm not entirely sure what borrowed or spawned is referring to (which I admit as a point-of-noobiness), and I'm not sure I track what Span is doing here.

I also would love some insight as to what a Token is referring to here:

https://github.com/Byron/gitoxide/blob/main/git-config/src/file.rs#L4-L23

As I'm not sure an instance that would have a Section and only a single Entry

Looking forward to learning more and being able to meaningfully contribute upstream!

Gitoxide aims to "become the foundation for a free distributed alternative to GitHub, and maybe even GitHub itself"

I have a similar project and must have missed the above gitoxide aim when I visited in October or would have contacted you then. I'm interested in collaboration or to see if your project is close enough to my goal that I could contribute and use it as is, or adapt some part etc.

I happened here today looking again for Rust projects to help with my p2p Git Portal project (see below). The Git Portal proof-of-concept uses git-bug to add issues and comments seamlessly to any git repository. I'm no git expert but believe the approach is sound and it performs well. git-bug can already import issues from github and can push/pull its own data to git services as a git extra CLI (git bug), all without messing with your project's own git data. The only problem I've found with git-bug is that it is written in Go. It was great for a quick proof-of-concept but is much less suitable for building a product for various reasons, so I'm again looking for options using Rust which is how I arrived here today.

So I'm wondering:

• what you know about/think of the approach used by git-bug (if anything)
• whether gitoxide is pretty much what my Git Portal is intended to be (i.e. web app hosted on p2p static storage, or a core part of that)
• whether gitoxide or some parts of it can be adapted for my use case (e.g. to add issue support to a Rust/Wasm web app)
• if so I'd love to contribute or collaborate, although I'm not keen on your MIT/Apache licensing for a decentralised project (I use GPLv3)

p2p Git Portal

Git Portal is a proof-of-concept Svelte + Go/Web Assembly app which can create and view git based issues alongside core git functionality such as browsing the worktree and listing commits. So functionally similar to github but in a test harness UI (someone is working with me on a more github like experience).

The Git Portal demo/poc is live here. First load can be slow because of the 12MB Go runtime, so be patient (instructions in the README). What you see there is a test harness which proves the concept. From that we'll improve the poc to a reasonable demo, but longer term I don't think it makes sense to continue with Go, and Rust is my first choice.

It is written in Go because I chose to build the poc using git-bug for issue support (and go-git for git functionality). It is compiled to Web Assembly and so runs in a browser from static storage without server side code. My aim is to run it from peer-to-peer storage on https://SafeNetwork.tech although it would be possible to use it as the basis of a decentralised app from any static or p2p storage.

Thanks for reading, I'll be interested to hear back although I realise code may be a higher priority and in the mean time I'll spend some time getting to know gitoxide.

Bump prodash to a version not pinned on time =0.2.22

This issue was created to track progress on RUSTSEC-2020-0056 and RUSTSEC-2020-0071, which are advisories caused by a dependency on time =0.2.22 (see time-rs/time#293 for more details).

The git-oxide project won't be affected by these advisories until local time support is added.

Requirements to close this issue:

• prodash dependency is bumped to a version not depending on time =0.2.22.
• Exceptions are removed in deny.toml.

Support signed commits and tags

It would be great to have the ability to create commits that are signed, I thought that this issue might be a good place to start talking about how that workflow should be designed.

Here is a post that describes how the signing process works with git2-rs:

https://blog.hackeriet.no/signing-git-commits-in-rust/

Basically it creates a commit object in memory, lets the user program sign it however they want, and then have a function that writes the object together with the signature string to the git repository.

That design have a couple of pros:

• no need for the library to interact with gpg, which can be tricky at best and a source of security holes at worst.
• support for multiple different signing algorithms, it's the user that decides if it should be signed with gpg or s/mime.

But it also adds complexity for the user of the library, that need to do all the heavy lifting of interacting with gpg (or other). And have multiple different ways to create a commit.

One other point is that there now exists a pure-rust pgp implementation called https://sequoia-pgp.org/ that might be better to use than the gpgme library.

How would you like to design the library functions for creating signed commits?

gitoxide in cargo

This is a fun research project to potentially drive feature development one day to allow replacing git2 with gitoxide.
In order to migrate, not all features would have to be present at first at the cost of compiling both gitoxide and git2 (Something which probably should be avoided).

Features used

• repository discovery
• open repository (without discovery)
• repository init with options
• list submodules (like here for example) and open their repo. Assure worktree and submodule APIs are similar.
• git status (requires index comparison with working tree and head to index)
  • untracked files - actually only needs dirwalk, code can be simplified
• open default git-config, i.e. all config that is global, not local to the repository.
• add files to index
  • add submodules to index by updating their hash in the index
• create commit from tree
  • get correct commit signature/actor
  • create tree from index
• write git configuration (local repository config)
• check path exists in git index
• get repository working tree dir
• Oid::from_hex()
• Git short hashes/ids
• Ref name to id
• check if a path is ignore
• Rev parse
• #734
• git gc (maybe, for registry) - note that this is shelled out to git, and it's about reducing the amount of pack files which we can already do pretty well I think, and loose objects could be included in the pack as well (even though undeltified)
• respect fetch.unpackLimit to keep repositories clean (cargo runs gc occasionally as well)
• git fetch
  • with tags
    • auto-tags
  • anonymous remotes as fetch-handle
  • detect spurious issues and retry
  • control http timeouts
  • respect proxy settings
  • configure user agent
  • make sure remote messages are unbuffered so they are realtime
  • ability to select specific progress messages via stable id
  • non-fast-forward due to crates.io squashing
  • custom authenticator credentials callback
    • credential helper access like git would (from git-config)
    • #1246
    • ntlm + kerberos HTTP authentication (accessed via git2::Cred::default())
    • multiple retries for credentials
• Owned tree objects alongside their owning repository with 'static lifetime (maybe that kind of caching isn't required). They do this with unsafe though.
• git checkout
• local clones that hard-link files (and bypass the git protocol) as used in cargo git-dependency checkouts (worktrees would be preferred to me)
• git hard reset with working tree with progress
• git submodule update or init + update
• gitoxide must build (and be tested on) 32 bit systems

Shortcomings of gitoxide in comparison to git2

• dependencies to binaries make it less usable on windows
  • depends on git binary for file:// protocol due to lack of native upload-pack functionality. Remedy is planned with Rust Foundation grant (stretch goal)
  • depends on ssh binary for ssh connection

Note that replacing clone/fetch related features is tracked in #449 .

Locations and purpose

cd cargo && rg git2:: -l

Potential issues

• cargo supports 32bit system, which limits mmap to 2Gb usually. For now this is alright but one day cursor based pack access probably needs to be implemented (behind a feature toggle, of course, or automatically configured depending on arch)

commit-graph parser

I have a project that would benefit from being able to make reachability queries using only commit-graph files, so I'm interested in adding support for reading commit-graph files to this project.

The README mentions commit graphs would be part of git-repository, but commit graphs feel more like git-odb's pack indices to me. In particular, I think it could share the fan-out and oid-bisect code from git-odb/pack/index. The reachability code could still be in git-repository.

So:

1. Do you have a strong preference where commit graph I/O should go?
2. I assume adding read-only support is acceptable for now. Is this correct?