Feature request: way to set up a mirror about crates.io

[skyzh]

We have built a tool to mirror static.crates.io at https://github.com/sjtug/mirror-clone . Meanwhile, we have also set up an experimental full-mirror of static.crates.io at https://s3.jcloud.sjtu.edu.cn/899a892efef34b1b944a19981040f55b-oss01/crates.io/crates/{crate}/{crate}-{version}.crate

[steveklabnik]

https://gmjosack.github.io/posts/dissecting-cratesio-minimum-mirror/

[happyaron]

@Turbo87 maybe a proprietary tool is not sufficient to close this issue, as a person who runs public mirror for various FLOSS, I'd like to find an open source solution, :)

[sfackler]

This would also involve adding support for non-Github auth (preferably LDAP) and non-S3/Github storage.

[alexcrichton]

Architecturally I believe it's relatively easy to do. A "mirror" would essentially just be a clone of the official index along with an updated download url in its configuration. Updating a mirror would just be updating the index and perhaps downloading all the new versions that were created.

Implementing this, however, and getting cargo to think that mirrors and normal registries are the same would be somewhat difficult.

@sfackler, those may be more useful for non-official entire instances, as I would suspect that a mirror would reflect whatever the crates.io site uses. That being said, I'd love for it to be easy to spin up a local instance of a registry to use entirely independently (with a number of configuration options like auth/storage). Might be a bit harder though :)

[sfackler]

At work we use Artifactory to both serve as a repository for internal packages and to provide a transparent mirror/cache of the maven.org packages we use. New packages are transparently pulled down as needed, but we don't mirror the whole world since that'd be crazy. Something like that would probably be best served as a separate project, though it'd hopefully be able to pull in a bunch of the logic/types from crates.io.

[weiznich]

I've written a simple caching server. Similar to Artifactory it pulls down transparently those crates that are really requested. Currently it is only a simple application without any interface or anything. Also it only mirrors an index. It's not possible to add or yank crates to the mirror.

[gilescope]

This feels linked to rust-lang/rust#44931

[ishitatsuyuki]

For initial fetch, we should provide daily database dumps which is less overhead to serve and better compressed.

For dynamic syncing, the Kubernetes model is a very friendly way to organize the API.

[happyaron]

Would like to bump this issue a bit since it looks like not getting love from guys for a long time - people from certain parts of the world (in my case, China mainland) does not have decent connectivity to some most used CDNs, so that mirroring is a must for getting a decent speed of using the repository.

Almost all the other languages have some form of mirroring being widely used, for example[1] we have CPAN/CTAN/CRAN, pypi, julia, etc. There are lots of Nexus instances to partially mirror the Maven Central repository. If crates.io can be mirrored in some way, it will surely make Rust users' lives easier in those places.

[1]https://mirrors.tuna.tsinghua.edu.cn/
[2]tuna/issues#941
a bit background: TUNA mirror is the largest syncing source of other mirrors in the country, with an average rsync traffic of more than 1Gbps

[Turbo87]

seeing as e.g. Artifactory now supports Rust (see https://www.jfrog.com/confluence/display/JFROG/Cargo+Registry) I guess we can close this :)

[Turbo87]

@happyaron absolutely, I would prefer that too. what I meant to say was that if Artifactory can build it then the necessary things are there for open source tools to also build something like this and there is nothing left on the crates.io side that we would need to build to enable this :)