foresterre / rust-releases

🛬 Identify the officially available Rust releases

rust-releases

rust-releases version	MSRV
0.21.1	1.51
0.22.0	1.53
0.23.0	1.63
0.24.0	1.63
0.25.0	1.63
0.26.0	1.63
0.27.0	1.67
0.28.0	1.68

* When unreleased, MSRV subject to change

Introduction

The Rust programming language uses deterministic versioning for toolchain releases. Stable versions use SemVer, while nightly, beta and historical builds can be accessed by using dated builds (YY-MM-DD).

Unfortunately, a clean index of releases is not available any more. I decided to research which resources where still available and found the following solutions:

1. Use the AWS index ^(source)
2. Build from individual release manifests ^(source)
3. Parse Rust in-repo RELEASES.md

Each of these options requires additional parsing, which is where this crate comes in: the rust-releases crate can obtain, parse and build an index from the above resources. This crate also provides methods to iterate over versions in a linear fashion, or by using a bisect binary search strategy.

Each data source implements the Source trait. Source provides a build_index method, which can be used to build a catalog of released Rust versions.

Implemented options

Type of data source	Trait	Available	Channels1	Speed2, 3	On disk cache size4	Notes
RustChangelog	Source	✅	Stable	Fast	-	Enabled by default. Disable by setting default-features = false for the rust-releases dependency in your Cargo.toml manifest.
	FetchResources	✅		Instant (<1 second)	~491 KB
RustDist	Source	✅	Stable, Beta & NightlyTo be implemented	Fast	-
	FetchResources	✅		Medium fast (~10 seconds)	~1 MB

¹: Currently most of the rust-releases public API supports only stable. Support for the beta and nightly channel is work-in-progress, and the table currently lists whether there is theoretical support for these channels.
²: Speed for the Source trait primarily consist of parsing speed
³: Speed for the FetchResources trait is primarily limited by your own download speed, and the rate limiting of the server from which the resources are fetched
⁴: Approximate as of 2021-03-03
⁵: While the channel manifests are the most complete source available, they're practically too slow to download without adding some incremental implementation first, while this would still require a large initial download. Since we currently do not use most of the data available in the manifest files, it's usually better to instead pick a different source. It's more likely we'll take the channel manifests, take a subset and compile it into a smaller source type.

Which data source should I use?

Since support for the beta and nightly channels is work-in-progress, I would advise to use the RustChangelog data source as it's a small download, immediately up-to-date on release and fast to parse. It only supports stable channel releases.

Alternatively, the RustDist or RustDistWithCLI data sources can be useful, especially when support for the beta and nightly channel are added. They both get their input data from the Rust AWS S3 distribution bucket. When using RustDist, the input data can be obtained with the FetchResources trait implementation. For RustDistWithCLI, you have to obtain the input data yourself (by running the aws cli with the following options aws --no-sign-request s3 ls static-rust-lang-org/dist/ > dist.txt^(source)).

Applications

cargo-msrv is a tool which can be used to determine the minimal supported Rust version (MSRV). It builds your Rust crate and checks whether the build succeeds or fails, as this gives the most complete idea whether a version will work with your (external) dependencies. cargo-msrv uses bisection, or a reverse-linear search, to find the lowest appropriate Rust version. For this, it needs to have an idea about the toolchains which have been released, and can be installed.

Originally we simply parsed the latest channel manifest, and then decreased the minor semver version, but this was obviously not great for many reasons, including:

• Except for the latest released version, we are left guessing the decreased version numbers actually exist
• Only stable versions were supported, not nightly, beta, or other channels
• Only 1.x.0 versions were supported

This was not ideal, thus rust-releases was born. Now cargo-msrv can iterate over Rust releases of which we know they exist and are available.

rust-releases's Issues

Let caching of input files for sources be optional

Add support for partial date based index downloading

i.e. we could ship an up to date versions index L, and only check for updates after the latest update in L

Publishing with cargo-release no longer works

• cargo release --workspace --exclude rust-toolchain --no-verify minor --execute

warning: updating rust-releases-io to 0.28.0 despite no changes made since tag v0.27.1
Release
  rust-releases-core 0.28.0
  rust-releases-io 0.28.0
  rust-releases-rust-changelog 0.28.0
  rust-releases-rust-dist 0.28.0
  rust-releases 0.28.0
? [y/N]
y
   Upgrading rust-releases-core from 0.27.0 to 0.28.0
   Upgrading rust-releases-io from 0.27.1 to 0.28.0
   Upgrading rust-releases-rust-changelog from 0.27.0 to 0.28.0
   Upgrading rust-releases-rust-dist from 0.27.0 to 0.28.0
   Upgrading rust-releases from 0.27.0 to 0.28.0
error: `cargo metadata` exited with an error: error: failed to select a version for the requirement `rust-releases-core = "^0.27.0"`
candidate versions found which didn't match: 0.28.0
location searched: C:\user\ws\rust-releases\crates\rust-releases-core
required by package `rust-releases-rust-changelog v0.28.0 (C:\user\ws\rust-releases\crates\rust-releases-rust-changelog)`

• cargo-release seems to first try to update every crate to bump to the given version increment, minor in this case
• but that fails for obvious reasons: internal dependencies aren't updated yet, so you can't just publish everything at the same time; you'll need to wait until your dependent versions are updated; i.e. update the dependency tree in topological order

`Release` data: add release date

Add incremental fetching for ChannelManifests source

ChannelManifests are available again: #9 (comment)

add feature gates for fetching resources

cargo-msrv when depending on rust-releases with feature [email protected]

error[E0412]: cannot find type `Error` in crate `rust_releases_core::semver`
  --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/rust-releases-rust-changelog-0.15.2/src/errors.rs:33:45
   |
33 |     SemverError(rust_releases_core::semver::Error, String),
   |                                             ^^^^^ not found in `rust_releases_core::semver`
   |
help: consider importing one of these items
   |
1  | use core::fmt::Error;
   |
1  | use std::error::Error;
   |
1  | use std::fmt::Error;
   |
1  | use std::io::Error;
   |

error: aborting due to previous error

For more information about this error, try `rustc --explain E0412`.
error: could not compile `rust-releases-rust-changelog`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `cargo-msrv v0.8.0`, intermediate artifacts can be found at `/tmp/cargo-installcaV7Yt`

Caused by:
  build failed
Error: Process completed with exit code 101.

As a result of the above rust-releases-rust-changelog has been yanked from crates.io, as cargo-msrv would fail to build when installing with cargo install cargo-msrv.

Curiously, when building rust-releases on its own, I could not reproduce this error yet.

Implement `FetchResources` for `DistIndex`

• Add empty FetchResources implementation for DistIndex.
• Add basic implementation which can access a bucket
• Find the correct bucket and check accessibility
  • rust-inventories bucket seems to be protected; getting access denied
  • static-rust-lang-org (A) seems accessible, but haven't checked whether the content is workable yet
• Handle paging (i.e. max 1000 objects are listed per call, if is_truncated = Some(true))
• Deserialize bucket content
• Complete FetchResources implementation
• Add caching
• Determine (& add) if we can add a custom user agent to Rusoto

2021-04-02:

• Fix impl Source for DistIndex, or create new struct to exist next to DistIndex

(A): impl Debug content for static-rust-lang-org bucket:

• bucket.static-rust-lang-org.txt

Add incremental fetching for RustDist source

investigate: rust-releases II

---

new(concept): Move caching layer down

// add crate: rust-releases-cache
// remove cache from client

// current:
// fn fetch(&self, resource: ResourceFile) -> Result<RetrievedDocument, Self::Error>
// into:

fn fetch(&self, resource: &[ResourceDescriptor]) -> Result<ResourceBundle, Self::Error>


struct ResourceBundle {
	documents: HashMap<DocumentDescriptor, Document>
}

impl From<ResourceDescriptor> for DocumentDescriptor { ... }

struct Cached<C: RustReleasesClient> {
	location: PathBuf,
	Duration,
}

impl<C: RustReleasesClient> Default for Cached<C> {
	fn default() -> Self {
		location: default_path().expect("Unable to find default path"),
	} 
}

impl<C: RustReleasesClient & !Cached<C>> RustReleasesClient for Cached<C> { ... }

Add additional index strategies

See foresterre/cargo-msrv#19 (comment) for possible strategies.

Currently implemented:

• Parse from (historical) channel manifests
• Use https://raw.githubusercontent.com/rust-lang/rust/master/RELEASES.md for Channel::Stable
• Use S3 index

Add additional support for Toolchains

https://rust-lang.github.io/rustup/concepts/toolchains.html
https://rust-lang.github.io/rustup/concepts/components.html

• Channel | Version
• Date(NaiveDate)
• Host (TargetTriple)
• impl From <(Channel | Version, Date, Host)> for ToolchainSpecification
• impl From <ToolchainSpecification> for (Channel | Version, Date, Host)
• Components(Vec<Component>)
• Proxy(Ref<Component>)
• Profiles (maybe?)
• custom toolchains (maybe?)

Update docs coverage and example coverage

typo in documentation

At https://docs.rs/rust-releases/0.16.0/rust_releases/#using-rust-releases
At Example using rust-releases + implemented source(s) as feature
Mistakenly, the last s in rust-releases-$RUST_RELEASES_SOURCE is missing

given:

features = ["rust-release-$RUST_RELEASES_SOURCE"]

should be:

features = ["rust-releases-$RUST_RELEASES_SOURCE"]

Same for example.

TODO:

• Fix typo in explanation
• Fix typo in example

RustDist: Implement `FetchResources` for `Nightly` channel

Add 'built-in' release source

Add a release source consisting of hardcoded facts.

New source: Detect new releases via rust-lang/rust repository

Investigate: replace chrono with time v0.3

As suggested here: chronotope/chrono#602 (comment)

Affected component: rust-releases-rust-changelog

Add merging of release indices

Move Source impl caches to their own cache folders

Consider: rust-dist over manual releasing with cargo-release

• Good project to try out cargo dist, since smaller project, but with many infra requirements:
  • Updating changelog
  • Tags + releases
  • Workspace crates

RustDist: Implement incremental fetching

Currently for all sources we re-fetch everything when the cache becomes stale. However, since versions do not become outdated, we can and should fetch newer releases incrementally. This is especially useful for nightly releases (since there are many of those).

To make this work, we should save the last usable offset (which unfortunately is not the same as the actual last offset, since the files are obtained in lexically order; but we can use a pattern to find this version).

Add the so far known index as base knowledge

• Downloading Rust release manifests is slow because of (extreme) rate limiting
• We can run the index process once every X time
• And let the program download these pre built indices from a github release page by default first

see also #4

investigate: https://static.rust-lang.org/manifests.txt has not been updated since 2020-02-23

Affects: ChannelManifests source
See also: rust-lang/rust#56971 (comment)

The 'new' dist index which was created because of rust-lang/rust#56971, has not seen an update since 2020-02-23. I missed this while implementing the ChannelManifests source, and it's a bit unfortunate because it offered a solid path to include beta and nightly releases as well.

Undeprecate ChannelManifests source

#9 (comment)

add feature gates per source

Update to rusoto from 0.46.0 to 0.47.0

Rework cache such that the location can be changed

Replace rusoto_s3 with aws-sdk-s3

Rusoto is not actively maintained and has multiple rustsec issues.
aws-sdk has been built by AWS Labs.

Add download on demand option

Currently, we download all release manifests at once (for a certain channel). This process takes a long time, especially for the nightly release channel. We may want to investigate whether we can download only the manifests if we (think we'll) need them.

This saves both bandwidth for users and the Rust project, and could significantly reduce preparation processing time.

`Release` data: add urls for components if available

Support beta and nightly channels

Currently a version always takes a semver version number. Nightly versions are however identified by their date. Before we can add support for Nightly to the RustDist source, we should make sure Nightly versions can be identified