frewsxcv / cargo-all-features

A Cargo subcommand to build and test all feature flag combinations.

cargo-all-features

Cargo subcommands that build and test all feature flag combinations for a crate.

Install

cargo install cargo-all-features

Usage

The following commands can be run within a Cargo package or at the root of a Cargo workspace.

Build crate with all feature flag combinations:

cargo build-all-features <CARGO BUILD FLAGS>

Check crate with all feature flag combinations:

cargo check-all-features <CARGO CHECK FLAGS>

Test crate with all feature flag combinations:

cargo test-all-features <CARGO TEST FLAGS>

Why?

If you have a crate that utilizes Rust feature flags, it’s common to set up a test matrix in your continuous integration tooling to individually test all feature flags. This setup can be difficult to maintain and easy to forget to update as feature flags come and go. It’s also not exhaustive, as it’s possible enabling combinations of feature flags could result in a compilation error that should be fixed. This utility was built to address these concerns.

Options

You can add the following options to your Cargo.toml file to configure the behavior of cargo-all-features under the heading [package.metadata.cargo-all-features]:

[package.metadata.cargo-all-features]

# Features "foo" and "bar" are incompatible, so skip permutations including them
skip_feature_sets = [
    ["foo", "bar"],
]

# If your crate has a large number of optional dependencies, skip them for speed
skip_optional_dependencies = true

# Add back certain optional dependencies that you want to include in the permutations
extra_features = [
    "log",
]

# Exclude certain features from the build matrix
denylist = ["foo", "bar"]

# Always include these features in combinations.
# These features should not be included in `skip_feature_sets` or `denylist`, they get
# added in later
always_include_features = ["baz"]

# The maximum number of features to try at once. Does not count features from `always_include_features`.
# This is useful for reducing the number of combinations run for a crate with a large amount of features,
# since in most cases a bug just needs a small set of 2-3 features to reproduce.
max_combination_size = 4

# Only include certain features in the build matrix
#(incompatible with `denylist`, `skip_optional_dependencies`, and `extra_features`)
allowlist = ["foo", "bar"]

The project also supports chunking: --n-chunks 3 --chunks 1 will split the crates being tested into three sets (alphabetically, currently), and run the requested command for the first set of crates only. This is useful for splitting up CI jobs or performing disk cleanups since for large workspaces check-all-features and friends can take a very long time and produce a ton of artifacts.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

cargo-all-features's Issues

Cargo udeps integration

Would it be possible to have an integration withcargo udeps? The goal would be to know if you define any dependencies which aren't used and takup time to compile for no reason for possible feature combinations

BTW lovely tool 😄

feature foo needs feature bar

I have a feature foo which required also feature bar to be enabled.
It looks like this can not be tested with cargo-all-feautures currently.

At the moment you can skip all combination of feature

skip_feature_sets = [
    ["foo", "bar"],
]

But I can not do something like this:

skip_feature_sets = [
    ["foo", "!bar"],
]

Alternative a new deps field can be added:

[[deps]]
  feature = "foo"
  needs = ["bar"]

Support for other commands

As I'm reading the documentation, only build-all-features and test-all-features are available. But it might be useful to support other commands too (I'm thinking of clippy but there might be others).

I'm wondering if an API like the following (a bit more generic) would be possible instead.

cargo all-features build
cargo all-features test
cargo all-features clippy

Bug: Tries too many / invalid combinations of features on beta/nightly Rust

I just added this to my CI. But I noticed when it runs on the Rust beta and nightly channels it tries a very odd combination of features. My crate has two optional dependencies, those are the only two features. The crate has env_logger and structopt as optional features.

On stable (only the relevant output):

$ cargo +stable build-all-features 2> /dev/null
    Building crate=udp-over-tcp features=[]
    Building crate=udp-over-tcp features=[env_logger]
    Building crate=udp-over-tcp features=[structopt]
    Building crate=udp-over-tcp features=[env_logger, structopt]

The above is exactly what I expect. But it does the following on beta:

 $ cargo +beta build-all-features 2> /dev/null
    Building crate=udp-over-tcp features=[]
    Building crate=udp-over-tcp features=[env_logger]
    Building crate=udp-over-tcp features=[structopt]
    Building crate=udp-over-tcp features=[env_logger]
    Building crate=udp-over-tcp features=[structopt]
    Building crate=udp-over-tcp features=[env_logger, structopt]
    Building crate=udp-over-tcp features=[env_logger, env_logger]
    Building crate=udp-over-tcp features=[env_logger, structopt]
    Building crate=udp-over-tcp features=[structopt, env_logger]
    Building crate=udp-over-tcp features=[structopt, structopt]
    Building crate=udp-over-tcp features=[env_logger, structopt]
    Building crate=udp-over-tcp features=[env_logger, structopt, env_logger]
    Building crate=udp-over-tcp features=[env_logger, structopt, structopt]
    Building crate=udp-over-tcp features=[env_logger, env_logger, structopt]
    Building crate=udp-over-tcp features=[structopt, env_logger, structopt]
    Building crate=udp-over-tcp features=[env_logger, structopt, env_logger, structopt]

This is not what I would expect it to try with only two features 😅

Clarify config options and transitively included features

Current Config (as I understand it)

• skip_feature_sets: A set of sets. Any feature set from the matrix that is a superset of any members will be dropped.
• skip_optional_dependencies: Doesn't include optional dependencies when building the matrix.
• extra_features: a set of features that will be included in every set in the matrix.
• allowlist: If present, the matrix will only be built using these features. Other features may be transitively included.
• denylist: If present these features will not be used to build the matrix but others may be transitively enabled.

Possible Sources of Confusion

• Perhaps skip_feature_sets can be renamed conflicts or conflicting for clarity.
• Default value for skip_optional_dependencies is false but the example in the README shows it as true and says in a comment that skipping them can improve performance.
• Perhaps allowlist should be renamed as it implies that any feature set that (directly or transitively) includes a feature that's not in the list will be dropped from the matrix. I'm not sure what a good name would be. The best I could come up with is seed because these values are the seeds from which the matrix is build.
• Perhaps denylist should be renamed to the opposite of whatever allowlist gets named for the same reason.
• Features that start with __ are skipped but this is not documented and there is no way to disable it.

Transitively Enabled Features

Currently, transitively enabled features are not taken into consideration when building the matrix. I think that they should be taken into account for a few reasons:

Smaller Matrix

More feature sets can be excluded from the matrix as duplicates, This would help with combinatorial explosion. Currently a crate with 10 features/optional-deps would have 1024 feature sets in its matrix. If some of those features enabled other features we can significantly cut down on testing time.

Better Restrictions

• We can allow users to deny transitively enabled features or optional dependencies. This would allow things like having a __nightly feature that is enabled by any other feature that uses nightly features and denying __nightly.
• If conflicting features get transitively enabled, they can be dropped from the matrix.

Option to use workspace.default-members instead of workspace.members

When used in the root of a workspace, the normal "cargo build" and "cargo test" commands (and similar) take the list of packages to build/test from the workspace.default-members key if it's present, and the workspace.members key if not. (The --workspace option makes them use workspace.members even if workspace.default-members exists.)

This mechanism can come in handy, for instance, if there are crates in the workspace which don't build on normal platforms, but which should still be subject to "cargo doc": these can be listed in "members" but not in "default-members".

However, "cargo build-all-features" and "cargo test-all-features" always operate on workspace.members, with no way to specify the use of default-members instead.

I think ideally build-all-features and test-all-features should copy the behaviour of build and test -- but failing that, at least an option to consult default-members instead would be very useful IMO.

support at-least-one groups of features (nested ideally)

In rustup we have two HTTP backends (curl and reqwest). These are both features, and both can be disabled, but at least one must be enabled.

Similarly, for reqwest we have two features selecting the TLS cert stack to use, and again at least one must be enabled.

We enable sensible defaults via the default-features mechanism.

So we have the following feature sets that are valid:
["curl-backend"]
["reqwest-backend", "reqwest-default-tls"]
["curl-backend", "reqwest-backend", "reqwest-default-tls"]
["reqwest-backend", "reqwest-rustls-tls"]
["curl-backend", "reqwest-backend", "reqwest-rustls-tls"]
["reqwest-backend", "reqwest-default-tls", "reqwest-default-tls"]
["curl-backend", "reqwest-backend", "reqwest-default-tls", "reqwest-default-tls"]

I'd love to be able to express this to cargo-all-features somehow. Right now I've just forced it to always have reqwest-backend and reqwest-rustls-tls on.

Doc test get run despite the `no_run` directive

Sometimes, doc tests with the no_run directive still get executed. This causes problematic examples (which don't work), to be run and fail the test.

Notify the user of the amount of combinations left at the end of each build stage.

This would be a welcomed improvement over the current system as it gives the user a rough idea of how much longer they will need to wait for the command to finish, as for projects with massive amounts of feature flags there will be a lot of different combinations meaning the command feels like it will take another eternity to finish.

Support chunking

This tool tends to fill up target directories really quickly. For large projects like ICU4X, this puts us in danger of hitting CI size limits. Furthermore, it just takes pretty long and it would be nice to be able to make use of parallelism.

A typical technique in CI systems is to support "chunking", where you have something like --chunks 3 --chunk 1, which splits the list of jobs into three chunks, and directs it to run just the jobs for the first one, so you can have the chunks get run on separate runners.

Thoughts?

Add flags to include/exclude specific features

In crates where I'm supporting older Rust versions, it's frequently the case that some of the features will only work on more recent Rust. So in order to use cargo all-features, I would need a way to list which features are actually expected to work for the Rust that I'm currently testing.

Another case is that many crates have features like "unstable" that would only be expected to work when using a nightly toolchain. That would need to be excluded when testing all stable features.

Allow to find out easy way what feature was enabled in transient in crate and what not

I usually do

1. Look into Cargo.lock

cargo check --verbose --job 1 --unit-graph -Z unstable-options 

3. cargo tree

I have the case

1. A has std feature.
2. It uses B having std, sure when A has std.
3. B uses C. C has std too. But be forgot to enable std on C.
4. So when I compile A, it fails, because we are in std feature, but C did not enabled it.

In some codebase it hard to debug - when there are dozens of crates in several layers with several adhoc features (I work at such). I use above tooling.

Would be awesome to have some oneliner to see all features in crates, why they are enabled or not.

argument processing incorrect in cargo subcommand mode

The code that looks for --help in env.args()[1] is in lib.rs in the run function is faulty: when run as cargo build-all-features --help the args vector will look like so:

$ cargo build-all-features --help
Args: Args { inner: ["...\\.cargo\\bin\\cargo-build-all-features.exe", "build-all-features", "--help"] }    Building crate=cargo-all-features features=[]

That is, arg0 is the binary, arg1 is the subcommand name, arg2 is the first argument.

Recommended way to test all crates in a workspace

It seems like cargo test-all-features --workspace tests the whole workspace for any combination of features.

I'd like to test each crate in the workspace for any combination of its own features.

---

Example where it's different:

• crate a with no features
• crate b with feature a (which adds an optional dependency on a)

Manual execution (the behavior I want): cd a; cargo test-all-features; cd ../b; cargo test-all-features

• Tests a with no features
• Tests b with no features
• Tests b with feature a

cargo test-all-features --workspace:

• Selects a with no features
  • Tests a with no features
  • Tests b with no features
• Selects b with no features
  • Tests a with no features
  • Tests b with no features
• Selects b with feature a
  • Tests a with no features
  • Tests b with feature a

As you can see, there are a lot of unneeded tests. With a great number of crates (30+), many kinds of tests (unittest, doc-tests, ...) and multiple features (1--10), the command takes too long to be useful.

miri support

Hi. I was about to write this utility and then I found yours :)

It would be nice to have miri test support. It would be exactly like test-all-features except it would run cargo miri test instead of cargo test. miri-test-all-features seems like a good name.

Thoughts?

wasm-pack support

For rust wasm projects we use wasm-pack to build and run unit tests.

Common commands including:

wasm-pack build
wasm-pack test

Is it possible to run wasm-pack commands with cargo-all-features?

Noisy output

The output of any given run is quite noisy. I propose only showing the output of failing runs and show a summary report instead. Perhaps showing the currently running set with a loading symbol if it's running in a smart terminal.

It seems to work on optional dependencies, not on features - is this intended?

How to reproduce (on MacOS, Rustc version 1.43.1, cargo-all-features version 1.2.0)

git clone https://github.com/crates-io/prodash
cd prodash
cargo build-all-features

What's expected is that it only activates features, along with their dependencies:

It seems to activate optional dependencies instead, along with all their permutations.

Add flags to print dry run of commands to support concurrent runs

Could there be a way to just print out the commands instead of executing them?

For our continuous integration setup in Github, we're noticing that the wall clock times of our simple build & test jobs have gone up in our pending switch to cargo-all-features. I suspect that it's due to all of the different combinations of feature sets being run serially.

In our CI, we have the ability to run jobs concurrently in order to save on wall clock time. In fact, our build & test job is set up that way using a "job matrix" in which we indicate which variables take on which enumerated sets of values, and the CI system creates the combinatorial number of jobs according to the Cartesian product of the enum sets. We currently run on 3 different OSes, and our enumerated set of feature sets is ["", "--all-features"], where the empty string actually refers to the default set.

So if there was a way to obtain the series of commands used to test each unique combination of features, then it would be possible to cut down on our wall clock time by a significant factor.

Does cargo-all-features work well with specifying parallel jobs using the -j option in cargo? Our CI system uses 2-core CPUs, which is something.

Independent of that, however, is the fact that the max distributed job concurrency allowed by the CI is 5 jobs. So having the sequence of commands used per unique combination of features could enable that improvement, too.

@sffc

Prefer the json crate to serde_json

By using procedural macros, this crate will became a no go for CI, since it will take forever to compile.

Chunk by feature set rather than crates

Crates dependencies tend to have an upper bound: as many as --all-features depends on.
However, combinations grow quadratically with the number of features to test.
Thus, it's much more valuable in terms of CI running times to split the total combinations into a given number of chunks so those can be run in parallel.

Specifying `-p` in a workspace still enumerates featuers from packages not specified on command line

I have a workspace with different crates with their own features. If I specify -p to select one or more packages with test-all-features it appears all features from unspecified packages are being enumerated and tested against the specified packages, causing "no such feature" errors to arise.

Add ability to allowlist features to be mixed in Cargo.toml

My crate has a lot of features and the exponential growth is becoming a problem. I don't actually care about the full matrix, I'd like a subset of them to be cargo-all-features-tested. Would it be possible to get a metadata key for this?

Optional dependencies that do not create a feature of the same name

Since Rust 1.60, it is possible to add an optional dependency without adding a feature of the same name (documentation). However, cargo-all-features expects these features to exist.

Reproduction:

# normal package declaration

[dependencies]
log = { version = "0.4", optional = true }

[features]
logging = ["dep:log"]

$ cargo test-all-features
     Testing crate=tester features=[]
# ...
     Testing crate=tester features=[logging]
error: Package `tester v0.1.0 (...)` does not have feature `logging`. It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.

Unexpected manifest-path behavior in workspace

Suppose I have a workspace with crates foo and bar:

- Cargo.toml
- foo
| - Cargo.toml
- bar
| - Cargo.toml

If I want to run cargo test-all-features on crate foo, I expect to be able to run this command from the root of the workspace:

cargo test-all-features --manifest-path foo/Cargo.toml

and have it run tests only for foo. Instead, I get output like this:

     Testing crate=bar features=[]
error: manifest path `foo/Cargo.toml` does not exist

As far as I can tell, test-all-features is running cargo test for all crates in the workspace and passing all arguments, including --manifest-path, to cargo test.

I need this to consider --manifest-path beforehand, though, as I am using actions-rs/cargo to test individual crates depending on which files got changed. actions-rs/cargo does not support any sort of working-directory input, so to work on a specific crate I have to use --manifest-path.

Is this a use case you'd be willing to support?

More combination modes

My crate currently has 10 features, it will then explode to 2^10 = 1024 combinations.
It'd be useful to be able to test some meaningful subsets of all the possible combinations.

I'd propose some modes:

one-by-one: enable each feature one-by-one
all-features: all features enabled
pseudo-random: pseudo-randomly pick N combinations
random: true-randomly pick N combinations
mix-and-match: divide features into M groups, and randomly pick R features from each group

For example, if we divide 10 features into 2 groups of 5, and picking up to 3 from each group, then:

((5 choose 3) + (5 choose 2) + (5 choose 1)) ^ 2 = 625 combinations.

After that, we can still randomly pick N from these possible combinations.

It would be a more reasonable set of combinations than picking purely randomly, as I observe that crates with many features usually divide their feature flags by function categories. As an analogy people can choose different wheels and engines when building a car.

What do you think about this?

Clarify behaviour - feature set vs feature list

Most places use a feature list but refer to it as a set. (When I was refactoring, I kept the list behaviour and called it FeatureList.) However, the deny list uses a set. My intuition is that using a set is the correct behaviour as there's never any reason to include the same feature twice. Is this correct?

build-all-features uses all disk space in github actions

You can see in this run here : https://github.com/rust-lang/rustup/actions/runs/4909419719/jobs/8765812125?pr=3340 - cargo build-all-features used all the available disk space building rustup.

Different feature combinations turn on different feature sets for dependencies like tokio, and this results in many built-once, reused-never dependencies. Which apparently when repeated for enough features, used up all the disk space.

I realise it would be slower, but having an option to clean between builds would be super valuable for us at least

`allowlist = []` doesn't do anything

I want to disable cargo-all-features for a single crate in my workspace that wraps another crate and forwards all the features. Setting the allowlist to empty doesn't do anything.

skip_optional_dependencies is broken with 1.60

With 1.60 and dep: syntax, cargo metadata now always reports a feature of the form "foo": ["dep:foo"] for optional dependencies. This leads to all optional dep combos being tested even with `skip_optional_dependencies, which bloats runtime significantly.

Add flag to opt-out of testing optional dependencies

Follow up from #2

Maybe you would like to actually put the name and date on the Apache License

And maybe MIT if you're not with georust