frewsxcv / cargo-all-features Goto Github PK
View Code? Open in Web Editor NEWA Cargo subcommand to build and test all feature flag combinations.
License: Apache License 2.0
A Cargo subcommand to build and test all feature flag combinations.
License: Apache License 2.0
This tool tends to fill up target directories really quickly. For large projects like ICU4X, this puts us in danger of hitting CI size limits. Furthermore, it just takes pretty long and it would be nice to be able to make use of parallelism.
A typical technique in CI systems is to support "chunking", where you have something like --chunks 3 --chunk 1
, which splits the list of jobs into three chunks, and directs it to run just the jobs for the first one, so you can have the chunks get run on separate runners.
Thoughts?
Could there be a way to just print out the commands instead of executing them?
For our continuous integration setup in Github, we're noticing that the wall clock times of our simple build & test jobs have gone up in our pending switch to cargo-all-features
. I suspect that it's due to all of the different combinations of feature sets being run serially.
In our CI, we have the ability to run jobs concurrently in order to save on wall clock time. In fact, our build & test job is set up that way using a "job matrix" in which we indicate which variables take on which enumerated sets of values, and the CI system creates the combinatorial number of jobs according to the Cartesian product of the enum sets. We currently run on 3 different OSes, and our enumerated set of feature sets is ["", "--all-features"]
, where the empty string actually refers to the default set.
So if there was a way to obtain the series of commands used to test each unique combination of features, then it would be possible to cut down on our wall clock time by a significant factor.
Does cargo-all-features
work well with specifying parallel jobs using the -j
option in cargo? Our CI system uses 2-core CPUs, which is something.
Independent of that, however, is the fact that the max distributed job concurrency allowed by the CI is 5 jobs. So having the sequence of commands used per unique combination of features could enable that improvement, too.
I usually do
Look into Cargo.lock
cargo check --verbose --job 1 --unit-graph -Z unstable-options
I have the case
std
feature.std
, sure when A has std
.std
too. But be forgot to enable std
on C.std
feature, but C did not enabled it.In some codebase it hard to debug - when there are dozens of crates in several layers with several adhoc features (I work at such). I use above tooling.
Would be awesome to have some oneliner to see all features in crates, why they are enabled or not.
In rustup we have two HTTP backends (curl and reqwest). These are both features, and both can be disabled, but at least one must be enabled.
Similarly, for reqwest we have two features selecting the TLS cert stack to use, and again at least one must be enabled.
We enable sensible defaults via the default-features mechanism.
So we have the following feature sets that are valid:
["curl-backend"]
["reqwest-backend", "reqwest-default-tls"]
["curl-backend", "reqwest-backend", "reqwest-default-tls"]
["reqwest-backend", "reqwest-rustls-tls"]
["curl-backend", "reqwest-backend", "reqwest-rustls-tls"]
["reqwest-backend", "reqwest-default-tls", "reqwest-default-tls"]
["curl-backend", "reqwest-backend", "reqwest-default-tls", "reqwest-default-tls"]
I'd love to be able to express this to cargo-all-features somehow. Right now I've just forced it to always have reqwest-backend and reqwest-rustls-tls on.
My crate currently has 10 features, it will then explode to 2^10 = 1024 combinations.
It'd be useful to be able to test some meaningful subsets of all the possible combinations.
I'd propose some modes:
one-by-one
: enable each feature one-by-one
all-features
: all features enabled
pseudo-random
: pseudo-randomly pick N combinations
random
: true-randomly pick N combinations
mix-and-match
: divide features into M groups, and randomly pick R features from each group
For example, if we divide 10 features into 2 groups of 5, and picking up to 3 from each group, then:
((5 choose 3) + (5 choose 2) + (5 choose 1)) ^ 2 = 625 combinations.
After that, we can still randomly pick N from these possible combinations.
It would be a more reasonable set of combinations than picking purely randomly, as I observe that crates with many features usually divide their feature flags by function categories. As an analogy people can choose different wheels and engines when building a car.
What do you think about this?
Suppose I have a workspace with crates foo
and bar
:
- Cargo.toml
- foo
| - Cargo.toml
- bar
| - Cargo.toml
If I want to run cargo test-all-features
on crate foo
, I expect to be able to run this command from the root of the workspace:
cargo test-all-features --manifest-path foo/Cargo.toml
and have it run tests only for foo
. Instead, I get output like this:
Testing crate=bar features=[]
error: manifest path `foo/Cargo.toml` does not exist
As far as I can tell, test-all-features
is running cargo test
for all crates in the workspace and passing all arguments, including --manifest-path
, to cargo test
.
I need this to consider --manifest-path
beforehand, though, as I am using actions-rs/cargo to test individual crates depending on which files got changed. actions-rs/cargo
does not support any sort of working-directory
input, so to work on a specific crate I have to use --manifest-path
.
Is this a use case you'd be willing to support?
This would be a welcomed improvement over the current system as it gives the user a rough idea of how much longer they will need to wait for the command to finish, as for projects with massive amounts of feature flags there will be a lot of different combinations meaning the command feels like it will take another eternity to finish.
The output of any given run is quite noisy. I propose only showing the output of failing runs and show a summary report instead. Perhaps showing the currently running set with a loading symbol if it's running in a smart terminal.
Follow up from #2
By using procedural macros, this crate will became a no go for CI, since it will take forever to compile.
In crates where I'm supporting older Rust versions, it's frequently the case that some of the features will only work on more recent Rust. So in order to use cargo all-features
, I would need a way to list which features are actually expected to work for the Rust that I'm currently testing.
Another case is that many crates have features like "unstable" that would only be expected to work when using a nightly toolchain. That would need to be excluded when testing all stable features.
Would it be possible to have an integration withcargo udeps? The goal would be to know if you define any dependencies which aren't used and takup time to compile for no reason for possible feature combinations
BTW lovely tool ๐
Sometimes, doc tests with the no_run
directive still get executed. This causes problematic examples (which don't work), to be run and fail the test.
Since Rust 1.60, it is possible to add an optional dependency without adding a feature of the same name (documentation). However, cargo-all-features
expects these features to exist.
Reproduction:
# normal package declaration
[dependencies]
log = { version = "0.4", optional = true }
[features]
logging = ["dep:log"]
$ cargo test-all-features
Testing crate=tester features=[]
# ...
Testing crate=tester features=[logging]
error: Package `tester v0.1.0 (...)` does not have feature `logging`. It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.
Crates dependencies tend to have an upper bound: as many as --all-features
depends on.
However, combinations grow quadratically with the number of features to test.
Thus, it's much more valuable in terms of CI running times to split the total combinations into a given number of chunks so those can be run in parallel.
I have a workspace with different crates with their own features. If I specify -p
to select one or more packages with test-all-features
it appears all features from unspecified packages are being enumerated and tested against the specified packages, causing "no such feature" errors to arise.
It seems like cargo test-all-features --workspace
tests the whole workspace for any combination of features.
I'd like to test each crate in the workspace for any combination of its own features.
Example where it's different:
a
with no featuresb
with feature a
(which adds an optional dependency on a
)Manual execution (the behavior I want): cd a; cargo test-all-features; cd ../b; cargo test-all-features
a
with no featuresb
with no featuresb
with feature a
cargo test-all-features --workspace
:
a
with no features
a
with no featuresb
with no featuresb
with no features
a
with no featuresb
with no featuresb
with feature a
a
with no featuresb
with feature a
As you can see, there are a lot of unneeded tests. With a great number of crates (30+), many kinds of tests (unittest, doc-tests, ...) and multiple features (1--10), the command takes too long to be useful.
You can see in this run here : https://github.com/rust-lang/rustup/actions/runs/4909419719/jobs/8765812125?pr=3340 - cargo build-all-features used all the available disk space building rustup.
Different feature combinations turn on different feature sets for dependencies like tokio, and this results in many built-once, reused-never dependencies. Which apparently when repeated for enough features, used up all the disk space.
I realise it would be slower, but having an option to clean between builds would be super valuable for us at least
My crate has a lot of features and the exponential growth is becoming a problem. I don't actually care about the full matrix, I'd like a subset of them to be cargo-all-features-tested. Would it be possible to get a metadata key for this?
skip_feature_sets
: A set of sets. Any feature set from the matrix that is a superset of any members will be dropped.skip_optional_dependencies
: Doesn't include optional dependencies when building the matrix.extra_features
: a set of features that will be included in every set in the matrix.allowlist
: If present, the matrix will only be built using these features. Other features may be transitively included.denylist
: If present these features will not be used to build the matrix but others may be transitively enabled.skip_feature_sets
can be renamed conflicts
or conflicting
for clarity.skip_optional_dependencies
is false
but the example in the README shows it as true
and says in a comment that skipping them can improve performance.allowlist
should be renamed as it implies that any feature set that (directly or transitively) includes a feature that's not in the list will be dropped from the matrix. I'm not sure what a good name would be. The best I could come up with is seed
because these values are the seeds from which the matrix is build.denylist
should be renamed to the opposite of whatever allowlist
gets named for the same reason.__
are skipped but this is not documented and there is no way to disable it.Currently, transitively enabled features are not taken into consideration when building the matrix. I think that they should be taken into account for a few reasons:
More feature sets can be excluded from the matrix as duplicates, This would help with combinatorial explosion. Currently a crate with 10 features/optional-deps would have 1024 feature sets in its matrix. If some of those features enabled other features we can significantly cut down on testing time.
__nightly
feature that is enabled by any other feature that uses nightly features and denying __nightly
.And maybe MIT if you're not with georust
With 1.60 and dep:
syntax, cargo metadata
now always reports a feature of the form "foo": ["dep:foo"]
for optional dependencies. This leads to all optional dep combos being tested even with `skip_optional_dependencies, which bloats runtime significantly.
Most places use a feature list but refer to it as a set. (When I was refactoring, I kept the list behaviour and called it FeatureList
.) However, the deny list uses a set. My intuition is that using a set is the correct behaviour as there's never any reason to include the same feature twice. Is this correct?
I have a feature foo
which required also feature bar
to be enabled.
It looks like this can not be tested with cargo-all-feautures currently.
At the moment you can skip all combination of feature
skip_feature_sets = [
["foo", "bar"],
]
But I can not do something like this:
skip_feature_sets = [
["foo", "!bar"],
]
Alternative a new deps
field can be added:
[[deps]]
feature = "foo"
needs = ["bar"]
I want to disable cargo-all-features
for a single crate in my workspace that wraps another crate and forwards all the features. Setting the allowlist to empty doesn't do anything.
I just added this to my CI. But I noticed when it runs on the Rust beta and nightly channels it tries a very odd combination of features. My crate has two optional dependencies, those are the only two features. The crate has env_logger
and structopt
as optional features.
On stable (only the relevant output):
$ cargo +stable build-all-features 2> /dev/null
Building crate=udp-over-tcp features=[]
Building crate=udp-over-tcp features=[env_logger]
Building crate=udp-over-tcp features=[structopt]
Building crate=udp-over-tcp features=[env_logger, structopt]
The above is exactly what I expect. But it does the following on beta:
$ cargo +beta build-all-features 2> /dev/null
Building crate=udp-over-tcp features=[]
Building crate=udp-over-tcp features=[env_logger]
Building crate=udp-over-tcp features=[structopt]
Building crate=udp-over-tcp features=[env_logger]
Building crate=udp-over-tcp features=[structopt]
Building crate=udp-over-tcp features=[env_logger, structopt]
Building crate=udp-over-tcp features=[env_logger, env_logger]
Building crate=udp-over-tcp features=[env_logger, structopt]
Building crate=udp-over-tcp features=[structopt, env_logger]
Building crate=udp-over-tcp features=[structopt, structopt]
Building crate=udp-over-tcp features=[env_logger, structopt]
Building crate=udp-over-tcp features=[env_logger, structopt, env_logger]
Building crate=udp-over-tcp features=[env_logger, structopt, structopt]
Building crate=udp-over-tcp features=[env_logger, env_logger, structopt]
Building crate=udp-over-tcp features=[structopt, env_logger, structopt]
Building crate=udp-over-tcp features=[env_logger, structopt, env_logger, structopt]
This is not what I would expect it to try with only two features ๐
How to reproduce (on MacOS, Rustc version 1.43.1, cargo-all-features version 1.2.0)
git clone https://github.com/crates-io/prodash
cd prodash
cargo build-all-features
What's expected is that it only activates features, along with their dependencies:
It seems to activate optional dependencies instead, along with all their permutations.
Hi. I was about to write this utility and then I found yours :)
It would be nice to have miri test
support. It would be exactly like test-all-features
except it would run cargo miri test
instead of cargo test
. miri-test-all-features
seems like a good name.
Thoughts?
When used in the root of a workspace, the normal "cargo build" and "cargo test" commands (and similar) take the list of packages to build/test from the workspace.default-members key if it's present, and the workspace.members key if not. (The --workspace
option makes them use workspace.members even if workspace.default-members exists.)
This mechanism can come in handy, for instance, if there are crates in the workspace which don't build on normal platforms, but which should still be subject to "cargo doc": these can be listed in "members" but not in "default-members".
However, "cargo build-all-features" and "cargo test-all-features" always operate on workspace.members, with no way to specify the use of default-members instead.
I think ideally build-all-features and test-all-features should copy the behaviour of build and test -- but failing that, at least an option to consult default-members instead would be very useful IMO.
The code that looks for --help
in env.args()[1]
is in lib.rs in the run
function is faulty: when run as cargo build-all-features --help
the args vector will look like so:
$ cargo build-all-features --help
Args: Args { inner: ["...\\.cargo\\bin\\cargo-build-all-features.exe", "build-all-features", "--help"] } Building crate=cargo-all-features features=[]
That is, arg0 is the binary, arg1 is the subcommand name, arg2 is the first argument.
As I'm reading the documentation, only build-all-features
and test-all-features
are available. But it might be useful to support other commands too (I'm thinking of clippy
but there might be others).
I'm wondering if an API like the following (a bit more generic) would be possible instead.
cargo all-features build
cargo all-features test
cargo all-features clippy
For rust wasm projects we use wasm-pack to build and run unit tests.
Common commands including:
wasm-pack build
wasm-pack test
Is it possible to run wasm-pack
commands with cargo-all-features
?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.