How to do conditional Skip? about testslide

[fornellas]

https://testslide.readthedocs.io/en/1.5.0/testslide_dsl/skip_and_focus/index.html#skip

xexample is the way to go for unconditional skip. However it is just regular Python, this also works:

if should_have_example:
  @context.example
  def whatever(self):

Real world example https://github.com/facebookincubator/TestSlide/blob/master/tests/mock_callable_testslide.py#L179.

I suppose we could have some syntax sugar for that like:

@context.example(skip=should_skip)

But it seems no to be strictly needed.

What do you think?

[xush6528]

The daily use case for my work is that we have some test examples that only runs on GPU CUDA devices.

But CI could pick building and testing our project in either environment, ones with CPU+GPU or ones that has CPU only.

We used @unittest.skipUnless(cuda.is_available()) a lot.

In TestSlide, we will have to write this to circumvent

If not cuda.is_available():
    return

And, more importantly, the test runner will not remind me that the test is skipped.

[xush6528]

@fornellas Thanks, I will try this way. Conditionally declare contexts and examples.

[fornellas]

Cool. I'm closing this then, as it seems you have a way for conditionally skip. Feel free to reopen if you feel like it deserves some syntax sugar.

[xush6528]

After trying it out, I feel it's still needed.

Because it would be great to have a "SKIP" item in test runner results, just like @context.xexample does.

In short, skipped tests are worth being noticed!

For example, I have a lot of tests that must be skipped when running on a machine without GPU avaiable.
With conditional test example declaration as you suggested above would silently hide these examples.

However, for a new developer onboarding to my tests, he/she might be unware of that, when running on a CPU only machine, there are many GPU tests being explicitly skipped.

If these skipped tests could be noticed by new developmers, they might find better test environment.

[fornellas]

NP. So, how do you think the syntax should go?

@context.example(skip=should_skip)

Is it ok like this?

[xush6528]

Thanks for opening it.

It looks great.

One thing worth noticing is that, after searching in my company's repo, I noticed that @unittest.skipUnless(condition) has 2X usage than @unittest.skip(condition).

That means with this simple syntax, we need to expect a lot of

@context.example(skip=not should_run_example)