Test command about cask

[doublep]

Just discovered that there is a development tool for Elisp packages, and yet it lacks the functionality I need the most...

In case anyone ever starts working on it, some feature requests:

• Let developer choose which tests to run (of course, all by default). E.g. in ERT this would be selector. (I presume [args] is meant for that...)

• Let developer choose which test files to load (again, all by default). So I would be able to narrow down tests to run using either selectors (which operate on all defined testcases), or files (which results in never even defining some testcases), or both.

Usecase: lets say I have a thousand of fairly slow tests and want to quickly run five out of them, as I'm fixing a particular bug.

[Fuco1]

There are tools for that, for example ert-runner which you can then execute with cask as

cask exec ert-runner

or buttercup which by default exports a binary

cask exec buttercup

Both of these accept -p flag as pattern for test cases to run.

[phillord]

I use cask emacs --load "assess-discover" --funcall assess-discover-run-and-exit-batch.

But the OP is correct. You have to know about these things and they don't appear in cask help. Adding ert-runner, buttercup and assess as subcommands would probably do 90% of the job, with cask test making a heuristic guess between the three.

I'd be happy to add, assuming it seems a good addition (and everyone is patient!)

[Fuco1]

@phillord No problem with a pull request, but I wonder if it's really that useful. After 6 years this is the first time someone is asking for the feature/bumping the thread.

Also keep in mind this will add maintenance overhead in case ert-runner, buttercup or assess changes in the future, cask will automatically stop (potentially) working correctly.

[doublep]

cask exec ert-runner

OK, after some tries I guessed that it runs files test/*-test.el, not everything under test. I couldn't find this in documentation, cask --verbose exec ert-runner, --debug or --dev print nothing at all. I have no idea what those options do, apparently not what one would expect.

You see no intermediate output with it (what if you run 1000 tests, and the third already fails?), and if you abort with C-c, you see no output at all.

Also, there is that bug about ert--print-backtrace on Emacs 27 that no-one is fixing.

buttercup

Maybe it's great, but I don't plan to switch from ERT currently.

cask emacs --load "assess-discover" --funcall assess-discover-run-and-exit-batch

I couldn't make it work, e.g. it opens a separate Emacs window, which then complains that assess-discover-run-and-exit-batch is only for batch mode. But:

cask emacs --load "assess-discover" --funcall assess-discover-run-and-exit-batch

does it not cry "I want cask test instead of a 80-character command" to you?

I wonder if it's really that useful

And I wonder what other people use Cask for. The only reason I looked for something like this is running regression tests from command line with simple syntax to narrow down to exact tests I need. Packaging? I just push to Github and let Melpa build packages. Developing several cross-dependent packages (cask link ...)? I have a homegrown solution of loading local-environment.el, which could e.g. modify load-path (and do whatever per-machine setup steps I want in general, so e.g. on Travis it installs the packages from Melpa). Or I could just add symbolic links.

Frankly, so far it looks like I'll keep using ugly, but at least working shell scripts I have.

[Fuco1]

The output is line-buffered so you need to use different reporter with ert-runner.

This is what I use

cask exec ert-runner --reporter ert+duration -p 'kill-symbol' 

You can look at other project's travis setups, that's what people use it for. And to have the same env available locally.

[phillord]

@doublep "I couldn't make it work, e.g. it opens a separate Emacs window, which then complains that assess-discover-run-and-exit-batch is only for batch mode."

Oh dear, my apologies. I shortened the command because it was just an example.

cask emacs --batch -q \
    --directory=. \
    --load "assess-discover" \
        --funcall assess-discover-run-and-exit-batch

@rejeep "but I wonder if it's really that useful."

It's a good question, and you have a point. But, I do remember when trying out cask that I was expecting a test option. Nowadays I work around the somewhat long cask command line with a make file. You are correct, that it adds to the maintainer load. The risks are there but I think are low, given the overlap in authorship between these four projects.

I'll have a go at the PR (it will take time!) and keep it as simple as possible, because as you say, why create work for the future.

[conao3]

There are many test frameworks, cask exec <framework> is better than cask test, I think.
Close for now, but please comment if anyone has a productive idea.