difference between running all tests and running one testfile about l3build

[josephwright]

I now understand what's going on: it's rather specific to this particular form of test. In tlb-rollback-003.lvt one finds

\begin{filecontents}{testpkg.sty}
% The current package ...
\DeclareOldRelease[v1]{2014-01-01}{testpkg-2014-01-01}
\DeclareOldRelease[v2]{2018/10-01}{testpkg-2018-10-01}
\DeclareNamedRelease{testpkg-beta}
\DeclareNamedRelease[beta4]{testpkg-beta}
\DeclareCurrentRelease[v3]{2019-07-01}

\ProvidesPackage{testpkg}[2019-07-21 v3.5 test package] % this has a
                                                        % minor change
                                                        % so a
                                                        % different date
\DeclareOption{dvips}{\typeout{ -> option dvips executed}}
\ProcessOptions
\typeout{This is the current testpkg! }
\end{filecontents}

which when run as a single test will happily generate testpkg.sty and give the desired results. However, when run as part of a sequence, we have tlb-rollback-2.lvt, which features

\begin{filecontents}{testpkg.sty}
% The current package ...
\DeclareOldRelease[v1]{2014-01-01}{testpkg-2014-01-01}
\DeclareOldRelease[v2]{2018-10-01}{testpkg-2018-10-01}
\DeclareCurrentRelease[v3]{2019-07-01}

\ProvidesPackage{testpkg}[2019-07-21 v3.5 test package] % this has a
                                                        % minor change
                                                        % so a
                                                        % different date
\DeclareOption{dvips}{\typeout{ -> option dvips executed}}
\ProcessOptions
\typeout{This is the current testpkg! }
\end{filecontents}

ans so 'wins'.

The current 'clean up' logic only moves auxiliary files with names of the form test.xyx to test.engine.xyz, and doesn't try to 'kill' anything else. That avoids having to copy/save/... every file every time for bulk testing, but does mean there can be some 'leakage'. We have two options

• Actively clean up after every test, and save just the outputs (.diff files) - will be slow(ish)
• Add a new variable for 'files to clean up after each test', something like checkcleanfiles?

The latter is trivial to implement and seems faster ...

[FrankMittelbach]

@josephwright ouch, bloody stupid on my part I should have realized that.

We could add checkcleanfiles though the name is perhaps not quite right as it is only the extras you clean (some cleaning happens anway doesn't it)? Are you thinking of something like

checkcleanfiles = { "testpkg*.*" }

or something?

I think it is probably ok to just remember that restriction (and perhaps even document it :-) ) and I change the names to not conflict

[josephwright]

@FrankMittelbach Documenting that everything runs in one place is likely the easiest solution: it's hard to imagine a case where one can't clean up anyway.

[blefloch]

The only case I can imagine of not being able to clean up is if different engines need different filecontents-generated files. By the way, isn't it enough to use `filecontents*`?

[FrankMittelbach]

that's the case I stumbled upon (using filecontents with different content but same filename). and no the star form doesn't help, that isn't overwriting that is only suppressing any commentary put at the top.

[josephwright]

I'm gong to sort this: just need to work out where in the docs to say 'be careful with generated files'.