Comments (3)
I think you should remove state before running a test, not after. In
other words, put the system state in the correct state and then run. If
you clean up after a failure, then you may delete useful debugging
information.
On Fri, Oct 2, 2015 at 7:23 AM, aaaaalbert [email protected] wrote:
We don't have a policy for whether unit tests must "clean up after
themselves" or not, i.e. may leave traces of the state / fact of their
exceution or not. Some tests clean up
https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_subprocess.py#L48-L67,
some don't despite causing significant artifacts that changes the
behavior of later re-runs through the same test
https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_allmodules.py#L21-L23
.
Pros of requiring cleanup
- If unit tests fail, this often means you will change code in
response, and re-run the tests. Stateless tests speed this up a lot --- no
re-cloning, re-building, etc.; Running the test again is like starting over
anew.- Forces you to think harder before/while you implement. You will see
problems like this in production code for sure, so why not train yourself
on writing tests beforehand?Cons of requiring cleanup
- It's not always obvious that unit tests have side effects, and what
these will be.- Cleaning things up correctly is complex, and probably requires lots
of duplicate code across tests.
A middle ground could be to somehow mark tests that require a re-build
after they have run. Then at least it's clear that problems may arise.
Reaching out to @JustinCappos https://github.com/JustinCappos and
@vladimir-v-diaz https://github.com/vladimir-v-diaz in particular ---
What do you think?—
Reply to this email directly or view it on GitHub
#66.
from utf.
Sure, makes sense.
Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense of RFC 2119? 😉
(Currently, the decision is left to the implementor.)
from utf.
In the case of success I think it would be a SHOULD.
However, there are also pre and post scripts for certain test modules
(groups of tests) and the complexity of having per-test cleanups on success
while leaving state on failure, is a mess.
The tests are here to make our lives easier. If being very rigid about our
test behavior causes a lot of work, that is not helpful. However, needing
to debug some specific test in a different way may cause us a lot of work
over time. I would at a minimum expect any "outlier" tests to include
copious comments explaining how they work and why.
(We should already have this for some of the resource tests that may crash
the system if the repy VM does not handle them correctly.)
On Fri, Oct 2, 2015 at 12:17 PM, aaaaalbert [email protected]
wrote:
Sure, makes sense.
Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense
of RFC 2119? [image: 😉](Currently, the decision is left to the implementor.)
—
Reply to this email directly or view it on GitHub
#66 (comment).
from utf.
Related Issues (20)
- Some unit tests fail, even after waiting HOT 2
- Unit test timeout fail after r6484 HOT 3
- [Newcomer] Some unit tests fail on Python 2.7.3 HOT 1
- Unit tests fail/take too long HOT 5
- Tests should stop if `subprocess` or `setup` scripts fail HOT 1
- Testing `utf` reveals missing dependencies HOT 1
- Pythonize automate `run_unit_tests.sh` HOT 1
- utf doesn't correctly capture stdout in some cases... HOT 2
- utf.py should returns an exception if some of the tests fail HOT 3
- Write comprehensive `utf` self-tests HOT 1
- Create `utf.r2py` HOT 4
- Strip Python/Android debug messages from output HOT 1
- Optparse deprecated HOT 1
- `ut_*_subprocess` should signal readiness to `utf.py` HOT 1
- Change subprocess/setup/shutdown sequence HOT 2
- `utf.py` to allow multiple `--file`s HOT 1
- Add option to show stdout, stderr
- Fix order of `#pragma repy` file name and args HOT 2
- Rename Repy-based tests to `.r2py` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from utf.