Comments (4)
Alternatively, how about a seash module?
from utf.
I'll summarize discussions with @lukpueh about this issue.
- This is supposed to be a tool for checking the correctness of code running inside a remote sandbox. Thus, we consider mostly the
repy_v2
andseattlelib_v2
tests for the candidate set. - The tool should not attempt to test the nodemanager controlling the test sandbox, nor should it test the interactions performed by
seash
or other components. These are orthogonal goals (see also SeattleTestbed/nodemanager#120). - Obviously, the tool won't run pure-Python unit tests, nor will it run setup/subprocess/shutdown scripts as they (mostly) depend on Python functionality.
#pragma
s and command-line options should be respected as usual. (Reuse functionality fromutf.py
where possible!)
What's mentioned above (having a utf.r2py
that runs test on its own vessel, or perhaps a seash
module for this functionality) has practical ramifications: The usual protocol is that if a test case prints anything, the test is considered to have failed. utf.py
can use plain output redirection to check that; if you run on a sandbox, you can either redirect log
using a security layer above the test case, or inspect the vessel log after the test ran. The former technique is probably what a utf.r2py
would do, the latter is apt for a seash
module.
I think that overall, the seash
module approach has the benefit of being simpler and less error-prone, as there is less code running on the vessel and the number of things that could go wrong is reduced. Potential non-transparency issues in dylink
or encasementlib
are avoided. On the negative side, checking for output might get more complicated due to --execinfo
and such.
from utf.
+1 for the seash
module, thinking DRY this sounds more sensible than porting utf.py
to repy
.
Also I think we could and should take it a step further and decouple the seash
module from seash
, to run said unit tests programmatically. This could be achieved by updating and using the experimental experimentlib.r2py, where @aaaaalbert has already implemented a lot of the required functionality.
The workflow could be something like this:
- Locally run:
python utf.py -f <test.r2py> --keys <keys> --vessel <vessel>
utf.py
executespython repy.py experimentlib.r2py
, e.g. in a subprocessexperimentlib.r2py
runs theseash
module- the module uploads the the test file to the vessel, executes it, downloads the log and feeds it to
utf.py
's checking routine -verify_results
And here is what we have to do:
- Implement upload--run--download-log--reset-log
seash
module - Modify
experimentlib.r2py
to useseash
modules - Modify
utf.py
to accept additional arguments for remote execution and to execute experimentlib.r2py - Strip away output generated by verbosity options, e.g.
--execinfo
- Somehow figure out which part of the output was stdout and which was stderr (this might be difficult)
- Make the output available to
utf.py
'sverify_results
function
from utf.
Thumbs up for thinking this through from the user's perspective too!
I should add that "seash modules" are really implementations of additional seash
commands, not modules in a Python sense. Thus, you can't really use a seash
module in other code; I did however write a few seash
modules to pull in functionality from other code! (And indeed, seash
's overall engineering is a DRY problem in itself, see SeattleTestbed/seash#103).
The good news is that we don't require any seash
module anyway --- experimentlib.r2py
has most of the required functions already, so we could use it for your proposed modifications to utf.py
almost any time.
from utf.
Related Issues (20)
- Some unit tests fail, even after waiting HOT 2
- Unit test timeout fail after r6484 HOT 3
- [Newcomer] Some unit tests fail on Python 2.7.3 HOT 1
- Unit tests fail/take too long HOT 5
- Tests should stop if `subprocess` or `setup` scripts fail HOT 1
- Testing `utf` reveals missing dependencies HOT 1
- Pythonize automate `run_unit_tests.sh` HOT 1
- utf doesn't correctly capture stdout in some cases... HOT 2
- utf.py should returns an exception if some of the tests fail HOT 3
- Write comprehensive `utf` self-tests HOT 1
- Should unit tests be stateless? HOT 3
- Strip Python/Android debug messages from output HOT 1
- Optparse deprecated HOT 1
- `ut_*_subprocess` should signal readiness to `utf.py` HOT 1
- Change subprocess/setup/shutdown sequence HOT 2
- `utf.py` to allow multiple `--file`s HOT 1
- Add option to show stdout, stderr
- Fix order of `#pragma repy` file name and args HOT 2
- Rename Repy-based tests to `.r2py` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from utf.