Comments (5)
Yes, i agree, this should be high priority on our roadmap. And good point about sometimes having known issues.
The way I've been thinking about it is that there are a few different layers to Orca_test:
- The spec classes (
OrcaSpec()
,TableSpec()
, etc.) - The testing interface (
assert_orca_spec(o_spec)
, etc.) - The low-level functions that check individual spec characteristics (most of the actual code)
What you're suggesting would be a new mode in the middle layer, so that we run a diagnosis report for a spec instead of just asserting it. Hopefully this will be easy to implement through some simple changes to the existing middle-layer functions.
This is a bit speculative, but potentially we could have an Orca_test "mode" setting, and a user could choose whether to run a simulation in "strict" mode or to just log the Orca_test warnings. Or, a user could have multiple sets of specs with different strictnesses. This would be a stretch to implement right now, but it's something to think about as we make architecture decisions.
from orca_test.
Yes, I think it can be a mid layer. It would call the same lower level calls, but each in a try
block that logs errors and continues. If any throw an error, than the mid layer will throw at the end.
from orca_test.
I got started experimenting on my fork. DIFF
Thoughts? Comments?
Catching each error, looses a lot of the traceback info.
P.S. sorry for the white space diff noise, pycharm atom fixed it.
from orca_test.
Excellent! I like this as a general behavior, that the functions like assert_column_spec()
don't raise an exception until they've checked every component of the spec.
Some thoughts:
- We should probably either log the intermediate error messages (how-to) or pass them along with the final exception, instead of just printing them. Not sure which is better. I'm imagining a use case where higher-level code is compiling a report or providing info to a GUI.
- I agree that losing the traceback info is a shame. Not sure what to do about that.
- It probably still makes sense to have another mode where the tests don't raise an exception at all, just log the messages. The easiest implementation might be to have
assert_orca_spec()
vstest_orca_spec()
where the latter just runs the former and catches the final exception.
from orca_test.
If we are logging log.exception(e). Which also gets the traceback. I don't know how to give the details to the caller.
from orca_test.
Related Issues (18)
- Ultra specific testing for debug purposes HOT 4
- YAML syntax and additional assertions on value consistency? HOT 3
- Overbroad Except Clauses HOT 4
- Split into multiple files HOT 1
- Possible bug in missing value assertions HOT 2
- Write unit tests HOT 3
- Add Python 3 cross-compatibility HOT 1
- Standardize docstrings and put together Sphinx documentation
- Foreign_key test report additional/missing values HOT 1
- Set defaults in test specs HOT 3
- Error messages should put table name in a consistent spot
- What can we learn from engarde HOT 1
- Slow performance & redundant tests HOT 10
- categorical columns
- Raise an error when an undefined key is included in a spec
- More informative error message for max_portion_missing
- Warnings or reports rather than assertions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from orca_test.