Comments (3)
Based on our conversation today, I'm proposing that
lightbeam
's structured output is enabled with a cli flag--results-file ./results.json
- output format is JSON, like this:
Is this too verbose? i.e., do we need the
{ "timestamp": "2023-05-15 02:04:18Z00", "api_url": "https://edfi3.somedistrict.org/api/", "source_dir": "/path/to/some/dir/", "resources": { "studentSchoolAssociations": { "records_processed": 100, "records_skipped" :10, "records_attempted": 90, "records_success": 75, "records_failed": 15, "failed_statuses": { "409": { "messages": [ { "message": "Invalid StudentReference", "count": 6, "line_numbers": [3, 7, 8, 31, 64, 88] }, { "message": "Invalid SchoolReference", "count": 9, "lines": [5, 9, 12, 13, 26, 27, 29, 30, 56] }, ] }, ... } }, ... } "total_records_processed": 100, "total_records_skipped": 10, "total_records_attempted": 90, "total_records_success": 75, "total_records_failed": 15 }
total_
across all resources?
We also discussed the exit codes lightbeam
should return in different circumstances, to trigger the AirFlow BashOperator task to raise an AirflowSkipException
or AirflowException
(per the Airflow docs). Is it correct that
- if
total_records_skipped
==total_records_processed
, lightbeam should return exit code 99? (so the Airflow task skips) - if
total_records_failed
>0lightbeam
should return exit code 1? (so the Airflow task fails)
One other issue has to do with reporting line numbers: lightbeam
can process either a single JSONL file (source_dir/students.jsonl
) or an entire directory of JSONL files (source_dir/students/*.jsonl
). In the latter case, how would line numbers be determined/interpreted?
Finally, one tiny question, should line numbers be 0-based or 1-based?
from lightbeam.
Some thoughts on this one as well:
- As with Earthbeam logging, let's consider renaming
timestamp
to a more descriptive name that says whether that time is when the run started vs ended. - As with Earthbeam logging, we can use exit code 99 to reflect a skipped run, but if there's a more transparent/generic exit-code to mark no action done, we may want to consider using that instead.
- I think we can narrow down the "totals" to remove
total_records_attempted
. Is there any case where a record that is not skipped is not attempted? - Regardless of whether we 0- or 1-index the failed lines, I think we need to create an accompanying script that takes a file and a list of line numbers and outputs the failures to the console. I don't see a world where manually searching through the files by line number will be easy for the ISS team.
- As for whether to 0- or 1-index the failed lines, that's a good question. Indexing from 1 will make more sense when opening the file manually, but indexing from 0 will make more sense if we consider the header row being line 0. Regardless of what we pick, we need to clearly communicate to the ISS team which one was chosen.
- To deal with multiple files for a given resource, unfortunately we may have to nest the payload further to specify the file alongside line numbers. Something like this:
{
"message": "Invalid StudentReference",
"total_count": 10,
"files": [
{
"file": "{FILENAME1}",
"count": 6,
"line_numbers": [3, 7, 8, 31, 64, 88]
},
{
"file": "{FILENAME2}",
"count": 4,
"line_numbers": [2, 6, 10, 35]
},
]
}
from lightbeam.
Closing this issue as it was implemented in this PR.
from lightbeam.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lightbeam.