sccwrp / checker2.0 Goto Github PK
View Code? Open in Web Editor NEWRevised version of the standard data checker, to be used as a template for starting any data checker app
Revised version of the standard data checker, to be used as a template for starting any data checker app
8/12/21 at 11:30am: Dropped a file in the checker, and it displayed an 'internal server error' which then displayed and repeated numerous times, individuals emails connected to application.
When i drag and drop my file over the browser window after fixing errors, the UI still says i had errors, while simultaneously showing the final submit button
Also, in the warnings tab, there are many tabs which show up that say "undefined"
The application is sending emails to all users who have active sessions open, when it should only be sending to current_app.maintainers, according to what i see in the code
This is upon final submit, as well as critical errors
Currently occurring in fish seines
Logic check merges two dataframes with a left join to mark the left dataframe with errors, but the error is flagged on a row of data that does not exist in the excel file (ex: row 12 has a logic error for another table, but row 3052 is marked).
excel tab 'protocol_metadata' was missing colname = ['contact_phone_number']. this should have just failed to match a dataset, but instead fails at running core checks
Excel markup routine was crashing with an error message 'F6.0' is not a valid coordinate
URL for the lookup list has "nese" hard coded, but it should not be like that
The project name should be some kind of variable that gets put in the url somehow, or the "script_root" variable or something
For Nutrients_Lab data, the checker application crashes when 'preparationdate' (date field) is filled with a timestamp. This is a specific case for the general issue across data template submissions for date fields.
We need to implement the scraper so that a user can view lookup lists to choose a correct value. This can also be implemented any way, as long as they have an easy way of viewing the lookup lists
When there are data in the unified tables (metadata), checker gives "index 1 is out of bounds for axis 0 with size 1" error.
The excel markup routine assumes columns A to Z, but doesnt accomodate for more than 26 columns, (AA, AB, AC, etc.)
The Notification Email needs to include a link to their marked excel file. It sends it as an attachment, but the email should also include a download link
The files that went through final submit successfully need to be moved to a separate folder as well for record keeping purposes
There should be the ability to drag and drop their submission file(s) onto the web page, and have it automatically upload th\he files and start the data checking routine
Checker gives error message: Failed to match a dataset.
The "scraper" is the component of the checker that allows the user to view lookup lists, but it is not working for any of the new checkers
If i drag and drop a file, it works and submits and checks it. If i drop a file again without reloading the page, it runs the routine again, as expected, since that is the feeature that the current checker has.
On the third time i drag and drop the file i get a critical, and in the logs i can see something like "SYSCALL EOF DETECTED" error or something like that
Something to do with sqlalchemy or something
The HTML and CSS need to be adjusted so that the thing looks decent, otherwise we can't move the thing into production
The Javascript code needs assertion statements (if they exist) for quick debugging in the future
The Python code can also use more thorough assert statements, and useful print statements for logging purposes
Also if theres a way to do actual unit testing, we should implement it (if it makes sense to do so. It may be like killing a fly with a sledgehammer)
After the data gets "dropped" there should be a loading gif to let the user know that something is happening in the background. There should also be a loading gif for after they hit final submit
This one was happening in the Cerio checker
failure occurs when either (a) contact_phone_number is a colname that appears twice in the table or (b) contact_phone_number and contact_phone_number.1 appears.
match report recognizes a similar dataset and states closest table, but enters core checks and fails during checkDataTypes
Protocol_Metadata tab does not match with tbl_protocol_metadata.
Protocol_Metadata tab needs to be transposed before Core Check.
Currently occurring with fish seines
Need to create a submission tracking table and submission tracking checksum, and appropriate code throughout the app to update those tables with the submission information, whether they do a final submit or not. We also need the "track" view to look at all submissions
specifically occurs for Fish Seines datatype when data for colname='scientificname' is empty in tbl_fish_abundance_data
I put an integer in the sampledate column and it criticaled because it said it cant merge an int column with an object column or something like that
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.