datacommonsorg / tools Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
There was a breaking change to the return type of the Stats REST APIs that caused the server to crash with TypeError exception when trying to retrieve "data" from "None".
When making a request using the stats data commons REST APIs, if there was no data to return for a given dcid, it would drop the dcid from the response. Now, instead, it will return dcid->None, which caused issues when trying to read the data value.
Example of return type for dcid without a value:
Exception Message:
data = response[dcid]['data']
TypeError: 'NoneType' object is not subscriptable
The tmcf+csv parser does not currently check for error cases while parsing. Error checking should be included in this step to give the user a precise location of where/why the parsing error is occurring so they can fix their file accordingly.
The parsing errors table displayed to the user should indicate which line of the TMCF and which row in the CSV file is causing the error.
To do this, ParseTmcf class must check for returned the logged errors. back-end/server-api.js will need to be modified to capture the errors found by ParseTmcf. If capturing these errors causes a change to the structure of App's state parsingErrs, then App.jsx will need to be modified. Finally, ParsingErrsTable.jsx component will need to be modified in order to display the CSV row data.
This issue was automatically created by Allstar.
Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code
Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.
Remediation Steps
To remediate, remove the generated executable artifacts from the repository.
Artifacts Found
Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.
Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar
This issue will auto resolve when the policy is in compliance.
Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.
https://datacommons.org/browser/dc/q18w8j7b2vxm4 should have been assigned a human readable dcid
There are no file names listed in the error table so if multiple files have been uploaded, it maybe more intensive on the user's side to figure out which file is causing the parsing error. It would be nice to have the file name in error table.
NOTE: I'm not 100% sure this was the root of the problem, but I can't find any other explanation and I can't reproduce previous API responses.
There was a change to the data returned when requesting the county population using the place-obs API, which caused the dashboard's server to crash.
The following API requests would return a key/value for both geoId/3651000 and geoId/2938000.
req_url="https://api.datacommons.org/bulk/place-obs",
req_json={"placeType": "County", "populationType": "Person", "observationDate": "2018"}.
The value contained the name of the city, but not the population.
Today, the API is suddenly no longer returning any key/value pairs for geoId/3651000 and geoId/2938000. So NaN is stored in the data frame as the name of the city.
During the calculations, when iterating over the names in the data frame, it encounters NaN as a name since no value was stored and the server crashes.
The tool can currently handle up to 50 MB MCF files if local dcids do not need to be resolved. For unresolved MCF files, the limit is around 30 MB.
Every 30 seconds (only tested on Chrome) while loading in an mcf file, a popup appears stating that page is unresponsive. The user is prompted to 'Exit Page' or 'Wait'. This does not mean the application has crashed, it is just taking a long time to load the mcf file. Theoretically, waiting a long enough amount of time should load in the MCF, given local memory does not run out of space. For example, uploading the first 1,000,000 lines of this CovidMobility file from gs://resolved_mcf/CovidMobility/CovidMobility.mcf (resulting in 33MB) takes about 1.5minutes to load in which the user is prompted 3 times to choose between exiting and waiting. For a 100MB file, I waited about 17minutes for it to load, but it never loaded.
A related problem is that if the user navigates to another tab while a file is loading, the application is blank upon returning.
The tool would be significantly more useful if it could handle files up to 100 MB consistently. For now it can load up to 50MB within a reasonable amount of time.
I am not sure exactly how this problem can be solved. If the parsing can be done asynchronously or perhaps break the file into smaller sections and parse in parallel, that could speed it up.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.