datacommonsorg / tools

Data Commons Tools

This is a collaborative repository for developing tools on top of Data Commons APIs. Such tools may include visualization widgets, algorithms for detecting anomalous data, and more.

About Data Commons

Data Commons is an Open Knowledge Graph that provides a unified view across multiple public data sets and statistics. We've bootstrapped the graph with lots of data from US Census, CDC, NOAA, etc., and through collaborations with the New York Botanical Garden, Opportunity Insights, and more. However, Data Commons is meant to be for community, by the community. We're excited to work with you to make public data accessible to everyone.

To see the extent of data we have today, browse the graph using our browser.

License

Apache 2.0

GitHub Development Process

In Github Repo, click on "Fork" button to fork the repo.

Clone your forked repo to your desktop.

Add datacommonsorg/tools repo as a remote:

git remote add dc https://github.com/datacommonsorg/tools.git

Every time when you want to send a Pull Request, do the following steps:

git checkout master
git pull dc master
git checkout -b new_branch_name
# Make some code change
git add .
git commit -m "commit message"
git push -u origin new_branch_name

Then in your forked repo, you can send a Pull Request. If this is your first time contributing to a Google Open Source project, you may need to follow the steps in contributing.md.

Wait for approval of the Pull Request and merge the change.

Support

For general questions or issues about tool development, please open an issue on our issues page. For all other questions, please send an email to [email protected].

Note - This is not an officially supported Google product.

tools's Issues

Covid Dashboard: Change in Stats REST API none return value causes crash

There was a breaking change to the return type of the Stats REST APIs that caused the server to crash with TypeError exception when trying to retrieve "data" from "None".

When making a request using the stats data commons REST APIs, if there was no data to return for a given dcid, it would drop the dcid from the response. Now, instead, it will return dcid->None, which caused issues when trying to read the data value.

Example of return type for dcid without a value:

Exception Message:

data = response[dcid]['data']
TypeError: 'NoneType' object is not subscriptable

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

• default/pycache/main.cpython-37.pyc

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

---

Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

MCF Browser - Poor performance with larger files

The tool can currently handle up to 50 MB MCF files if local dcids do not need to be resolved. For unresolved MCF files, the limit is around 30 MB.

Every 30 seconds (only tested on Chrome) while loading in an mcf file, a popup appears stating that page is unresponsive. The user is prompted to 'Exit Page' or 'Wait'. This does not mean the application has crashed, it is just taking a long time to load the mcf file. Theoretically, waiting a long enough amount of time should load in the MCF, given local memory does not run out of space. For example, uploading the first 1,000,000 lines of this CovidMobility file from gs://resolved_mcf/CovidMobility/CovidMobility.mcf (resulting in 33MB) takes about 1.5minutes to load in which the user is prompted 3 times to choose between exiting and waiting. For a 100MB file, I waited about 17minutes for it to load, but it never loaded.

A related problem is that if the user navigates to another tab while a file is loading, the application is blank upon returning.

The tool would be significantly more useful if it could handle files up to 100 MB consistently. For now it can load up to 50MB within a reasonable amount of time.

I am not sure exactly how this problem can be solved. If the parsing can be done asynchronously or perhaps break the file into smaller sections and parse in parallel, that could speed it up.

MCF Browser - No FIle Name in Error Table

There are no file names listed in the error table so if multiple files have been uploaded, it maybe more intensive on the user's side to figure out which file is causing the parsing error. It would be nice to have the file name in error table.

StatVar renaming script

https://datacommons.org/browser/dc/q18w8j7b2vxm4 should have been assigned a human readable dcid

Covid Dashboard: NYT geographic exceptions no longer returned by place-obs APIS

NOTE: I'm not 100% sure this was the root of the problem, but I can't find any other explanation and I can't reproduce previous API responses.

There was a change to the data returned when requesting the county population using the place-obs API, which caused the dashboard's server to crash.

The following API requests would return a key/value for both geoId/3651000 and geoId/2938000.

req_url="https://api.datacommons.org/bulk/place-obs",
req_json={"placeType": "County", "populationType": "Person", "observationDate": "2018"}.

The value contained the name of the city, but not the population.

Today, the API is suddenly no longer returning any key/value pairs for geoId/3651000 and geoId/2938000. So NaN is stored in the data frame as the name of the city.
During the calculations, when iterating over the names in the data frame, it encounters NaN as a name since no value was stored and the server crashes.

MCF Browser - TMCF Parsing Error Checking

The tmcf+csv parser does not currently check for error cases while parsing. Error checking should be included in this step to give the user a precise location of where/why the parsing error is occurring so they can fix their file accordingly.
The parsing errors table displayed to the user should indicate which line of the TMCF and which row in the CSV file is causing the error.

To do this, ParseTmcf class must check for returned the logged errors. back-end/server-api.js will need to be modified to capture the errors found by ParseTmcf. If capturing these errors causes a change to the structure of App's state parsingErrs, then App.jsx will need to be modified. Finally, ParsingErrsTable.jsx component will need to be modified in order to display the CSV row data.