rogertangos / lostleaks Goto Github PK

Similar to what @ainzero did with the Eversource Preprocessor. Keep in mind that NationalGrid assigns its leaks and repairs unique keys (unlike eversource) and that it sometimes uses shortcodes for cities and towns.

@MichaelRyanWebber wrote some clustering code. It plays leaks and repairs in sequence, and then spits out a list of leaks where there was no repair nearby. It's clean code.

It needs to be moved into this repo and edited in a few ways:

• It work with the repo's file structure
• It should accept a ft/meter parameter for the radius of a repair's effectiveness
• You should be able to pass it the name of a repair and a leak file. (The geocoding scripts in PR #1 should spit this out.)

You'll want to coordinate with @ainzero on what sort of file format(s) clustering will ingest.

This is a good issue for getting familiar with the repo and code structure. It involves modifying a few parameters, but not doing too much coding.

We're operating on assuming all the data files will follow the convention used in the 2016 year. This is fine,but I'd like to just have everyone agree on this going forward (or some other format possible).

We could use some preprocessing code specific to getting the repairs for each gas company processed. I'll start work on this today, I don't think it'll take too much, as much of the groundwork was laid when we wrote code to pre process leaks.

Utility company .csv files in the 2. Pre-Process folder need to be sent to a geocoding API to get their coordinates, formatted address, address type, etc.

There should probably be an abstract Geocoding? class that sends data to google maps, along with a child class (or two) for each utility company, since they each structure their data slightly differently. I wrote this before in another repo... but it'd be awfully nice to use inheritance this time.

The script should live somewhere above the /data directory, and responses from the API should be stored in full in the /data/YYYY/3. JSON directory. Each utility company should probably have its own folder for storing data.

Coordinate with @ainzero, as he's working on this.

This relates to #1 . For each utility, we need to figure out how to store the data we get from them.

Requirements:

• Data should be easily accessible to python
• Data preserved includes
  • exact query placed to the geocoding api
  • the ID of the associated leak/repair
  • full JSON response from google

Nice to Have:

• txt format (so csv, or something that humans can at least kinda parse)
• a search tool

Possible paths include using python pickles, CSV's, or a SQLite database

We could use some code (preferable multi-threaded) that would take our preprocessed data, geolocate it, and write it to a standardized format. This is something I've started working on.

A script to convert KML data into CSV will greatly reduce the complexity to correlate new ppm gas leak data collected while driving around.

Currently, both leaflet and clustering use csv data for visualization and computation. ppm data might as well do the same.

This is a design and code issue, and can be broken up as necessary. Please feel free to ask.

Basically, we need some ways to visualize our data. That may or may not involve updating / recreating a website. (The current website website is static html + css + js and is stored in the old NatGas repo)

Have some ideas for how data about natural gas leaks could be visualized? Feel free to pitch them! We're all ears.

We need to move @MichaelRyanWebber 's code to this repo. Needs:

• It needs to be able to ingest JSON files from the JSON folders.
• Specify what data will be thrown out. (i.e. what location_type)
• Save output, including metadata, like who ran it, when, etc.

https://github.com/MichaelRyanWebber/clustering