agrc / api-client

The official UGRC API client

UGRC API Client

The Utah Geospatial Resource Center (UGRC) is the State of Utah’s map technology coordination office. The UGRC creates, maintains, and stores geospatial data in the State Geographic Information Database (SGID), a one-stop shop to hundreds of data layers developed, aggregated, or acquired by state government. UGRC’s API is an http-enabled service for accessing this valuable geospatial data.

From querying any data layer in the SGID to geocoding addresses against the most accurate statewide roads dataset, the application of the API is endless. The API also powers UGRC’s widgets, toolboxes, and add-ins, which can help you navigate the sea of data in the SGID.

The UGRC API Client is an app to help make geocoding addresses simple. This app allows the user to use the UGRC API without any licensed software or programming knowledge; Drag and drop a file and then click start.

Check out our introductory blog post and our getting started video.

Development Specifics

Certificates

An apple developer certificate is required to sign the application for distribution.

1. Using keychain's certificate assistant, request a certificate from a certificate authority.
   • Leave CA Email address empty
   • Request is Saved to disk
2. Provide the certificate request to Certificates, Identifiers & Profiles to create a Developer ID Certificate
   • Only the account holder of the Developer Group can create this type of certificate.
3. Import the certificate to keychain and then export it as a p12 file for the deployment pipeline.
   • Store the password required for exporting for the GitHub Actions secrets.

Deployment pipeline set up

1. Create a prod and dev GitHub repo environment.

2. Store the password and the p12 certificate as GitHub Action secrets in the environment:

   • gh secret set APPLE_CERTIFICATE -b$(base64 -i ~/certificate.p12) --env=prod
   • gh secret set APPLE_CERTIFICATE_PASSWORD -b<password> --env=prod
   • gh secret set WINDOWS_CERTIFICATE_PASSWORD -b<password> --env=prod

3. Add the rest of the environment variables as secrets:

   • APPLE_IDENTITY: the name of the developer id certificate name as it appears in keychain
   • APPLE_TEAM_ID: the team id to notarize under viewable on your Apple Developer Account "Membership details", page
   • APPLE_USER_ID: the full email address of your Apple Developer Account
   • APPLE_PASSWORD: the app-specific password (not your Apple ID password) created on https://appleid.apple.com/account/manage

Deploying a new version

GitHub Actions will create and update a draft release with every merged pull request. When a release is desired, publish the draft release. Another GitHub action will sign the binaries and upload them to the release assets. The API Clients will silently download the update and the installation will occur on the next restart of the app.

Uploading Debug Symbols to Sentry

Each time Electron is upgraded, new debug symbols need to be uploaded to the Sentry servers. This can be done by creating a sentry.properties file and then running: npm run upload-debug-symbols.

api-client's Issues

show address being geocoded on the status page

street and zone

allow user to choose output spatial reference

right now we don't give any access to api parameters

[Request]: Show Offline Status

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

If the app is offline, we should let the user know up front so that they don't waste time going through the screens.

[Bug]: Crash when selecting fields

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

1.2.1

Electron Version

15.1.2

Operating System

Windows

What happened?

App window turns white and displays nothing after a street name field or zone name field is selected

Steps To Reproduce

add_tester.csv
Add table
Select a field name

Relevant log output

react-dom.production.min.js:5193 Error: Action type: undefined is an unknown reducer type. {"action":"UPDATE_FIELDS","payload":"ROADNAME","meta":"street"}
at Ie (GeocodeContext.js:34)
at immer.esm.js:949
at e.produce (immer.esm.js:962)
at immer.esm.js:947
at Object.sa [as useReducer] (react-dom.production.min.js:3859)
at n.useReducer (react.production.min.js:389)
at use-immer.module.js:18
at Ge (GeocodeContext.js:39)
at ia (react-dom.production.min.js:3762)
at Ga (react-dom.production.min.js:4406)

analytics

some things that would be nice to know from the people using this and would also be some great stats to show the user as well...

• number of row
• total time taken
• match percentage
• average geocoding score
• street and zone fields
• file type used

[Bug]: Funny characters in dialog title on windows

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

1.3.0

Electron Version

not sure

Operating System

Windows

What happened?

As reported by @eneemann...

Steps To Reproduce

No response

Relevant log output

No response

Test api key

When the api key is entered, execute a request to make sure it 200's otherwise show a message with some common fixes etc.

[Request]: Remember Window Position

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

It's a pain to resize and reposition the window each time I restart the app.

Describe the solution you'd like

It would be super-helpful if this app remembered its window size and position between sessions.

create completion view

a completion view should show

• number of row
• total time taken
• match percentage
• average geocoding score
• street and zone fields

Maaaaaaaybe some type of button to format the stats in a way that could be shared (tweeted, emailed, etc)

Better Download Links

This is not very user-friendly:

I wonder if there's a way to give users more clear download buttons like "Window" and "macOS" rather than expecting them to know which of these files is the correct one for their OS.

Maybe we could have generic download links on the Readme? The hard part would be updating them with each release. Maybe we could do that via a GitHub action. Or maybe we need a product page on gis.utah.gov that dynamically queries the electron updates endpoint and presents the latest link to the user.

Or maybe we could get rid of the version numbers in the release assets and link to them like this.

display version number

place in the footer somewhere

[Bug]: Files without a header don't seem to geocode first row

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

1.5.1

Electron Version

16.0.1

Operating System

Windows

What happened?

If I provide a csv without a header that has 522 rows and select the proper matching fields, the preview only shows 521 rows and the output only has 521 rows as well, missing the top row.

Steps To Reproduce

1. Delete the header row from a csv. Note the number of rows and the top row.
2. Geocode, selecting the proper fields.
3. Compare number of rows geocoded and output to input file.

Relevant log output

No response

Feedback

add a form for feedback to capture any thing the user might want to tell us.

Drag and drop

enable drag and drop of common geocoding files

CSV
XLS, XLSX
Shapefile?

Add ability to do a point-in-polygon analysis with data from OpenSGID

Give users the ability to add data from polygons from OpenSGID to the output.

move package to devDependency

api-client/package.json

Line 79 in 5d2b9fd

"@electron-forge/maker-dmg": "^6.0.0-beta.61",

[Request]: Add email confirmation step to getting started instructions

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

People are following the instructions in the api client to a T and not following the instructions in the web api developer console. So they create a key and it doesn't work because their new account isn't verified.

Describe the solution you'd like

Add another step to verify your account in the api console.

linked fields disappear upon page navigation

when returning to the data page after selecting a file, the field linking is reset.

[Request]: Add ability to geocode CSV without header row

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

If I provide a csv without a header that has 522 rows and select the proper matching fields, the preview only shows 521 rows and the output only has 521 rows as well, missing the top row.

Describe the solution you'd like

@jacobdadams how would you want this to work in regards to selecting columns and letting the app know there is no header?

create geocoding view

this should show a

• progress bar
• rows processed
• rows remaining
• active match rate %
• average score
• time elapsed
• estimated time remaining
• cancel button

Auto-update is not working properly

If you open the prod app, no update notification ever shows up. However, you can get it to show up by running it with the --inspect parameter: ./UGRC\ API\ Client.app/Contents/MacOS/ugrc-api-client --inspect=9229.

Cant change api key

when the api key is set the back button on the data page does not work

screen design

create max width and updates for large screens.

Data Page: Next button appears after clearing existing file

Steps to reproduce:

1. Choose a file
2. Click the clear button

Expected that the next button will be hidden again until a file is chosen.

rename info to plan

the plan page needs to be styled and renamed

[Request]: handle sub-addresses

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

addresses containing sub-addresses do not geocode, and return a score of 0.

Describe the solution you'd like

geocode the address without the sub-address (aka: remove the sub-address) - or even better, incorporate the sub-address in the geocoding process.

Display Output Table as Grid

It would be nice if we displayed at least a selection of the output CSV for the user to review after the geocoding process is complete.

links open in electron window

links set to target blank do not open in the default browser.

Allow more file types

for the MVP we stripped the accepted file types to CSV only. We should figure out what other file types people expect and implement them.

[Bug]: rows with sub-address in 'street' field returns with geocode score of zero

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

1.5.2

Electron Version

16.0.1

Operating System

Windows

What happened?

records/rows in my .csv file that contained a sub-address in the street field did not geocode and returned with a score of 0. it could be b/c the sub-address was prefixed with a pound (#). Example:

street, zone
81 E 100 N #23, 84535

Steps To Reproduce

I generated the list of addresses from the open sgid address points using this query:
select fulladd as street, zipcode as zone from location.address_points limit 50

here's the output .csv from that query:
sample_geocodes.csv

Relevant log output

No response

build a fun way to link fields

https://anseki.github.io/leader-line/ looks like a good library to use.

[Request]: Allow csv fields to be combined

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

My CSV has the street address split up into multiple columns and I have to merge them before I can use the app.

Describe the solution you'd like

It would be nice to be able to have a UI that allows me to choose fields and arrange them so that their combination can be used as the input for geocoding.

Provide results as a file

Duplicate input file schema with the geocoding results appended.

Fields to append: X, Y, score, match address

When spatial reference is 4326, use Lat Lon instead of x and y columns.

Implement Sentry

I've gone down a huge rabbit hole trying to implement Sentry. It would be good for both of us to take a look at it together. I've push a wip commit of where I got to. My main issue is trying to call init from the preload script. I've read that it need to be called in main, renderer, and preload. I feel like I'm one webpack config from getting it working.

I did run node sentry-symbols.js to upload the electron symbols.

Some helpful links:

https://docs.sentry.io/platforms/javascript/guides/electron/

getsentry/sentry-electron#366

This comment makes me think that we should just wait until v3 final of @sentry/electron is released.

Originally posted by @stdavis in #75 (comment)

[Bug]: Frozen during longer geocoding session

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

1.2.1

Electron Version

15.1.2

Operating System

MacOS

What happened?

I started a larger geocoding session and the UI froze after a few minutes. Nothing in the console or network tab.

Steps To Reproduce

test_addrs.csv

Start a geocoding session with the test data above...

Relevant log output

[2021-10-14 16:23:44.834] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.0
[2021-10-14 16:23:44.844] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-14 16:23:45.487] [info]  checking-for-update
[2021-10-14 16:23:45.750] [info]  update-not-available
[2021-10-14 16:40:42.358] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.0
[2021-10-14 16:40:42.364] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-14 16:40:42.887] [info]  checking-for-update
[2021-10-14 16:40:43.490] [info]  update-not-available
[2021-10-14 17:01:40.427] [info]  update-electron-app config looks good; aborting updates since app is in development mode
[2021-10-14 17:16:35.737] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.0
[2021-10-14 17:16:35.747] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-14 17:16:36.203] [info]  checking-for-update
[2021-10-14 17:16:36.513] [info]  update-not-available
[2021-10-14 17:20:56.027] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.0
[2021-10-14 17:20:56.030] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-14 17:20:56.424] [info]  checking-for-update
[2021-10-14 17:20:56.731] [info]  update-available; downloading...
[2021-10-14 17:21:04.506] [info]  update-downloaded [
  {
    preventDefault: '[function] function preventDefault() { [native code] }',
    sender: {
      checkForUpdates: '[function] function checkForUpdates() { [native code] }',
      getFeedURL: '[function] function getFeedURL() { [native code] }',
      setFeedURL: '[function] function setFeedURL() { [native code] }',
      quitAndInstall: '[function] function quitAndInstall() { [native code] }',
      _events: {
        error: '[function] e=>{h("updater error"),h(e)}',
        'checking-for-update': '[function] ()=>{h("checking-for-update")}',
        'update-available': '[function] ()=>{h("update-available; downloading...")}',
        'update-not-available': '[function] ()=>{h("update-not-available")}',
        'update-downloaded': '[function] (e,t,r,s,o)=>{h("update-downloaded",[e,t,r,s,o]);const n={type:"info",buttons:["Restart","Later"],title:"Application Update",message:"win32"===process.platform?t:r,detail:"A new version has been downloaded. Restart the application to apply the updates."};l.showMessageBox(n).then((({response:e})=>{0===e&&c.quitAndInstall()}))}'
      },
      _eventsCount: 5
    }
  },
  "# What's Changed\r\n" +
    '## 🐛 Bug Fixes\r\n' +
    '\r\n' +
    '- add window controls on windows platform @stdavis (#66)\r\n',
  'v1.2.1 🌈',
  '1970-01-01T00:00:00.000Z',
  'https://github.com/agrc/api-client/releases/download/v1.2.1/UGRC.API.Client-darwin-x64-1.2.1.zip'
]
[2021-10-14 17:21:30.568] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.1
[2021-10-14 17:21:30.572] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-14 17:21:30.990] [info]  checking-for-update
[2021-10-14 17:21:31.251] [info]  update-not-available
[2021-10-14 17:26:30.574] [info]  checking-for-update
[2021-10-14 17:26:30.852] [info]  update-not-available
[2021-10-15 08:13:49.439] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.1
[2021-10-15 08:13:49.448] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-15 08:13:50.405] [info]  checking-for-update
[2021-10-15 08:13:50.393] [info]  update-not-available
[2021-10-25 14:25:52.997] [info]  feedURL https://update.electronjs.org/agrc/api-client/darwin-x64/1.2.1
[2021-10-25 14:25:53.003] [info]  requestHeaders { 'User-Agent': 'update-electron-app/2.0.1 (darwin: x64)' }
[2021-10-25 14:25:53.510] [info]  checking-for-update
[2021-10-25 14:25:54.251] [info]  update-not-available
[2021-10-25 14:30:53.006] [info]  checking-for-update
[2021-10-25 14:30:53.302] [info]  update-not-available
[2021-10-25 14:35:52.977] [info]  checking-for-update
[2021-10-25 14:35:53.275] [info]  update-not-available
[2021-10-25 14:40:52.968] [info]  checking-for-update
[2021-10-25 14:40:53.262] [info]  update-not-available
[2021-10-25 14:45:52.961] [info]  checking-for-update
[2021-10-25 14:45:53.538] [info]  update-not-available
[2021-10-25 14:50:52.932] [info]  checking-for-update
[2021-10-25 14:50:53.224] [info]  update-not-available
[2021-10-25 14:55:52.927] [info]  checking-for-update
[2021-10-25 14:55:53.203] [info]  update-not-available
[2021-10-25 15:00:52.919] [info]  checking-for-update
[2021-10-25 15:00:53.212] [info]  update-not-available
[2021-10-25 15:05:52.891] [info]  checking-for-update
[2021-10-25 15:05:53.191] [info]  update-not-available
[2021-10-25 15:10:52.879] [info]  checking-for-update
[2021-10-25 15:10:53.177] [info]  update-not-available
[2021-10-25 15:15:52.871] [info]  checking-for-update
[2021-10-25 15:15:53.139] [info]  update-not-available
[2021-10-25 15:20:52.845] [info]  checking-for-update
[2021-10-25 15:20:53.436] [info]  update-not-available
[2021-10-25 15:25:52.837] [info]  checking-for-update
[2021-10-25 15:25:53.112] [info]  update-not-available
[2021-10-25 15:30:52.821] [info]  checking-for-update
[2021-10-25 15:30:53.117] [info]  update-not-available

App Crash Report

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

App Version

0.0.0-dev

Electron Version

15.1.2

Operating System

macOS Big Sur

What happened?

Steps To Reproduce

Relevant log output

Error message

window.ugrc.aBadFunctionName is not a function

Stack trace

TypeError: window.ugrc.aBadFunctionName is not a function
    at eval (webpack-internal:///./src/pages/Data.jsx:86:17)
    at invokePassiveEffectCreate (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:3960:1047)
    at HTMLUnknownElement.callCallback (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:657:119)
    at Object.invokeGuardedCallbackDev (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:677:45)
    at invokeGuardedCallback (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:696:126)
    at flushPassiveEffectsImpl (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:3968:212)
    at unstable_runWithPriority (webpack-internal:///./node_modules/scheduler/cjs/scheduler.development.js:465:16)
    at runWithPriority$1 (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:2063:1059)
    at flushPassiveEffects (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:3960:237)
    at eval (webpack-internal:///./node_modules/react-dom/cjs/react-dom.development.js:3945:110)

[Request]: Explain how to recover from a fast fail

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Fast fail instructs you that the data is bad but that isn't always the case. We can give better instructions of things to geocode successfully.

api-client/src/pages/Geocoding.jsx

Lines 106 to 108 in 9a1fe40

	A fast failure occurs when the <span className="font-bold">first {stats.failures} records</span> do not
	succeed to geocode. This is often an indication that the data being processed is not correct. You should
	try to fix the data and re-run the job.

Describe the solution you'd like

The way I see it there are 3 reasons for a fast fail to occur. Starting at the bottom,

1. Your api key is invalid.
2. The fields are mapped incorrectly.
3. The data has 25 initial bad rows.

The solutions are simple

1. Click the api key in the top right to see if you get a thumbs up. Create a new key if you fouled this up. We should read the api response and highlight this option.
2. Check your field mapping. Does the sample data look correct?
3. Remove or reorder the first 25 rows of your data.

Add these items to the page to help the task get back on track.

[Request]: Add loading indictor to api key validation

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Sometimes the api check takes a few seconds and it appears that the app is frozen.

Describe the solution you'd like

No response

Better macOS Distribution via .dmg

We should look into creating a dmg with a link to the Applications folder and instructions telling the user to drag and drop the app into applications. That way it will be in a more permanent location on their machine.

I guess the other option is a pkg installer that does it for them.

cancel does not work well

The UI doesn't update to notify that the operation was cancelled and
the button doesn't cancel the process on the first click.

create about page

	applicationName: 'UGRC API Client',
    applicationVersion: app.getVersion(),
    authors: fs
      .readJSONSync(contribFile)
      .map(({ name }: { name: string }) => name),
    copyright: '© UGRC',
    credits: 'https://github.com/agrc/?',
    iconPath,
    version: process.versions.electron,
    website: 'https://api.mapserv.utah.gov',

[Request]: Add error boundaries

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

When the application crashes the screen goes white and nothing is available except to close it.

Describe the solution you'd like

Add error boundaries around code that can fail and a global one to catch all unhandled exceptions. That should give people a better opportunity to submit bugs with more relevant information.

The error boundary should have a link to submit an issue and a button or something to copy the relevant crash information to paste into the issue.

refs #71

[Request]: Extract sample street and zone from selected fields

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Someones schema could not be very descriptive and they could select the wrong fields causing them to fast fail.

Describe the solution you'd like

If we pulled a row from the csv when we grab the headers, we could show example data based on their selections. This would make it simple to verify that the correct fields were chosen.

Api key does not update when navigating between views

inputting a new api key and then clicking next does persist the data but when navigating back the input shows the old value. A refresh shows the new data.

split field linking from data upload

the page gets a bit long and might be better off on it's own

Release Drafter is suggesting an incorrect version number

https://github.com/agrc/api-client/releases/tag/untagged-5fc3a354463c400c3ed5 should be v1.3.0, no?

Windows signing

We have a 365 day cert. Let's sign the windows installer.

@nathankota we'll need to keep the cert in the calendar to renew.

stuck on api key screen when api key is set

if the api key is set and a user navigates back to that screen, the next button is disabled until changes are made to the input.

exiting the geocoding page does not stop the job

if you go to the geocoding page, start geocoding, then hit back or view the about page, the geocoding continues. Then if you return to the geocoding page, it starts geocoding again and the stats go all crazy bouncing between.

[Request]: Rename Street Field

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

Most people think of street as the street name of an address. For example 301 south main street. main could be thought of as the street field name.

Describe the solution you'd like

Rename the label to something like Street Address field name. This should signify the full street address (301 south main street) but not the full mailing address (301 south main street salt lake city, utah 84111).