18f / analytics-reporter-api

The Analytics API maintains the schema for the database that the Analytics Reporter writes to.

Analytics API

A system for publishing data retrieved from the Google Analytics API by the Analytics Reporter. The analytics API serves data written to a Progress database by the Analytics Reporter in response to HTTP requests.

Setup

The Analytics API maintains the schema for the database that the Analytics Reporter writes to. Because of this, the Analytics API must be setup and configured before the Analytics Reporter starts writing data.

First, the database needs to be created:

createdb analytics-reporter

Once the database is created, the app can be cloned and the dependencies can be installed via NPM. The install script has a postinstall hook that will migrate the database.

git clone [email protected]:18F/analytics-reporter-api.git
cd analytics-reporter-api
npm install

Once all of the dependencies are installed, the app can be started.

npm start

The API is not available at http://localhost:4444/

Note that the API will not render any data until Analytics Reporter is configured to write to the same database and run with the --write-to-database option.

Using the API

The Analytics API exposes 3 API endpoints:

• /reports/:report_name/data
• /agencies/:agency_name/reports/:reportName/data
• /domain/:domain/reports/:reportName/data

Each endpoint renders a JSON array with the most recent 1000 records the Analytics Reporter has generated for the given agency and report. If no records are found, an empty array is returned.

Records are sorted according to the associated date.

Limit

If a different number of records is desired, the limit query parameter can be set to specify the desired number of records.

/reports/realtime/data?limit=500

The maximum number of records that can be rendered for any given request is 10,000.

Page

If the desired record does not appear for the current request, the page query parameter can be used to get the next series of data points. Since the data is ordered by date, this parameter effectively allows older data to be queried.

/reports/realtime/data?page=2

Running the Tests

The Analytics API application is backed by a test suite that uses Mocha to run tests.

Before running the test suite, a database for the tests will need to be created.

createdb analytics-api-test

Then the tests can be invoked via NPM. The test script has a pretest hook that migrates the database.

npm test

Creating a new database migration

If you need to migrate the database, you can create a new migration via knex, which will create the migration file for you based in part on the migration name you provide. From the root of this repo, run:

`npm bin`/knex migrate:make <the name of your migration>

See knex documentation for more details.

Running database migrations

Locally

npm run migrate

In production

In production, you can run database migrations via cf run-task. As with anything in production, be careful when doing this! First, try checking the current status of migrations using the migrate:status command

cf run-task analytics-reporter-api --command "knex migrate:status" --name check_migration_status

This will kick off a task - you can see the output by running:

cf logs analytics-reporter-api --recent
# the output will look something like...
2021-07-19T14:31:39.89-0400 [APP/TASK/check_migration_status/0] OUT Using environment: production
2021-07-19T14:31:40.16-0400 [APP/TASK/check_migration_status/0] OUT Found 3 Completed Migration file/files.
2021-07-19T14:31:40.16-0400 [APP/TASK/check_migration_status/0] OUT 20170308164751_create_analytics_data.js
2021-07-19T14:31:40.16-0400 [APP/TASK/check_migration_status/0] OUT 20170316115145_add_analytics_data_indexes.js
2021-07-19T14:31:40.16-0400 [APP/TASK/check_migration_status/0] OUT 20170522094056_rename_date_time_to_date.js
2021-07-19T14:31:40.16-0400 [APP/TASK/check_migration_status/0] OUT No Pending Migration files Found.
2021-07-19T14:31:40.17-0400 [APP/TASK/check_migration_status/0] OUT Exit status 0

To actually run the migration, you would run:

cf run-task analytics-reporter-api --command "knex migrate:latest" --name run_db_migrations

See knex documentation for more details and options on the migrate command.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

analytics-reporter-api's Issues

Duplicate reports

Apparently, the site and domain reports are really the same thing. Would there be any reason not to remove the site report from /developer?

502 Bad Gateway: Registered endpoint failed to handle the request

I have been using the API for some time to access the publicly available data.

About a month or so ago, I noticed that I would sometimes get a 502 Bad Gateway response.

However, I have been unable to pull data yesterday (4/7/21); I keep getting a 502 Bad Gateway: Registered endpoint failed to handle the request.

The specific URL that I am trying to hit is:
https://api.gsa.gov/analytics/dap/v1.1/domain/healthcare.gov/reports/domain/data?api_key=MY_API_KEY&after=2019-01-01&limit=10000

I have also tried it with api_key=DEMO_KEY and without the date and limit parameters.

I also tried a different government website and I got the 502 error as well.

Today (4/8/21), I tried again. I hit the domain Healthcare.gov this morning and it retrieved data without a hitch.

However, I then tried cuidadodesalud.gov and it came back with the 502 gateway error. I tried it a second time and it bounced with the 502 error again.

To test, I went back and tried to hit healthcare.gov and this time it also returned the 502 gateway error.

Handle Node 10.x.x buildpack deprecation

The Node.js buildpack no longer supports Node 10.x.x, which was the default previously (and what is being used by our app currently). See this announcement.

We should investigate upgrading to a newer version of Node, and/or pin the version to an earlier buildpack release until we're able to test the changes with a newer version of Node.

Configure New Relic database instrumentation

Right now, the app doesn't report DB metrics/queries etc to New Relic. After some initial investigation, this is likely due to the fact that we use knex, and the New Relic Node agent doesn't support knex. There are a few workarounds mentioned in this thread, though it's not clear if those will work for us.

Download report bug with `domain` route

As per @lauraGgit, the way the route works for the download report is that is take the initial 1000 default limit and culls the domain data from that. So, for example, using /domain/epa.gov/report/download returns about 30 results, because there were 30 relevant datapoints from the initial 1000 limit.

Is there a way we can make this more consistent?

Restrict access to api.data.gov APIs

We'll want to restrict access to just the api.data.gov APIs. Here's the instructions for that.

It'd be awesome if this was optional so orgs who don't want to front their API with api.data.gov don't have to.

Investigate slow endpoints / performance

As part of looking into #122 and looking at metrics in New Relic, it appears that many standard queries (i.e. ones that are run as part of handling the core 3 endpoints) are much slower than they should be. There are seemingly appropriate indexes in place, but requests still seem a lot slower than they should be.

The 502s reported in the issue linked above are likely just the worst case scenarios for queries that are slow in general; so if we can figure out why those are slow in general, we'll likely also address the 502s issue.

Setup new relic monitoring

We have a new relic account, but we do not have monitoring setup for the API.