18f / pulse

How the federal .gov domain space is doing at best practices and policies.

This project was decommissioned by the Office of Government-wide Policy and redirected to digitaldashboard.gov in June of 2020.

=================

The pulse of the federal .gov webspace

How the .gov domain space is doing at best practices and federal requirements.

Documentation	Other Links
Setup and Deploy Instructions	System Security Plan
a11y scan process	Ideas for new sections to add to the site
Ongoing Work	Backlog of feature requests and ideas
ATO artifacts	Open Source Reuse of the site
Project Information

Setup

Pulse is a Flask app written for Python 3.5 and up. We recommend pyenv for easy Python version management.

• Install dependencies:

pip install -r requirements.txt

• If developing the stylesheets, you will also need Sass, Bourbon, Neat, and Bitters.

gem install sass bourbon neat bitters

• If editing styles during development, keep the Sass auto-compiling with:

make watch

• And to run the app in development, use:

make debug

This will run the app with DEBUG mode on, showing full error messages in-browser when they occur.

Initializing dataset

To initialize the dataset with the last production scan data and database, there's a convenience function:

make data_init

This will download (using curl) the current live production database and scan data to the local data/ directory.

Install domain-scan and dependencies

Download and set up domain-scan from GitHub.

domain-scan in turn requires pshtt and sslyze. These can be installed directly via pip.

Pulse requires you to set one environment variable:

• DOMAIN_SCAN_PATH: A path to domain-scan's scan binary.

However, if you don't have pshtt and sslyze on your PATH, then domain-scan may need you to set a couple others:

• PSHTT_PATH: Path to the pshtt binary.
• SSLYZE_PATH: Path to the sslyze binary.

Configure the AWS CLI

To publish the resulting data to the production S3 bucket, install the official AWS CLI:

pip install awscli

And link it to AWS credentials that allow authorized write access to the pulse.cio.gov S3 bucket.

Then run it

From the Pulse root directory:

python -m data.update

This will kick off the domain-scan scanning process for HTTP/HTTPS and DAP participation, using the .gov domain list as specified in meta.yml for the base set of domains to scan.

Then it will run the scan data through post-processing to produce some JSON and CSV files the Pulse front-end uses to render data.

Finally, this data will be uploaded to the production S3 bucket.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

pulse's Issues

phase out master branch?

If we plan to heavily use the gh-pages branch, anyone have a problem with me porting over the material from master to gh-pages and killing the master branch?

[Note: I'd do this after the basic-frontend branch was merged.]

Make this repo public

Hi folks -

The other github.com/gsa admins and I are surveying the private repos and trying to tighten our belt with private to postpone the need to upgrade our plan. I think this repo is fine (even will benefit from) being public. Can we confirm whether we have ay need to keep it private? If there are needs, can we confirm their timetable and see when they might change?

User Interviews with CIO or other stakeholders

Had a quick discussion with @konklone this morning about possibly interviewing one of the CIO's we have been in contact with to talk about the dashboard. It would be great to get clarification on information that they would find useful.

Here are a few suggested questions:

• What agency are you overseeing?
• What are some of the challenges you face when making decisions?
• Is there a particular tool or website that you use for helping make those decisions?
• What kind of information do you find most useful when determining the security protocols used by your web properties?
• Is there a standard that you have in place that the websites you manage are required to take to comply with security standards?

Revise wireframes based on external feedback before initial implementation

Depends on #58.

Kick off the design process for a proper front-end

• Create basic front end layout and structure
• Gather user data and information directed at the product
• Write archetypes for our users (e.g. CIO)
• Start developing various wireframe versions for the team to iterate over

Create a snapshot of at least some of the data we need to display - HTTPS

Participate in the DAP

The front-end should itself participate in the DAP, and possibly have its own Google Analytics account, with the proper privacy knobs on.

Draft a statement of work

Decide which "federal agencies" to display

Currently, there are 126 agencies identified in the data as "federal agencies" in charge of websites. Several of them are agencies in charge of just one domain, others may only have one or two, and many are not what one would traditionally think of as federal agencies (Amtrak, for example). Single-site agencies won't make compelling data visualizations (everything will be 100% in either direction), so I would propose that we remove them from visualizations, while still making their data available in the table.

Create one-time snapshot of .gov DAP participation

I generated analytics.csv by matching the federal (exec+leg+jud) .gov domain list to the 3800 sites the DAP says participate.

This is almost good enough, but the ~300 exec branch domains DAP says participate doesn't line up with this -- there are more domains there (and that's only exec branch).

Let's figure out why that is, so we can finalize the snapshot.

Question: What should the url be?

Build a backlog

Dashboard - Brainstorming Ideas

Add ideas here

Pick a reliable tool to determine DAP participation for a given domain/subdomain

Change placeholder text on live pulse.18F.gov page

Could we make the placeholder text at the top of the pulse page a little bit more official sounding in case someone who comes across the page is not technical? There has been some concern that the pre-alpha language might be confusing to non-technical folks. I know we're not publicizing anything yet, so no outside parties should get onto the site in theory, but some folks at OGP are uneasy about the language as it stands nonetheless. I'd suggest "This website is in a very early stage of development. No information or content found here should be considered definitive."

Thanks, and sorry for the nitpickiness of this issue.

Plot out implementation implications

cc: @juliaelman, @vzvenyach

Ideas for later versions of the DAP dashboard

Post ideas below...

Get Feedback on wireframes v2

Put simple front-end on the live domain

Under proper HTTPS and participating in the DAP.

Depends on #17.

Send a note to agency CIOs

Letting them know this is happening, along with any data we plan on displaying.

Publish list of federal .gov subdomains

Pulse.cio.gov currently runs on top of the list of federal .gov 2nd level domains (e.g. fbi.gov, irs.gov). The dataset of federal .gov subdomains (e.g. blog.trade.gov, calendar.nih.gov) is not currently made public but hopefully will be at some point. Publishing it would allow the pulse scans to gain another level of detail.

Programmatic access to DAP data

We need to:

• Create a developer project in the Google dashboard. Noah's working through the IT limitations here.
• Have DAP authorize the credentials associated with this developer project to access their data.

Plan long run O&M work

Ensure how this will be paid for and maintained in 6, 12, 18, 36 months, etc.

Create a snapshot of at least some of the data we need to display - 18F

Coordinate a way to quickly export up-to-date official .gov domain data

Create a simple-as-necessary front-end

...that displays this snapshotted data, with whatever “under construction” caveats are necessary.

Draft a statement of need

Prepare wireframes for initial implementation

Building on #55

Compare official .gov fields indicating whether site is up to our list

The .gov registrar has a field in one of their spreadsheets that tracks whether the site is up or down. We should compare this against our own dataset, based on site-inspector, to see if there are many major discrepancies.

cc @jtexnl - I think this would be a good one for you. (I can formally "assign" issues once you've joined the org.)

Pick a reliable tool to determine HTTPS/HSTS compliance for a given domain/subdomain

Initial ATO

• Get delegated authority from OGP to use 18F's ATO process.
• Prepare template ATO for 18F DevOps.
• Have something up there besides the word "PULSE".
• Meet at scheduled time with DevOps to complete process and obtain the ATO.

Set up new Cloud Foundry space/org

• Have a new instance/org setup
• Add users to that instance/org
• Add manifest.yml and travis.yml, per instructions outlined here >> https://github.com/18F/cloud-foundry-notes/blob/static_site_instructions/content/apps/static.md

Weigh future integration of subdomains into dashboards

Work out Terms issues with SSL Labs

We need to get a modification to the SSL Labs API T&C.

Methodology and field description

We need some FAQ and documentation about what this is, what it's doing, how it's doing it, and why.

An About page, with an FAQ

A project description, and some FAQ material. It can be the same page.

Delegate that domain name’s DNS to 18F’s infrastructure

Update pulse.cio.gov to match wireframes v4

Create Documentation for Maintenance

Move the repository into the 18F GitHub organization

Set up Travis CI

It should at least run the commands in our README (e.g. npm install and bower install and gulp [something]) to exercise them and make sure we're not breaking stuff. The default gulp command watch-es, which isn't what we want for CI.

Create a proper (and evolving) design/UX and information architecture

perform usability testing

Visual theme from wireframes

Based off of #55 and #59, create v4 of the wireframes. This will be what we plan to end sprint 2 with.

Scope out cloud resource expenses over the long haul

Arrive at consensus on wireframes for the phase 1

The steps for this are:

• Get Feedback on wireframes v2
• Design Wireframes v3
• Gain external feedback on wireframes v3
• Design wireframes v4

Pick a reliable tool to determine whether a given domain/subdomain is live and/or a redirector for another domain.

'Design' v1 of the dashboard

Brainstorm: Future Dashboard Possibilities

Such as:

• More HTTPS detail, e.g. SHA-1, forward secrecy
• Does the site require “www”? Does it require not using “www”?
• Load time (server-side)
• Mobile friendliness
• Mixed content detection (linking to insecure resources)
• Use of third party services
• 508 compliance (poss. with http://pa11y.org/)

Add more ideas below.

Auto-update site when updating repo

Could be via webhooks, or using CF continuous deployment -- one of the two.