digideskio / pulse

How the .gov domain space is doing at best practices and federal requirements.

The pulse of the federal .gov webspace

How the .gov domain space is doing at best practices and federal requirements.

Other forks of the project in use include:

• https://nrkbeta.no/https-norge/
• https://https.jetzt
• https://govuk.jamiemagee.co.uk/

Setup

Pulse is a Flask app written in Python 3. We recommend pyenv for easy Python version management.

• Install dependencies:

pip install -r requirements.txt

• If developing the stylesheets, you will also need Sass, Bourbon, Neat, and Bitters.

gem install sass bourbon neat bitters

• If editing styles during development, keep the Sass auto-compiling with:

make watch

Initializing dataset

To initialize the dataset with the last production scan data and database, there's a convenience function:

make data_init

This will download (using curl) the current live production database and scan data to the local data/ directory.

Deploying the site

The site can be easily deployed (by someone with credentials to the right server) through Fabric, which requires Python 2.

The Fabric script will expect a defined ssh configuration called pulse, which you should already have defined in your SSH configuration with the right hostname and key.

To deploy to staging, switch to a Python 2 virtualenv with fabric installed, and run:

make staging

This will cd into deploy/ and run fab deploy.

To deploy to production, activate Python 2 and fabric and run:

make production

This will run the fabric command to deploy to production.

Updating the data in Pulse

The command to update the data in Pulse and publish it to production is simple:

python -m data.update

But you will need to do some setup first.

Install domain-scan and dependencies

Download and set up domain-scan from GitHub.

domain-scan in turn requires site-inspector 1.0.2 (not 2.0) and ssllabs-scan.

Pulse requires you to set one environment variable:

• DOMAIN_SCAN_PATH: A path to domain-scan's scan binary.

However, domain-scan may need you to set a couple others if the binaries it uses aren't on your path:

• SITE_INSPECTOR_PATH: Path to the site-inspector binary.
• SSLLABS_PATH: Path to the ssllabs-scan binary.

Configure the AWS CLI

To publish the resulting data to the production S3 bucket, install the official AWS CLI:

pip install awscli

And link it to AWS credentials that allow authorized write access to the pulse.cio.gov S3 bucket.

Then run it

From the Pulse root directory:

python -m data.update

This will kick off the domain-scan scanning process for HTTP/HTTPS and DAP participation, using the .gov domain list as specified in meta.yml for the base set of domains to scan.

Then it will run the scan data through post-processing to produce some JSON and CSV files the Pulse front-end uses to render data.

Finally, this data will be uploaded to the production S3 bucket.

Ideas for later versions

This project is an initial pass - there is much more information that can be represented in dashboards to great effect. Below are some of the further ideas for both for future work on this project. Feel free to add your ideas here, too.

• For the DAP Dashboard
  • Number of pages from a domain reporting into DAP
  • Number or list of subdomains from a domain reporting into DAP
  • Test the deeper config options that the DAP snippet should be employing, such as IP anonymization, Event tracking, Demographics turned off, and ?????. (Possibly using headless browser)
• Does the site require “www”? Does it require not using “www”?
• Load time (server-side)
• Mobile friendliness (poss. using Google's Mobile Friendly Test)
• Mixed content detection (linking to insecure resources)
• Use of third party services
• 508 compliance (poss. with http://pa11y.org/)
• Any other items listed in the OMB letter to OGP passing along .gov domain issuance
• Lighter or fun things - like how many domains start with each letter of the alphabet, what the last 10 that came out were, etc.
• 2FA or Connect.gov ? - Not sure how it would work but note Section 3's requirement in this EO
• Anything from/with itdashboard.gov
• open source
• Look at what Ben tracked
• IPv6
• DNSSEC
• What else can we get from Verisign?

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.