rspeed / django-lookout

An API endpoint which collects reports from Content Security Policy (CSP) and HTTP Public Key Pinning (HPKP)

Django Lookout

Django Lookout is an API endpoint for collecting and processing automatic incident reports send by your visitors' web browsers. Currently that includes both Content Security Policy (CSP) and HTTP Public Key Pinning (HPKP), but support for additional report types is planned.

Before getting started you should familiarize yourself with the standards and their potential pitfalls (especially HPKP). The risks can be mitigated significantly by using Django Lookout along with report-only policies, which would still allow you to be notified of potential attacks without the risk of accidentally rendering your web site inaccessible.

It's important to note that Django Lookout only handles the reporting part of the process. Setting the headers which tell browsers what to do, or even where to send reports, is outside its scope. You'll need to set the report-uri property for CSP and/or HPKP to point to your Django Lookout endpoint.

Install and Configure

See documentation.

Standards

Currently Supported

HTTP Reporting API

A generic incident reporting API that can be used by all of the following standards. Django Lookout automatically converts reports using "legacy" schemas to the generic schema.

Content Security Policy

Browsers will (optionally) block unauthorized content and send an incident report if a resource is requested which isn't permitted by the policy.

HTTP Public Key Pinning

Browsers supporting HPKP will (optionally) block connections and send an incident report if the site doesn't use the specified HTTPS certificate in the future.

Planned Support

These standards are planned to be supported in Django Lookout 1.0:

Network Error Logging

Browsers supporting NEL will send incident reports if a networking error is encountered when requesting content.

Expect-CT

Browsers supporting Report-CT will send an incident report if it receives a certificate which doesn't adhere to Certificate Transparency guidelines.

Expect-Staple

Browsers supporting Expect-Staple will send an incident report if a TLS handshake with the site doesn't include an OCSP response.

Browser Implementation Status

The only standard currently supported across all major browsers is Content Security Policy. It's hoped that the HTTP Reporting API will significantly improve the situation in the future.

Note

This table only considers a feature supported if it includes reporting functionality.

Note

Internet Explorer is excluded due to the fact that it doesn't support any of these features via standard headers.

Standard	Chrome	Firefox	Edge	Safari
Content Security Policy (CSP)	Supported	Supported	Supported [1]	Supported [2]
HTTP Public Key Pinning (HPKP)	Supported [3]	Not Supported [4]	Not Supported [5]	Not Supported
HTTP Reporting API	Not Supported [6]	Not Supported	Not Supported	Not Supported
Network Error Logging (NEL)	Not Supported [7]	Not Supported	Not Supported [8]	Not Supported
Expect-CT	Supported [9]	Not Supported [10]	?	?
Expect-Staple	?	?	?	?

Additional Information

Content Security Policy

• Google Web Fundamentals – Content Security Policy
• Content Security Policy - An Introduction

HTTP Public Key Pinning

• Google – Rolling out Public Key Pinning with HPKP Reporting
• Guidance on setting up HPKP

Tools and Similar Projects

• Observatory by Mozilla. General website security testing suite.
• securityheaders.io. Testing suite for security-related HTTP response headers.
• django-csp-reports. A similar project specifically for CSP reports.
• report-uri.io. A commercial service which serves a similar purpose. They also have some useful free testing tools.

Footnotes

[1]	Supported as of build 15002

[2]	Unknown when support was added

[3]	Supported as of Chrome 46

[4]	No support for report-uri

[5]	Under consideration

[6]	Complete, but not yet in a production release

[7]	In development

[8]	Under consideration

[9]	Supported as of Chrome 61

[10]

Planned

django-lookout's Issues

Improve report data rendering in the admin

Currently it's using Pygments to add syntax highlighting, but that's sub optimal. Instead, use the JSON schemas and their metadata to show useful information.

Basic Tests

Create tests for schema validation, database storage, and log file output.

No matching schema when reports are wrapped with an array

The draft Reporting API spec allows multiple reports to be submitted at the same time by wrapping them in an Array (JSON)/ List (Python). This application doesn't handle that properly, and instead reports that the submission doesn't match any known schemas.

Create Exceptions

Create exceptions so parsing/schema errors can be reported to the browser.

RuntimeWarning: DateTimeField received a naive datetime

Due to USE_TZ being enabled. Need to replace datetime.datetime.now with django.utils.timezone.now.

Log unknown schemas

Special-case logging reports which don't match any known schemas.

Add missing fields to CSPSchema

Would be useful to have for the admin display.

Specifically:

• effective-directive (same as violated-directive)
• status-code

Plus three that are present in Chrome's reports:

• source-file
• line-number
• column-number

Management command to clear old incident reports

Named clearincidentreports. Similar to clearsessions.

• Option to keep reports newer than a certain date/time period
  • ./manage.py clearincidentreports --before 2017-06-01
  • ./manage.py clearincidentreports --before 1 week
• Ability for it to be run automatically through a cronjob, etc.
• Add the ability to flag incident reports to keep.
  • Using a custom Admin Action.
  • From the individual report view. Might conflict with #19.

migration problem: lookout.0003_auto_20171008_2324: Multiple primary key defined

Using
Python 3.6
Django-Lookout 0.1.1
Django 2.0.2

Trackback:

Running migrations:
  Applying lookout.0003_auto_20171008_2324...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1068, 'Multiple primary key defined')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 25, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 122, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/operations/fields.py", line 84, in database_forwards
    field,
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/schema.py", line 40, in add_field
    super().add_field(model, field)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 421, in add_field
    self.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 117, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1068, 'Multiple primary key defined')

Merge modules in the logging package

They were split into separate modules during an earlier stage of pre-development planning, but have since been simplified significantly.

Name Change?

Better to do this now rather than later. I don't really like the name. Other possibilities:

• Django-Incident-Reporting-API
• Django-Client-Incident-Reporting
• Django-Out-of-Band-Reporting
• Django-Incident-Reporting-Endpoint

No matter what it will start with "Django" (since it's a Django app), but that won't be included in the package name.

Submit to PyPI

Starting at v0.1.

Depends on #7.

TypeError when normalizing legacy HPKP when USE_TZ is False

TypeError: can't subtract offset-naive and offset-aware datetimes

When USE_TZ is false, timezone.now produces a datetime object without a timezone component. There are two possible solutions:

1. Convert the object output by django.utils.dateparse.parse_datetime to UTC, then strip it of its timezone.
2. Require that USE_TZ be enabled.

I'm currently leaning towards the second option, as the default output of django-admin startproject has it enabled.

Basic Documentation

Separate from README.

Outline steps to install and configure a typical Django-HTTP-Reporting-API installation.

Remove report data from `schema.BaseSchema` in favor of `models.Report`

The BaseSchema class represents both report data and its schema, but the data is redundant with the Report model. Originally I didn't want to add model records by default, but I've decided that should be optional through a specific setting, rather than the logging configuration.

• Remove any code which stores report data in BaseSchema and its subclasses.
• Modify views.ReportView to create the model instance immediately.
• Remove logging.DatabaseLogHandler, which would be redundant.
• Update setup instructions in README.md to reflect these changes.
• Modify logging.ReportMessage.msg to contain a model object, rather than schema.

Add a mapping between report type and a verbose name.

For example: a report with the type "csp" could be displayed as "Content Security Policy violation report".

Useful for #5.

Add an "enforced" field to the model

NullBooleanField which indicates whether or not action was taken, or if it was only reported. This varies by the type of report. For some it will always be true (network errors), some always false (deprecation notices), and others will depend on the server configuration (CSP, HPKP).

Add missing schemas

• nel / nel-report (legacy): Network Error Logging
• expect-ct
• expect-staple

Possibly split these into a separate task later:

• deprecated: Use of a deprecated API.
• intervention: The browser prevented some action.
• crash: The browser process crashed while on the site.
• Javascript errors?
• CSS / HTML parsing errors?

Improve README

The readme file needs some improvements:

• Provide a better explanation of the project's purpose.
• Add links to documentation, tutorials, and projects which will help people setting up their sites for CSP and HPKP. Info for Out-of-Band reporting can probably wait until it's closer to being adopted.
• Describe the Reporting API and its relevance to the project.
• Add a table detailing the various specifications and their support by browser.

Don't use `django.apps` to get your own models

In logging.py. It's just unnecessary.

Add type annotations

Useful for API documentation and debugging.

Remove setup dependency on `pypandoc`

It's way more trouble than it's worth. Just run pandoc directly using subprocess.

Set up continuous integration

Requires #2

Model creation method should be in a model manager

Currently it's in the model class itself.

Fix readme on PyPI

It isn't rendering the RST as HTML.

Generate additional test cases based on actual browser reports

To improve real-world coverage, every known report should have its own test case.

• Make a list of every type of HPKP and CSP error.
• Create a website that is able to make browsers generate reports.
  • Ability to set (and possibly configure) both HPKP and CSP headers.
  • Ability to violate those settings.
• Create a second website for capturing reports.
  • Report API endpoint that doesn't really do anything.
  • Configure mitmproxy to sit in front of the report site, or integrate through its API.
• Run through each known error type using each of the major browsers.
• Export the raw requests and use them to create new test cases.

This will be an unrelated and rarely used codebase, so it'll go in an empty branch.

Create templates for plaintext report rendering

Logging to files currently just dumps the JSON as a string. It would be more useful if cleanly formatted.

Add a custom Admin view for reports

Currently it's overriding the template for the edit view, but makes it somewhat inflexible.

Remove `setuptools_scm` from `package_version`

It's far more likely to be redundant than successful.