cal-itp / benefits Goto Github PK
View Code? Open in Web Editor NEWTransit benefits enrollment, minus the paperwork.
Home Page: https://docs.calitp.org/benefits
License: GNU Affero General Public License v3.0
Transit benefits enrollment, minus the paperwork.
Home Page: https://docs.calitp.org/benefits
License: GNU Affero General Public License v3.0
This list is up for discussion.
The following can be configured in settings.py
.
Setting these ensures the cookies expire and are deleted when the user's browser closes (makes them session cookies)
CSRF_COOKIE_AGE
= NoneSESSION_COOKIE_AGE
= NoneSESSION_EXPIRE_AT_BROWSER_CLOSE
= TrueWe also want to prevent the CSRF cookie from being sent in cross-browser requests. E.g. a user clicks on a link to our site, from another site - in this case we don't want any previous CSRF cookie already in their browser sent. It's unlikely to be an issue in our app since there are no user logins etc. but for consistency with our other cookies (session and language).
CSRF_COOKIE_SAMESITE
= "Strict"Now that https://docs.calitp.org/eligibility-server/specification/ is live, we can remove the duplicate docs here and instead point to that site.
Update:
Remove:
.vscode/launch.json
server.json
entry in .gitignore
.pre-commit-config.yaml
runServices
entry in .devcontainer.json
localhost/data/server.json
localhost/keys/server.key
(we still need the public key)Edit: we actually want to keep the runServices
entry, since we want the server starting up for the devcontainer, by default.
HTTP Strict Transport Security or HSTS:
For sites that should only be accessed over HTTPS, you can instruct modern browsers to refuse to connect to your domain name via an insecure connection (for a given period of time) by setting the “Strict-Transport-Security” header. This reduces your exposure to some SSL-stripping man-in-the-middle (MITM) attacks.
From https://docs.djangoproject.com/en/3.2/ref/middleware/#http-strict-transport-security. We may want/need to do this at the Nginx level instead, see https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
Crucial to this decision is the understanding that the AWS Load Balancer Azure App Service handles SSL connections for this app, and not the Django/Nginx system; the domain's certs and HTTP -> HTTPS redirection is all managed at the AWS Load Balancer App Service level.
See for example this app's nginx.conf in which the Nginx server is listening on port 8000 for connections - this is because the Load Balancer sits between the user and Nginx, handling the SSL and translation into the app container. In which case, trying to set this from within the app container may cause more problems than it solves.
With #152 we now publish each environment image (dev
, test
, prod
) to GitHub Container Registry.
This needs to be documented, with some small usage notes, likely on the Deployment page
This spec: https://github.com/cal-itp/benefits/pull/170/files#diff-1637a357180023d868d2fca69d3d4e2a68f7b58c61aa71b016977ca95ee672fcR36 is currently set to skip()
because it does not pass with the way the app is currently set up.
Previously, the Eligibility Server API had fake merchant auth token routes to accept calls from the abc
transit agency in development mode, but these have since been removed - as of this PR https://github.com/cal-itp/eligibility-server/pull/31/files#diff-e5991fd559847cf92df779bd8b72643e9b64926355be788735b3c1cfcabca211L219
Now, developers cannot get the eligibility/confirm
page to load with valid test credentials b/c the app looks for valid tokens before it loads the page. There are several ways we can address this:
eligibility/confirm
page without payment buttons. (This addresses the Cypress issue but not the local dev one).The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.
This line in localhost/docker-compose.yml
: https://github.com/cal-itp/benefits/blob/dev/.devcontainer/compose.yml#L56 does not function as we'd assumed it would. Docker Compose is not checking to make sure that the image is using the latest main
tag from here: https://github.com/cal-itp/eligibility-server/pkgs/container/eligibility-server
server:
image: ghcr.io/cal-itp/eligibility-server:main
Locally, I had to run docker pull ghcr.io/cal-itp/eligibility-server:main
before running benefits
in the Dev Container to get the latest main
image. I was running a main
image from several updates ago, and Docker Compose wasn't updating it.
We may have to update the docker-compose.yml
file with a specific versioned tag number or the specific SHA from Eligibility Server.
This is only an issue in a specific kind of environment; for example if this app were deployed on a public terminal (e.g. at a library or a transit agency customer service center).
Any time a user interacts with the Eligibility form, they could potentially see entries from other users who previously interacted with the form, whether or not those prior interactions were successful validations.
We can attempt to turn off autocomplete on form input elements, if this is deemed a big enough risk.
However, it should be noted that this is not as straightforward as it may seem, see for example these SO threads:
It sounds like Chrome in particular does a lot of work to undo choices web developers make and turn autocomplete back on.
I thought this was fixed as part of #86 when we implemented a specific Django route/view to respond to /healthcheck
requests.
But apparently not, because Amplitude is full of ELB healthcheck events:
Drilling in to any of these shows similar details:
Created this ticket after #187 added Prettier linting for CSS and JS/JSON.
(Or find another tool for linting Django-flavored HTML)
#122 introduced an initial Integration/End-to-End test setup, but the only way to run tests right now is on localhost by hand.
We'd like to instead run the tests on each PR as a merge check in GitHub Actions. Similar to the work over in cal-itp/eligibility-server#16.
Basic outline is:
on
PR to any branchCYPRESS_baseUrl
and other needed environmentAcceptance Criteria
Resources
As a rider with a MST Courtesy Card, I want the option to sign up with a MST Courtesy Card so I can enroll in the contactless fare-discount demonstration.
Acceptance Criteria
Resources
current dev has old Cal-ITP Logo as favicon.
If a user knows the /enrollment
endpoint, and has configured their session with a Transit Agency, they can navigate to the credit card enrollment page and completely bypass the EV check. This is a serious bug!
This page must ensure the user has previously verified eligibility criteria
We build a Docker image and push it to AWS for use in the production app, but this sits inside a private ECR repository.
It could be useful to have the image on a public repo as well. Based on the work from cal-itp/eligibility-server#22, we can add an additional docker/build-push-action
step to our dev
deployment that pushes into the GitHub Container Registry:
- name: Build and push main
uses: docker/build-push-action@v2
if: ${{ github.event_name != 'release' }}
with:
push: true
tags: ghcr.io/${{github.repository}}:<branch>
We can do this for each of our 3 environments to always have the current image running in each environment, available in GHCR as well.
Since #135 landed, additional docker/build-push-action
steps should go very fast.
The devcontainer is successfully set up.
An error is thrown when trying to run the postAttachCommand
, specifically when we're trying to install Cypress.
benefits_client:latest
benefits
folder as a remote container in VS CodeFrom console logging:
Running the postAttachCommand from devcontainer.json...
[117148 ms] Start: Run in container: /bin/bash localhost/bin/init.sh
[117388 ms] Port forwarding connection from 59748 > 37657 > 37657 in the container.
[117389 ms] Start: Run in container: /root/.vscode-server/bin/6cba118ac49a1b88332f312a8f67186f7f3c1643/node -e
pre-commit installed at .git/hooks/pre-commit
[WARNING] The 'rev' field of repo 'https://github.com/compilerla/conventional-pre-commit' appears to be a mutable reference (moving tag / branch). Mutable references are never updated after first install and are not supported. See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. Hint: `pre-commit autoupdate` often fixes this.
[INFO] Initializing environment for https://github.com/compilerla/conventional-pre-commit.
...
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[119288 ms] [15:54:45] Extracted extension to /root/.vscode-server/extensions/.c2d52185-63c1-4349-ac91-40616d2fdbc8: ms-python.python
[119306 ms] [15:54:45] Renamed to /root/.vscode-server/extensions/ms-python.python-2021.10.1365161279
[119310 ms] [15:54:45] Extracting completed. ms-python.python
[119311 ms] [15:54:45] Extension installed successfully: ms-toolsai.jupyter-keymap
[119311 ms] [15:54:45] Extension installed successfully: ms-toolsai.jupyter-renderers
[119311 ms] [15:54:45] Extension installed successfully: ms-toolsai.jupyter
[15:54:45] Extension installed successfully: ms-python.python
[119316 ms] Extension 'ms-python.python' v2021.10.1365161279 was successfully installed.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://gitlab.com/pycqa/flake8.
[120807 ms] [15:54:46] Extracted extension to /root/.vscode-server/extensions/.1a0abef0-8ca2-4923-8aa5-ec1605a86354: ms-python.vscode-pylance
[120821 ms] [15:54:46] Renamed to /root/.vscode-server/extensions/ms-python.vscode-pylance-2021.10.3
[120823 ms] [15:54:46] Extracting completed. ms-python.vscode-pylance
[120823 ms] [15:54:46] Extension installed successfully: ms-python.vscode-pylance
[120826 ms] Extension 'ms-python.vscode-pylance' v2021.10.3 was successfully installed.
[121800 ms] Extensions cache, remote removals: None
[INFO] Initializing environment for https://github.com/pycqa/bandit.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://gitlab.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/bandit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
pre-commit installed at .git/hooks/commit-msg
up to date, audited 171 packages in 765ms
25 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Cypress cannot write to the cache directory due to file permissions
See discussion and possible solutions at
https://github.com/cypress-io/cypress/issues/1281
----------
Failed to access /root/.cache/Cypress:
EACCES: permission denied, mkdir '/root/.cache/Cypress'
----------
Platform: linux (Debian - 11)
Cypress Version: 8.7.0
@thekaveman and I speculate it has something to do with the "quirks with local filesystem (bind) mounts" with Docker CE/EE on Linux (see VS Code docs on non-root users). Windows users can see everything in the Linux container because there isn't a way to map the permissions. This comment may have some helpful details too.
There are other GitHub issues showing this error:
We should implement best-practices for building Docker images in GitHub Actions.
These include:
build-push-action
See more: https://docs.docker.com/ci-cd/github-actions/#optimizing-the-workflow
The EligibilityType
model contains two properties used by different phases of the system:
EligibilityType.name
is used in the eligibility phase as part of the API call to verifyEligibilityType.group_id
is used in the enrollment phase as the group to enroll eligible users intoHowever, as noted and fixed in #89, we may have multiple EligiblityType
with the same name
and different group_id
. For example, a senior
type for each transit agency, each with a different group_id
.
The fix in #89 was somewhat of a hack, as these concerns could (should?) be split out to allow reusing EligibilityType
instances. Thinking of a refactor along the lines of:
Before
class EligibilityType(models.Model):
"""Currently used for both eligibility and enrollment"""
name = models.TextField()
label = models.TextField()
group_id = models.TextField()
class TransitAgency(models.Model):
# other fields ...
eligibility_types = models.ManyToManyField(EligibilityType)
After
class EligibilityType(models.Model):
"""Used for eligibility checks - reused across agencies"""
name = models.TextField()
label = models.TextField()
class BenefitGroup(models.Model):
"""Used for enrollment - at least one per agency"""
name = models.TextField()
group_id = models.TextField()
eligibility_type = models.ForeignKey(EligibilityType, on_delete=models.PROTECT)
class TransitAgency(models.Model):
# other fields ...
benefit_groups = models.ManyToManyField(BenefitGroup)
As a rider with a courtesy card that is not valid, I want to see a useful error message so that I can be informed of next steps.
Acceptance criteria
Resources
Since we're getting close to being live, now is a good time to implement a versioning and release strategy. This will help us talk about development and the application itself in more real terms like "These features are part of version XYZ".
prod
branch, via a GitHub Actions workflow.A very rough outline of a process could look like:
prod
to kick off a deploy to AWS, like nowreecetech/version-increment
softprops/action-gh-release
(separate workflow runs when new tag shows up in GitHub)v
prefix in the version numbers? Our consensus was no, and reecetech/version-increment
doesn't support this anyway. See reecetech/version-increment#8 for more.MAJOR.MINOR.PATCH
, or alternatively, "calver" scheme YEAR.MONTH.RELEASE
where RELEASE
is the Nth release that month. Both are supported by reecetech/version-increment
.For versioning scheme, I kind of like calver for this application, given the other reporting requirements etc. that are monthly or month-based.
Acceptance Criteria
Resources
Need a basic client-side check that cookies are enabled (and UI if not) as soon as the app loads, since we require cookies to store the temporary transaction data.
Hardcoding localhost ports to map into Docker containers is unnecessary maintenance overhead, especially when working on multiple projects each with a number of containers/services and trying to choose ports that allow all services to run simultaneously.
Ideally Docker would just pick an available port during container startup and use that. We'd like to get away from managing these values / environment variables:
DJANGO_LOCAL_PORT
DOCS_LOCAL_PORT
5678
hardcoded as the debug port for dev
The following values will be moving from this project into eligibility-server
as part of the work in #121, so we'll want to apply the same approach there as well:
5000
hardcoded as the server
port5789
hardcoded as the debug port for server
We'll need to consider:
docker compose ...
commandsThis SO post is a good starting place: https://stackoverflow.com/a/56447023/453168.
Create a new model class:
class PemData(models.Model):
"""An API certificate or key in PEM format."""
id = models.AutoField(primary_key=True)
text = models.TextField()
Run manage.py makemigrations
to generate migration file, copy the new defs into 0001_initial
then delete the migration file
Refactor EligibilityVerifier
, PaymentProcessor
, TransitAgency
to reference the new model
Refactor localhost/data/client.json
to the new model
Refactor other consumers of model class as needed
Now that we have more JavaScript in the way of Cypress tests, we should introduce some linting and formatting to the devcontainer settings and pre-commit.
The X-XSS-Protection header can be used to manage certain browser's protection against reflected cross-site scripting (XSS), stopping a page from being loaded if an attack is detected. In modern browsers, the Content-Security-Policy header can provide better protection against XSS and setting X-XSS-Protection might be redundant (#203 tracks CSP implementation).
See more at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection
We'll want the following header/value:
X-XSS-Protection: 1; mode=block
This can be done in a new Middleware and configured in settings.py
for all requests/responses.
.github/workflows/ecs-deploy-prod.yml
based on existing files for dev
and test
.
prod
environment
AWS_ACCESS_KEY_ID
AWS_ACCOUNT
AWS_BUCKET
AWS_SECRET_ACCESS_KEY
.env
and config.json
files to bucket
ALLOWED_HOSTS
aws-cli
image to ECRdev
-> test
-> prod
to initiate the deployTransitAgency
modelThe bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.
@thekaveman https://docs.calitp.org/benefits/getting-started/development/ - should update this doc: https://github.com/cal-itp/benefits/blob/dev/docs/getting-started/development.md
Originally posted by @machikoyasuda in #116 (comment)
Our discount provider's API includes a field customer_ref
on the customer object, which can be set one time to a unique value (uniqueness as defined by the API caller, not the discount provider). This value is visible in the provider's Merchant Portal and serves as a way to correlate the customer with an outside record.
benefits
, so there is no way to track uniquenessbenefits
does not ask for any data beyond the basics necessary for eligibility verification, so we have no meaningful values to fill in here (e.g. email address)uuid.uuid4()
I lean towards 1 since we don't really have a need for this right now, and I don't believe it is a required field in the API.
We currently pin conventional-pre-commit
to v1.0.0
There have been some recent syntax and support improvements on that hook. A new tagging/versioning mechanism means v1
will always point to the latest v1.x.x
tag.
Let's pin to v1
so we always get the latest.
We have benefits
and the eligibility-server
each with Python code that implements one half of the Eligibility Verification API.
We could consolidate these two disparate halves into a single Python package that would:
Note that last point is particularly relevant as we are now in the process of upgrading dependencies in the server to match recently posted updates to the same dependencies here (#131, #132).
The corollary tracking issue in server is cal-itp/eligibility-server#38
benefits/eligibility/api.py
is where the client-side implementation lives.
RequestToken
and ResponseToken
are almost entirely independent from any Django-based context or references. Where they do make use of Django objects (e.g. a TransitAgency
model is passed in) we could instead pass the needed data (e.g. the agency's ID), directly.
Client
is only slightly more tied into the Django-specifics and could be easily refactored as well.
The Content-Security-Policy (CSP) header (with the frame-ancestors
directive) replaces the now deprecated X-Frame-Options
header, to instruct the browser about appropriate actions to perform if a site is included inside an <iframe>
.
See more at https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
We already have Django's built-in Clickjacking/X-Frame-Options
features enabled. Since this app should never be run from an <iframe>
, let's create another Middleware that sets the CSP header like so:
Content-Security-Policy: default-src 'self'; frame-ancestors 'none';
The localhost/server
directory contains a simple Flask app implementing a very basic Eligibility Verification API server. This server can be used for testing benefits
with only local resources via the service defined in localhost/docker-compose.yml
. Read more about running the test verification server.
cal-itp/eligibility-server
(suggestions welcome)localhost/server
project therepre-commit
, mkdocs
, etc.)There are a number of advantages in having a standalone server project. A more robust implementation can help us design better testing scenarios for the client (benefits
), while creating a path towards expanding into new eligibilities that may require custom verification logic. Separate projects can move at different speeds and respond to needs more nimbly.
The initial goals would be:
benefits
in a local development context for testing (e.g. via a public Docker image)During the transition, we don't need any major changes or features added for the server; the primary goal is to split the two projects apart while maintaining their current compatibility.
The status quo, where the server lives side-by-side in this same repository, is not all that bad. It simplifies local usage of the test server, and keeps a tight relationship between the client and server tokenization and encryption code. We could alternatively refactor the server into a top-level directory and work on it as a more monorepo concept.
Corollary to #109.
We want to prevent the dev
deployment from occurring for changes that only touch the docs/ directory.
The following can be configured in settings.py
LANGUAGE_COOKIE_HTTPONLY
= True (same as SESSION_COOKIE_HTTPONLY
)LANGUAGE_COOKIE_SAMESITE
= "Strict" (same as SESSION_COOKIE_SAMESITE
)LANGUAGE_COOKIE_SECURE
= True (same as SESSION_COOKIE_SECURE
)Update all our docs that were pointed at Server's docs as part of #137 to point to the new home on API: https://docs.calitp.org/eligibility-api/specification
GitHub Action:
benefits/**
filesWe aren't using main
as the default branch. Let's rename it to prod
to more closely align with the deployment environment it corresponds with.
Let's also rename the GitHub environment to prod
to match.
dev
.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.