cal-itp / littlepay

Cal-ITP API implementations and admin tasks for Littlepay.

Cal-ITP Littlepay

Cal-ITP API implementations and admin tasks for Littlepay.

Usage

$ littlepay -h
usage: littlepay [-h] [-v] [-c CONFIG_PATH] {config,groups,products,switch} ...

positional arguments:
  {config,groups,products,switch}
    config              Get or set configuration
    groups              Interact with groups in the active environment
    products            Interact with products in the active environment
    switch              Switch the active environment or participant

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -c CONFIG_PATH, --config CONFIG_PATH
                        Path to a readable and writeable config file to use.
                        File will be created if it does not exist.

Install

Use pip to install from GitHub:

pip install git+https://github.com/cal-itp/littlepay.git@main

Getting started

If this is your first time using littlepay, create a configuration file (using defaults):

littlepay config

The location and basic info from your current config file are printed in the terminal:

$ littlepay config
Creating config file: /home/calitp/.littlepay/config.yaml
Config: /home/calitp/.littlepay/config.yaml
Envs: prod, qa
Participants: cst
Active: qa, [no participant]

Add configuration

The configuration file is a YAML dictionary tracking envs and participants:

active:
  env: qa
  participant: ""
envs:
  prod:
    url: ""
  qa:
    url: ""
participants:
  cst:
    qa:
      audience: ""
      client_id: ""
      client_secret: ""
    prod:
      audience: ""
      client_id: ""
      client_secret: ""

There are two envs by default, the base API URL should be completed for each:

• qa is for the testing system
• prod is for the live system

Add specifics for the participants you manage based on information received from Littlepay support.

Use a different config file

littlepay config /path/to/new/config.yaml

Or

littlepay --config /path/to/new/config.yaml

The most recent config used is saved for next time.

Switch the active target

For envs:

littlepay switch env <env_name>

And participants:

littlepay switch participant <participant_id>

Work with groups

$ littlepay groups -h
usage: littlepay groups [-h] [-f GROUP_TERMS] [--csv] {create,funding_sources,link,migrate,products,remove,unlink} ...

positional arguments:
  {create,funding_sources,link,migrate,products,remove,unlink}
    create              Create a new concession group
    funding_sources     List funding sources for one or more concession groups
    link                Link one or more concession groups to a product
    migrate             Migrate a group from the old Customer Group format to the current format
    products            List products for one or more concession groups
    remove              Remove an existing concession group
    unlink              Unlink a product from one or more concession groups

options:
  -h, --help            show this help message and exit
  -f GROUP_TERMS, --filter GROUP_TERMS
                        Filter for groups with matching group ID or label
  --csv                 Output results in simple CSV format

List existing groups

Print all groups in the active environment:

littlepay groups

Filter for groups with a matching ID or label:

littlepay groups -f <term>

Multiple filters are OR'd together:

littlepay groups -f <term1> -f <term2>

Create a new group

littlepay groups create <label>

Delete an existing group

With confirmation:

littlepay groups remove <group_id>

Without confirmation:

littlepay groups remove --force <group_id>

Work with products

$ littlepay products -h
usage: littlepay products [-h] [-f PRODUCT_TERMS] [-s {ACTIVE,INACTIVE,EXPIRED}] [--csv] {link,unlink} ...

positional arguments:
  {link,unlink}
    link                Link one or more products to a concession group
    unlink              Unlink a concession group from one or more products

options:
  -h, --help            show this help message and exit
  -f PRODUCT_TERMS, --filter PRODUCT_TERMS
                        Filter for products with matching product ID, code, or description
  -s {ACTIVE,INACTIVE,EXPIRED}, --status {ACTIVE,INACTIVE,EXPIRED}
                        Filter for products with matching status
  --csv                 Output results in simple CSV format

List existing products

littlepay products

Filtering works the same as for groups, matching against product ID, code, or description:

littlepay products -f <term>

littlepay products -f <term1> -f <term2>

Also supports filtering by status (ACTIVE, INACTIVE, EXPIRED):

littlepay products -s EXPIRED

littlepay products -f <term> -s ACTIVE

List linked products for one or more groups

For each group, output the group's linked products. Builds on the filtering sytax.

E.g. to list linked products for all groups:

littlepay groups products

Or to list linked products for a specific group:

littlepay groups -f <group_id> products

Link and unlink a product to one or more groups

For each group, link the given product to the group. Builds on the filtering syntax.

E.g. to link a product to all groups:

littlepay groups link <product_id>

Or to link a product to a specific group:

littlepay groups -f <group_id> link <product_id>

Unlinking groups from a product works the same:

littlepay groups -f <group_id> unlink <product_id>

Link and unlink a group to one or more products

For each product, link the given group to the product. Builds on the filtering syntax.

E.g. to link a group to all products:

littlepay products link <group_id>

Or to link a group to a specific product:

littlepay products -f <product_id> link <group_id>

Unlinking products from a group works the same:

littlepay products -f <product_id> unlink <group_id>

Version and release

The package version is derived from git metadata via setuptools_scm.

Pushing a tag with the correct format generates a new GitHub release, which can then be finalized and published.

Release a new version

1. Ensure you are on the latest commit of main:

   git checkout main
   git pull

2. Create an annotated, Calver-formatted tag based on the release year, month, and sequence counter:

   git tag -a YYYY.MM.N

   You may also create a release candidate, by appending -rcX where X is the release candidate sequence counter:

   git tag -a YYYY.MM.N-rcX

   In either case, provide a short comment for the tag.

3. Push the tag to GitHub:

   git push origin YYYY.MM.N

4. Observe the Release workflow

littlepay's Issues

Implement API endpoint for group migration

This is related to the strange behavior and empty objects {} we're seeing in the get_concession_group_linked_funding_sources() response.

From Littlepay:

This issue is historical from concessions being created through the Merchant Portal; there is a correction to the empty objects being returned in the pipeline but we have a method of migrating Concession Groups that will ensure compatibility; the details of which are listed here: https://docs.littlepay.io/reference/backoffice/tag/Products/#tag/Products/operation/MigrateConcessionGroup

Acceptance criteria

• The GroupsMixin implements a migrate_concession_group() or similar function with the spec defined in the above linked endpoint docs

Update README docs with new sub-commands

Follow up to #41 and #47, let's update the README with information about the new groups sub-commands:

• funding_sources
• migrate

Acceptance criteria

• The new sub-commands are listed in the help output under the Work with groups heading
• A new (brief) section for each sub-command shows its usage, following the existing sections/examples

Command to migrate a concession group

Building on the API from #46 and the existing groups command (i.e. take into account filtering).

Usage

$ littlepay groups [-f GROUP_TERMS] migrate

Acceptance criteria

• A new sub command for groups is defined called migrate (or similar)
• The new sub command builds on the groups filter, migrating group(s) that match the given filter if any
• The new sub command uses the API endpoint implemented in #46 to do the migration
• Tests are written to cover the new sub command

Don't return RESULT_FAILURE when no active env is configured

Follow up and similar to #20

We want to apply the same logic for the active.env configuration -- i.e. not having an active env should display a warning in the console output, but not return an error response. Instead return RESULT_SUCCESS.

Command to list funding sources currently linked to a group

Building on the API from #28 and the existing groups command (i.e. take into account filtering).

Usage

$ littlepay groups [-f GROUP_TERMS] funding_sources

Acceptance criteria

• A new sub command for groups is defined called funding_sources (or similar)
• The new sub command lists the funding sources currently linked to one or more groups
• The new sub command builds on the groups filter, listing funding sources for group(s) that match the given filter if any
• The new sub command uses the get_concession_group_linked_funding_sources() API endpoint implementation to get its data
• The new sub command prints the resulting GroupFundingSourceResponse to stdout (don't worry about csv output)
• Tests are written to cover the new sub command

Background

We define CLI commands in the littlepay/commands subdirectory: https://github.com/cal-itp/littlepay/tree/main/littlepay/commands

Each command has its own module/file in that directory:

• configure
• groups
• products
• switch

The littlepay/main.py module/file sets up the CLI argument parsing and is the entrypoint for the CLI.

This is blocked by the fix for #42, which should be done first. #42 is done

Avoid errors when responses contain additional fields

A pattern in this library is to take JSON returned from the API and use it to hydrate response objects, e.g. FundingSourceResponse, GroupFundingSourceResponse, etc. See _get, _post, and _put.

If the JSON contains fields that we didn't define in our response class, currently we'll get a TypeError. For example, see cal-itp/benefits#2365.

It would be nice to avoid throwing an error.

Raise `ValueError` in `Config.active_participant` if there is no active participant.

I think an improvement to the CLI experience could be to raise a ValueError in Config.active_participant if there is no active participant.

I don't think it needs to block this PR though, so it could be done as a follow-up if at all

Originally posted by @angela-tran in #20 (review)

Extract functions from main command logic in `groups` and `products` to improve readability

As a part of troubleshooting / discussing the issue documented in #50, we decided it could be helpful to extract out functions from the main line of code in littlepay.commands.groups and littlepay.commands.products, with the "main line" being the groups function and products function respectively.

Mark a new release

With all the work from #10, we should have a new version tagged/released.

Can be saved for the same time that #27 is implemented to aid in testing that workflow.

Refactor API paging parameter name

In discussion with Littlepay over the issues encountered during #53, we learned that at least for some endpoints, the paging parameter is called per_page instead of perPage as documented.

Here are the endpoints we implement that use paging, and the current parameter name in the docs:

Function	Paging param in docs	Additional notes
get_concession_groups	per_page
get_concession_group_linked_funding_sources	perPage	should be per_page according to this thread with Littlepay
get_concession_group_products	per_page
get_products	perPage	no change
#60 (get list of concession groups linked to a funding source)	per_page	Done in #62

Acceptance criteria

• Endpoints above that use per_page are updated in the code

Implement endpoint to get a list of groups for a funding source

This is the inverse of the endpoint implemented for #28. Having this endpoint will theoretically make the enrollment check in Benefits easier, since we'll be able to get a (smaller) list of groups per funding source, rather than looking through the (potentially much larger) list of all funding sources for a group.

Implement GET endpoint for groups linked to a funding source.

The funding source ID is a URL path parameter.

Returns a JSON body response like:

{
  "list": [
    {
      "id": "2009b586-9c53-43c6-b274-5615adb856ee",
      "group_id": "35fe8fe8-a69a-4fa6-816c-553463d29777",
      "label": "Test Capping 1",
      "created_date": "2021-11-22T23:03:03Z",
      "updated_date": "2021-11-22T23:03:03Z",
      "expiry_date": "2021-11-22T23:03Z"
    },
    ....
  ],
  "total_count": 10
}

Git scrape Product and Group information for all agencies

We would like to understand changes over time to the Littlepay Product and Group details for each agency we work with. This has come up recently as we learned about a change to a Product ID after that Product was duplicated to create a new Product -- we didn't expect this to create a TWO new Products 🤷

So we'd like to capture the output of our littlepay groups and littlepay products CLI commands, in CSV files stored in this repository, to be able to assess those changes over time more easily.

Acceptance Criteria

• For each agency in production (MST, SacRT, SBMTD as of opening this issue)
• On some schedule (at least once per day)
• Run littlepay groups and littlepay products
• Store the data in a corresponding CSV file in this repository: data/[agency slug]_[groups|products].csv
• With column headers matching the fields in the corresponding GroupResponse and ProductResponse objects
• Also store the output of littlepay groups products in a CSV file in this repository: data/[agency slug]_linked_groups_products.csv
• With column headers group_id, product_id
• A new commit is made each time this process runs, for each agency, when any data file is changed

Additional context

Sort of related to the work in cal-itp/benefits#1889, can be implemented in workflow in a similar fashion.

More about the git scraping technique.

The version flag should not involve a config file

From work on #31:

The last problem to overcome was that littlepay -v was not only printing the version, but also creating a new config file since this workflow doesn't point it at existing file. This caused the output of littlepay -v to be something like this:

Creating config file: 
/home/runner/.littlepay/config.yaml 
0.1.dev1g9bfec47 

which leads to the error message about a multi-line output variable not being set correctly.

I resolved this by just calling littlepay before doing anything else.

Originally posted by @angela-tran in #31 (comment)

Just printing the version on the CLI should not involve a configuration file, instead short-cutting to print the version directly.

Implement GET funding sources linked to a group

Implement GET endpoint for funding sources currently linked to a group.

The group ID is a URL path parameter.

Returns a JSON body response like:

{
  "list": [
    {
      "id": "9002b586-9c53-43c6-b274-5615adb856ee",
      "participant_id": "test_participant",
      "concession_expiry": "2024-07-03T00:00:00Z",
      "concession_created_at": "2024-07-03T00:00:00Z",
      "concession_updated_at": "2024-07-03T00:00:00Z"
    },
    {
      "id": "6003a697-1d64-54d7-a385-4726bec767ff",
      "participant_id": "test_participant",
      "concession_expiry": null,
      "concession_created_at": null,
      "concession_updated_at": null
    }
  ],
  "total_count": 2
}

Note the date fields concession_* in the list portion could be null if/when the linked funding source has no expiry information associated with its linkage.

Add optional expiry_date parameter to link_concession_group_funding_source

Follow up to work in #15.

Final requirements to be specified after final design from Littlepay is approved.

Requirements as they stand now

• Optional body parameter is called concession_expiry
• ISO 8601 formatted datetime in UTC i.e. YYYY-MM-DDTHH:mm:ssZ

Prepare for publishing

• Add CODEOWNERS for @cal-itp/benefits-admin, require 1 approving review for merges to main
• Rename package to calitp-littlepay (in pyproject.toml)
• Version maintained as string in pyproject.toml; use CalVer (read by littlepay/__init__.py for use inside the codebase).
• Create GH Actions environment for release workflow
• Auto GH Release on tag

Publish releases to PyPI

See #11 for prior work on this.

We'll want this to work similarly to the release process for hashfields and that workflow can be used as a template/basis for this one.

This work will be important for the Low-income rider enrollment pathway in Benefits, since we want Benefits to be able to reference a stable, published version of this library.

Acceptance criteria

• The PyPI publish happens in response to pushing a correctly formatted tag
• For -rcX tags, publish to TestPyPI
• For non -rcX tags (e.g. actual version tags), publish to PyPI
• The package name in both TestPyPI and PyPI is calitp-littlepay
• The package is published to both TestPyPI and PyPI using a trusted publisher

Implement Customer enrollment into Concession Group

Overview

This is implemented today using the Customer API in Benefits here: https://github.com/cal-itp/benefits/blob/dev/benefits/enrollment/api.py

Docs for the Back Office API are here: https://docs.littlepay.io/back-office/littlepay-hosted-card-tokenisation/

Todo

• Acquire access token for card tokenization #13
• Use the card token to get the funding source #14
• Link the funding source to the concession group #15

Fix `GroupsMixin` methods to match API documentation

The expected request path and body for including expiration as described by the API documentation are slightly different from the notes that we were referencing.

Acceptance criteria

• The implementation matches API documentation for:
  • Linking
  • Updating

Current config path file is not created with custom config_path arg

See the recent failures in benefits for a reproduction: https://github.com/cal-itp/benefits/actions/runs/8390023580

In a fresh environment, when calling littlepay config config_path.yaml

The code does not go through the pathway of ensuring the .current file (defaulting to ~/.littlepay/.current) exists.

This was previously hidden by the erroneous top-level config = Config() in e.g. littlepay.commands.groups, which was always being executed first.

Without arguments, Config() goes through the _get_current_path() helper, which ensures the .current file exists

Expected behavior

Calling littlepay config config_path.yaml in a fresh environment should ensure the .current file exists in the default location ~/.littlepay.

There should not be an error and .current should be written to point to the provided config_path.yaml.

Migrate QA and production groups

After #41 and #47 are complete, we'll want to carefully migrate the existing groups and check our work along the way to ensure no breakages.

Todo QA

MST

• List funding sources for groups: b4191ff9-0103-474e-b8b6-0028d712bf1c, 723baf7a-91d6-4ba3-bfc5-e73d44f63cbe, 4e84b9bd-22ca-42e9-a0b5-27c5d2f3f634
• Migrate the same groups
• List funding sources for the same groups, compare
• Smoke test enrollment to these groups in dev or test Benefits

SacRT

• List funding sources for group: d9b41fcd-eb89-41b1-a492-e7dc1598334f
• Migrate the same group
• List funding sources for the same group, compare
• Smoke test enrollment to this group in dev Benefits

SBMTD

• List funding sources for groups: 31f35561-38d8-4b41-90f1-b74518d4854b, 3ac376c8-3939-4484-b3a2-87c9dfbfc68e
• Migrate the same groups
• List funding sources for the same groups, compare
• Smoke test enrollment to these groups in dev or test Benefits

Todo Prod

MST

• List funding sources for groups: e919852a-5e47-478d-b4bc-7362ca3706e6, 5170d37b-43d5-4049-899c-b4d850e14990, ab9982fb-c2f8-4116-9ffb-ced0ca20bf25
• Migrate the same groups
• List funding sources for the same groups, compare

SacRT

• List funding sources for groups: 80271c54-29f6-465c-ad33-0f6ef18a15be
• Migrate the same groups
• List funding sources for the same groups, compare
• Smoke test enrollment to this group in test Benefits

SBMTD

• List funding sources for groups: ce3d04bf-3c51-4f4c-9779-82e5a02dc73a, 46e95628-b4ad-4b36-b2b2-abb6b1acd10a
• Migrate the same groups
• List funding sources for the same groups, compare

Fix definition of `GroupFundingSourceResponse` to match API documentation

We're seeing that GroupFundingSourceResponse as it's currently defined does not match the structure of the actual response returned when getting a list of funding sources linked to a concession group. This is from reading the API documentation as well as from testing against the live QA environment.

Acceptance criteria

• GroupFundingSourceResponse matches API documentation

Commands that support group filtering output 0 as their matching groups

To Reproduce

1. Run littlepay groups to see what groups are available
2. Pick one of these commands: link, unlink, or migrate
3. Comment out its API call so that it doesn't actually run.
   • For example, for link, comment out this line
4. Run your command with any arguments it needs, e.g. littlepay groups link blah
5. Notice that the command output says there were 0 matching groups

Expected result

The output should show which groups were operated on.

Additional context

@angela-tran Great news! Do you know why it's printing the last line?

👥 Matching groups (0): qa, mst

Originally posted by @thekaveman in #49 (comment)

Implement PUT funding source expiry date endpoint

Supports updating an existing (i.e. already linked) funding source in a concession group, to provide the expiry date.

Works the same way / same parameters as #17

Response codes:

• 200 on success
• 400 if the request is badly formatted
• 404 if the group does not exist

Response body is similar to that in #28 (only the funding source information for the one that was updated is returned):

{
  "list": [
    {
      "id": "9002b586-9c53-43c6-b274-5615adb856ee",
      "participant_id": "test_participant",
      "concession_expiry": "2024-07-03T00:00:00Z",
      "concession_created_at": "2024-07-03T00:00:00Z",
      "concession_updated_at": "2024-07-03T00:00:00Z"
    }
  ],
  "total_count": 1
}