repository-service-tuf / repository-service-tuf-api

Repository Service for TUF: API

Repository Service for TUF (RSTUF)

Repository Service for TUF (RSTUF) is a collection of components that provide services for securing content downloads from tampering between the repository and the client (for example, by an on-path attacker).

RSTUF security properties are achieved by implementing The Update Framework (TUF) as a service.

Repository Service for TUF is platform, artifact, language, and process-flow agnostic.

RSTUF simplifies the adoption of TUF by removing the need to design a repository integration -- RSTUF encapsulates that design.

Repository Service for TUF (RSTUF) is designed to be integrated with existing content delivery solutions -- at the edge or in public/private clouds --alongside current artifact production systems, such as build systems, including; Jenkins, GitHub Actions, GitLab, CircleCI, etc. RSTUF protects downloading, installing, and updating content from arbitrary content repositories, such as a web server, JFrog Artifactory, GitHub packages, etc.

If a user wants to integrate RSTUF into an existing CI/CD pipeline the only requirement is to make a REST API request to RSTUF:

The same can be said when a user wants to integrate RSTUF into an existing distribution platform:

Thanks to the REST API, integrating RSTUF into existing content delivery solutions is straightforward. Furthermore, RSTUF is designed for scalability and can support active repositories with multiple repository workers.

At present, RSTUF implements a streamlined variant of the Python Package Index (PyPI)'s PEP 458 – Secure PyPI downloads with signed repository metadata. In the future, RSTUF will grow to provide additional protections through supporting the end-to-end signing of packages, comparable to PyPI's PEP 480 – Surviving a Compromise of PyPI: End-to-end signing of packages.

How does Repository Service for TUF compare to other solutions?

Rugged: Repository Service for TUF is a collection of services to deploy a scalable and distributed TUF Repository. RSTUF provides an easy interface to integrate (the REST API) and a tool for managing the Metadata Repository (CLI).

PyPI/PEP 458: Repository Service for TUF is a generalization of the design in PEP 458 that can be integrated into a variety of content repository architectures.

Using

Please, check the Repository Service for TUF Guide for the instructions about deployment, using and more details.

Contributing

This git repository contains high-level documentation guides and component integrations.

Check our CONTRIBUTING.rst for more details on how to contribute.

Please, check the Repository Service for TUF Development Guide.

Questions, feedback, and suggestions are welcomed on the #repository-service-for-tuf channel on OpenSSF Slack.

repository-service-tuf-api's Issues

Example shows expiration keys in string instead integers

Example shows expiration keys in string instead integers

The data_example/repository_settings/settings.json has different type.

Implementation for the endpoint that GET the TUF Metadata

It depends on #5

Implement the logic to fetch the Metadata from the Storage and show it as a JSON Response.

Add to development environment the kaprien-mq

Implement to the kaprien-rest-api development environment the kaprien-mq service to receive all tasks.

Add the docker-repo-worker to the Development mode

Add the docker-repo-worker to Development mode (make run-dev)

Remove python-tuf 2.0.0

TUF Library (python-tuf) can be removed from kaprien-rest-api

Depends on

• #55

Implement Key Storage AWS KMS

(To be refined) AWS Key Management Service

Refactoring names in the implementation for API Models and implementation/endpoint functions

Improve the names for the API Models and the functions names.

Response -> BootstrapResponse
get -> get_bootstrap

Implement Metadata Storage support for S3

(to be refined)

Add users and tokens

Implement a Users, Tokens and Scopes

Base URL as a settings

The information Base-URL needs to be required in the bootstrap.

The base-URL is a piece of information about where targets files will be downloaded by future client (python-tuf for example)

Example:
https://www.example.com/downloads/product_a/XXXX/product_a_installer_1.0.0.tar.gz
https://www.example.com/ is the base-URL
downloads/product_a/ is the delegation */*/* from Targets Role -> BIN Role
XXXX/product_a_installer_1.0.0.tar.gz will be the delegation from BIN Role to BINS Role followed by file name

Give bootstrap and add_targets to specific queue

Move both tasks bootstrap and add_targets to metadata_repository_write

Create the initial endpoint for bootstrap the Kaprien API Service

This is the initial version/structure for the Kaprien API Service

The basic initial API is the bootstrap. That will give the initialization of the service.
The bootstrap needs to receive as a payload the settings and the metadata and contain the full API documentation.

The minimum documentation is also necessary for operating the development.

settings
Needs to contain all information about the roles

• number of keys
• threshold
• expiration
• keys (when online keys -- bins, snapshot, and timestamp), credentials
• paths (delegated to bin)
• number_hash_prefixes (delegated from bin to bins)

metadata
Needs to contain all metadata data using python-tuf Metadata in JSON (dictionary format)
role name containing the version (no version means 1)
All versions after the initial delegation need to be available:
1. root
1.targets, 2.targets (delegating to bin)
1.bin, 2.bin(delegating to bins)
1.snapshot, 2.snapshot (delegation to bin) and 3.snapshot (delegation to bins)

Implement in the Repository return all Metadata

Implement in the kaprien_api.api.tuf.repository the mechanism that retrieves the Metadata from the repository.

Clean up the ``kaprien.tuf.repository``

Clean up the kaprien.tuf.repository
The idea is to remove the Metadata initialization once it is done always by the kaprien-cli
Reuse the TUF data-structure for the Delegation as well.

Unit tests for the Endpoint GET bootstrap

Implement the Unit Tests using the API (FastAPI methodology) to test the GET bootstrap Endpoint

Unit tests for the Endpoint POST bootstrap

Add Unit Tests, implement using the FastAPI methodology, to the endpoint POST bootstrap

Implement endpoint for add new targets

Implement the endpoint in the REST API to add new targets

(This issue does not cover the publisher part to send to kaprien-mq)

Add endpoint to validate the token

Check the token and return remaining hours

Add option to add task_id to the target custom fields

Add possibility to the target payload include the task_id in the Targets custom field

'payload': {
  'targets': '...',
  'add_task_id_to_custom': bool
}

It will have registered to the Metadata the task id responsible for adding the task.

Implement the API documentation to GET the Metadata

The issue is to create the API Documentation for the endpoint that will show the Metadata.
This issue does not cover implementation that retrieves the the Metadata from TUF Repository.

Include the classes for tests on `tests/api/test_metadata.py`

Include the classes to organize the tests in the tests/api/test_metadata.py

See test_bootstrap as reference

How to handle the if system is bootstrapped or not?

We use across many API endpoints the tuf_repository.is_initialized to say that the system is Bootstrapped but it is not safe.
In case, for example, storage (filesystem or SaaS) is unavailable we will say that the system is available to bootstrap.

It is not part of the MVP (as a MUST) but it will be a VERY GOOD TO HAVE.
The system needs to combine access to the Repository Metadata/Storage Service + if the security (#27)

It requires changes in all endpoints.

Move bootstrap action to kaprien-repo-worker

• Remove the metadata written from the kaprien-rest-api
• Send it to write queue in kaprien-mq

Requires:

• repository-service-tuf/repository-service-tuf-worker#13

Improve Bootstrap endpoints

• Return the Metadata in case of success
• Change the return code to 400 in case the system has already a metadata
• The error message can be more precise: "System already has a Repository Metadata in the Storage."

Add a description for the endpoint bootstrap

Add to the GET and POST bootstrap endpoints the descriptions
See endpoint metadata GET method as reference.

Add unit tests for ``kaprien_api/utils``

Implement Unit Tests for kaprien_api/utils.

Implement the publisher to the kaprien-mq

This implements the high-level publisher to the kaprien-mq.
The publisher will be used to add new tasks to message queue

Add kaprien-cli to the kaprien-rest-api container

Add kaprien-cli to the kaprien-rest-api container.

This will enable the admin to run operations from the container itself.

CI to build development Docker Image

Implement the CI to build the Docker Image as version development.
The version development is the latest version merged in the main branch.

Submit new targets to kaprien-mq

After receiving the payload, submit it to the kaprien-mq to be processed by kaprien-repo-worker

Add Unit Tests for ``kaprien_api/services``

Add Unit Tests for kaprien_api/services.

Incorrect usage for tempfile

Besides it does not create an issue to the tests, the curret usage is tempdir()

from tempfile import tempdir

https://github.com/KAPRIEN/kaprien-rest-api/blob/1f2cc2a0eab6313f2f2ad5ff02c9a09ac9df301e/tests/unit/test_app.py#L1

Add UT to the `kaprien_api/services`

Implement UT to the kaprien_api/service

Make the token unique

https://github.com/kairoaraujo/kaprien-rest-api/blob/88403c7cb978f4368636627c005cb6451faf2fd7/kaprien_api/token.py#L143

Add a unique id to make each token unique.

Add endpoint '/settings' to retrieve the settings

Implement the endpoint 'settings' with method GET to retrieve the Kaprien API settings

Implement UT to the services backends configuration

Implement UT to the services backend configurations

• STORAGE_BACKEND
• KEYVAULT_BACKEND

Revoke token

Implement a way to revoke a token (?)

Add endpoint to check if the ACCEPTED request (task) is done

Add endpoint to check if the ACCEPTED request (task) is FINISHED.
Enabling the possibility to create callbacks to the API end wait for an "important task" (ex: bootstrap)

Move the return code to 202 (Accepted)

https://github.com/KAPRIEN/kaprien-rest-api/blob/ab7a79f33ab3e5455af39d755bac91ca4c389e88/kaprien_api/api/targets.py#L12

This will be processed asynchronously and must be changed to HTTP 202 instead of 200.
https://restfulapi.net/http-status-202-accepted/

Also, include the task id in the response.

Add to README.md the requirements to run the project

Software requirement for development

• Docker
• docker-compose
• Python
• pip

Update the features. Add targets is done.

Current App name is FastAPI, rename to Kaprien API

Rename from swagger API Documentation http://127.0.0.0:8000 the Application name from FastAPI (default) to Kaprien API and add a description if possible.

Fix container database path

Define the database path in the Dockerfile

./database does not exist in /opt/kaprien-rest-api/database container build.

Standardize the responses

{
   "data": {"string": "Any"},
   "message": "string"
}

Fix README.md uppercase to lowercase for pip/pipenv command

$ Pip install pipenv should be $ pip install pipenv

Missing information in settings about number of hash bins

It is missing the information about the number of hash bins chosen by the user during the bootstrap.
Simple to save it in the settings (Dynaconf)

[Story] Implement a security authentication layer (i.e.: API Keys)

Design and document.

Implement UT for `kaprien_api/tuf`

[Why low priority] There is low coverage because it's not used yet by the existent API.

Implement UT for kaprien_api/tuf
If needed sub-issues can be created for it

• hash_bins.py low-hanging-fruit, just copy from kaprien-cli
• interfaces.py
• repository.py

Implement add targets

Implement database migrations

User database migrations.
Automatically add new scopes to the admin user.

Move tokens from days to hours

Token expiration time is per day, move it to hours.