sigstore / rekor

Software Supply Chain Transparency Log

Rekor

Rekór - Greek for “Record”

Rekor's goals are to provide an immutable tamper resistant ledger of metadata generated within a software projects supply chain. Rekor will enable software maintainers and build systems to record signed metadata to an immutable record. Other parties can then query said metadata to enable them to make informed decisions on trust and non-repudiation of an object's lifecycle. For more details visit the sigstore website.

The Rekor project provides a restful API based server for validation and a transparency log for storage. A CLI application is available to make and verify entries, query the transparency log for inclusion proof, integrity verification of the transparency log or retrieval of entries by either public key or artifact.

Rekor fulfils the signature transparency role of sigstore's software signing infrastructure. However, Rekor can be run on its own and is designed to be extensible to working with different manifest schemas and PKI tooling.

Official Documentation.

Public Instance

Rekor is officially Generally Available with a 1.0.0 release, and follows semver rules for API stability. This means production workloads can rely on the Rekor public instance, which has a 24/7 oncall rotation supporting it and offers a 99.5% availability SLO for the following API endpoints:

• /api/v1/log
• /api/v1/log/publicKey
• /api/v1/log/proof
• /api/v1/log/entries
• /api/v1/log/entries/retrieve

For uptime data on the Rekor public instance, see https://status.sigstore.dev.

More details on the public instance can be found at docs.sigstore.dev.

The attestation size limit for uploads to the public instance is 100KB. If you need to upload larger files, please run your own instance of Rekor. You can find instructions for doing so in the installation documentation.

Installation

Please see the installation page for details on how to install the rekor CLI and set up / run the rekor server

Usage

For examples of uploading signatures for all the supported types to rekor, see the types documentation.

Extensibility

Custom schemas / manifests (rekor type)

Rekor allows customized manifests (which term them as types), type customization is outlined here.

API

If you're interested in integration with Rekor, we have an OpenAPI swagger editor

Security

Should you discover any security issues, please refer to sigstore's security process

Contributions

We welcome contributions from anyone and are especially interested to hear from users of Rekor.

Additional Documentation

In addition to this README file, this folder contains the additional documentation:

• oid-info.md. Rekor OID values.
• types.md. Information about how to sign and upload data in different pluggable types.

rekor's Issues

api subdomain

apit.,rekor.dev > GKE server

Add direct server support for attached signature formats

cc @puiterwijk

This could include things like

• RPMs which bundle signatures into a special header
• Maven artifacts
• The Windows PE file format
• ELF xattrs

Document mirror setups

There might be a few already we can pull from in the Trillian examples. It could be as simple as a bash for loop that tails one log with rekor-cli loginfo and rekor-cli get --uuid, then uploads those entries to another Rekor server instance.

Ref #210

Finalise CLI commands and subproject roles.

hey @dlorenc , opening this issue for us to round up the approach for the CLI app and its role (and current sub commands / flags).

Currently as it stands:

% rekor-cli --help
  add         Rekor Add Command
  get         Rekor get command
  getleaf     Rekor Get Leaf Command
  help        Help about any command
  update      Rekor update command

We have discussed extending the following commands

rekor upload --signature=<signature-file> --public-key=<public-key-file> --artifact=<URL or file> # generates the rekord file (This would used by the node maintainer or the user in fact)
rekor verify --signature=<signature-file> (optional) --artifact <URL or file> # this would be run by the user. This would require the public key. 
rekor lookup-signatures-by-artifact --artifact=<URL or file> # It looks up all signature entries for this artifact and returns them. This requires a map[sha][]RekorLeaf
rekor lookup-signatures-by-public-key --public-key=<public-key>

I propose we do the following.

• rekor-cli changes its binary namespace to just rekor
• rekor then becomes the main tool that oss project maintainers use to record into the rekor server their release artifacts and verify entry.
• We remove (or refactor when possible) all of the commands: add, get, getleaf, update and we create a new tool called rekorctl as a home for these subcommands along side get signatures by artifact / public key we discussed and added recently.

This then makes the developer tool simple and we maintain this with stability as key (and a lower attack surface as a result of its simple implementation). The tool just uploads and verifies. Anyone wanting more of a swiss army knife style tool can use rekorctl which I will go into a little futher down (and will be home to the commands removed from the maintainers tool rekor).

rekor

Upload

% rekor upload --help

Upload a signed artifact into the rekor transparency log.

Usage:
  rekor upload [flags]

Flags:
  -h, --help             help for upload
      --signature string The signature generated during artifact signing
      --public-key string   The public key, of the GPG key pair used to sign the artifact
      --artifact-url string The url of where the artifact is stored
      --artifact-path string The file location of the artifact 
     --artifact-sha Mandatory if using `--artifact-url`

Verify

% rekor verify --help

Verify a signed entry exists within rekor

Usage:
  rekor upload [flags]

Flags:
  -h, --help             help for upload
      --signature string The signature generated during artifact signing (optional)
      --public-key string   The public key, of the GPG key pair used to sign the artifact
      --artifact-url string The url of where the artifact is stored
      --artifact-path string The file location of the artifact 

rekorctl

We then migrate / refactor the commands removed from rekor and the new commands to a binary namespace / project called rekorctl (or rekoradm for admin, I am open to naming). This will be a tool used by those wanting to query the rekor instance a bit deeper beyond just making and verifying entries like our project maintainers (like a said a swiss army knife). Possible audience could be server admins or security researchers / auditors.

%  rekorctl --help
  get 
  getleaf     Get Leafs
  sigs-by-artifact Look up all signature entries for this artifact and returns them.
  sigs-by-pubkey Look up signatures by public key
  help        Help about any command
  update      Rekor update command

% rekorctl getleaf --help

Get a leaf by either the index or by the artifact

Usage:
  rekor getleaf [flags]

Flags:
  -h, --help             help for upload
      --leafindex string The signature generated during artifact signing (optional)
      --artifact-url string The url of where the artifact is stored
      --artifact-path string The file location of the artifact 

This will then do the same as the current getleaf command and download the provenance file to the users drive using $<random_string>.txt

The server CLI will remain largely as it is, with the serve flag (although we could add some stuff over time like running in DEBUG mode etc).

Idea: Timestamping Authorities

Rekor sort of acts as a timestamp witness today. A rekor entry can be returned with the timestamp at which it was included in our log. This requires that the user trust rekor's clocktime, though.

RFC3161 outlines Timestamping authorities. There is a go client here: https://github.com/digitorus/timestamp

A timestamping authority is a service that uses X.509 certificates to sign a timestamp along with an opaque payload. This signature can be used as a third-party time attestation.

Meta Part: The timestamp attestation is signed. Rekor stores signatures. Rekor could store its own timestamp signatures, to act as a self-proof of time accuracy. From time to time, Rekor would generate a payload with it's own observed time and get it signed by one (or more) timestamp authorities.

The public key for this authority, the payload, and the signature form a Rekor entry, which could then be stored in Rekor itself.

incorrect CLI persisted log state error

There are scenarios for the CLI where the server queried may not be the same as where the log state information that persisted in the ~/.rekor/ directory was obtained from. If this is the case, the CLI prints an incorrect error state in comparing states.

Add instrumentation to the servers

I poked around a bit and it appears we have two main options for metrics: Prometheus and OpenCensus. Trillian appears to support both as well: https://github.com/google/trillian/blob/master/monitoring/prometheus/metrics.go

Here's what I'm hoping to get:

• Cloud-agnostic instrumentation code. The code should only know about the metrics we need to expose, not where they're going.
• Minimal operational complexity
• Since we're running on GKE now, it would be nice for the metrics to show up in the Stackdriver console.

With Prometheus, we would expose metrics on a /metrics endpoint. Then, a Prometheus server would scrape those metrics and upload to Stackdriver.

With OpenCensus, we would use a Stackdriver exporter directly from the Rekor code. This is a bit too direct of a coupling for my liking, but the exporter can be configured to point anywhere easily. I think using this is probably our best bet to avoid having to run a separate Prometheus service.

If we ever have to move off of GKE, we can swap out a few lines to export to wherever we want.

client loginfo: prints unauthenticated root hash and tree size

The root hash and tree size that is printed by the client command loginfo is prepared here:

rekor/cmd/cli/app/log_info.go

Line 71 in 3c6ce16

cmdOutput := &logInfoCmdOutput{

As far as I can tell that output is unauthenticated because it is not connected to the signed log root which is verified here:

rekor/cmd/cli/app/log_info.go

Line 115 in 3c6ce16

lr, err := tcrypto.VerifySignedLogRoot(verifier.PubKey, verifier.SigHash, &sth)

On board CNCF projects

Seek to make an accetable proposal to onboard some CNCF projects into rekor.

• Select candidate projects from teh CNCF eco-system
• Outline a process of how they can start to sign and store rekor manifests into rekors transparency log with automation and ease of use as key.
• Outline a process of how users can then benefit from the signed manifests to attest the inetgrity of release downloads.
• Implement the above proposal into the selected CNCF candidate projects.

Assess SPDX inclusion for k8s release BOM implementation

Related to the following in k8s release SIG : kubernetes/release#1837 (comment)

Explore inclusion of SPDX manifests (XML 😦 ) , namely:

• What sort of values would be critical (place in the t-log) and what could go in ExtraData.
• What sort of signing system is used, how would we ensure non-repudiation.
• How would k8s release display entry and make it valuable, perhaps an inclusion URL pointing to the UUID?

Return body decoded

The entry URL contains a base64 body. Decode this prior to return to the client.

computation of hash when fetching remote artifact is redundant

When using the CLI to create an entry in the log for a remote entry, we currently require the caller to specify the SHA256 hash as well as provide the digital signature. Since verification of the signature will by definition include validation of the hash, we can remove this out of the Rekor processing flow.

@SantiagoTorres

Allow user to set their own meta data

Users can set their own meta data key / values.

Probably need to implement a map interface.

Final productionization stuff:

• Rate limiting?
• API key allow-listing?
• Monitoring?
• Load testing?

E2E tests can fail prematurely because docker-compose services are not healthy before the test run starts

In our CI step for E2E tests, the CLI is rebuilt from scratch in a new GitHub environment; this means that the go module dependencies must be fetched. The delay introduced during this fetch masks a race-condition in thee2e-test.sh script in that the tests can start running before the rekor services and dependencies have come up (this is usually because we are creating a MySQL database for the first time which requires some time to initialize).

Implement a version flag

We need a way to pull the tag into a $version and have it as a flag for the CLI

Plugable Types

This is a placeholder to scope out plugable types. At present we have the following generic manifest type:

{
    "URL": "",
    "SHA": "",
    "PublicKey": "",
    "Signature": ""
}

My proposal is that we add a namespace.

{
	"type": "rekord",
	"version": "0.1",
	"spec": "https://github.com/projectrekor/rekor/rekord_spec.txt",
	"body": {
		"URL": "https://example/release/my_release.tar.gz",
		"SHA": "83jfj8we89903uhejw88…",
		"PublicKey": "PUBLIC_KEY",
		"Signature": "SIGNATURE"
	}
}

• note: I am sure the above can be improved and open to suggestions. I wanted to get the initial intent scoped out.

ns will then map to a type in rekor/pkg/types/rekord.go which in turn will have its own validation checks, version control, signing library (gpg etc).

I see this as work we should do sooner rather then later, to save us coupling too much to the current type and tooling. Changes should be minimal, so we can keep our existing work mostly intact - we will likely need some sort of parent parser who then marshalls the types to their mapped types/<ns>.go

@dlorenc @bobcallaway , please do share thoughts. I will expand this out over the next few days.

CLI: Store the log info locally between invocations (the latest signed tree head)

When loginfo is called, we should store the latest signed tree head locally (in a .rekor directory).

Then, when loginfo is called again, we should perform an inclusion proof between those two signed tree heads.

This can help audit the log and prove it is append-only.

Consider storing the full x509 Certificate

When an entry is added with an x509 certificate instead of a public key, we strip the cert and only store the public key bytes.

The CT Log stores the full cert, but we can only search it by timestamp or full cert, not public key.

If we store the full cert in rekor, we can more easily perform joins between the logs and discovery of certs.

Downsides to storing the full cert:

• More data, more changes for bad content making it into the log
• Duplication of signatures (we could end up with entries for the public key and the cert)

Other ideas:

• Store both! Add one entry for the cert and one for the public key. This might be confusing though.
• Only store certs if they were issued by fulcio, otherwise store the public key.

I think right now I prefer storing the whole thing...

config rekor.yaml confusion

The docs at https://sigstore.dev/get_started/client/ point at ~/.rekor/rekor.yaml for the default yaml.
The code at cmd/cli/app/root.go leaves config empty (I've added the path above), and the help points to $HOME/.rekor.yaml
state is stored in the .rekor dir, so the docs would be right, I think.

I would find it useful to store my public-key, oops public_key url there

pin releases in k8s config

pre-release step: review all k8s configs and ensure we use tags/digests where appropriate.

Integration tests

Perform integration tests in CI (github actions)

Stand up an instance of mariadb, log server / signer and the rekor server and then make an upload and validate it. Make sure a meaningful return code is made to inform a CI pass / fail.

ERROR: logging before flag.Parse

When starting rekor, the following error appears:

go run cmd/server/main.go serve
2020-12-07T11:38:46.453Z	INFO	app/root.go:97	Using config file: /home/luke/go/src/github.com/lukehinds/rekor/rekor-server.yaml
ERROR: logging before flag.Parse: I1207 11:38:46.458754   34544 admin.go:151] Initialising Log 847182197821869483...
ERROR: logging before flag.Parse: I1207 11:38:46.461381   34544 admin.go:162] Initialised Log (847182197821869483) with new SignedTreeHead:
key_hint:"\x0b\xc1˸i5\r\xab" log_root:"\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00 \xe3\xb0\xc4B\x98\xfc\x1c\x14\x9a\xfb\xf4șo\xb9$'\xaeA\xe4d\x9b\x93L\xa4\x95\x99\x1bxR\xb8U\x16Nl=\x0b\x8b(D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" log_root_signature:"0E\x02 d\xcd\xc0\xac\xbd\x99x\xc4%`[k\x8c\xbaz\xe4\xf0q\xe68?Rn\xb9\xbc)\x1f\x99͵\x9b\xa7\x02!\x00\xe3ښ\xef\x83\xe0\xe5\xbf;\xe2\xf9\xe2\x8b}\xaf\xb3b\xc3+T:\x0b\xa4\x9a\x85\xa6I\xe6\xed\xce\x1a\xd9"
2020-12-07T11:38:46.462Z	INFO	api/server.go:55	Starting server...
2020-12-07T11:38:46.462Z	INFO	api/server.go:61	Listening on 127.0.0.1:3000

Spent a while looking into this and trying to find the root cause, it seems to be an upstream issue in glog , see here

@dlorenc you have a better insight into go packaging, any ideas?

Ρεκόρ meaning.

Fist of all can I just say that the project looks really interesting and I'm looking forward to playing around with it.

---

Mostly just FYI: The word Ρεκόρ in Greek translates to something like "previous conduct/performance/high value" or "track record", rather than what I assume you were going for in naming the project. And it would be used in terms such as "world record", "in record time", "set a new record". (https://www.wordreference.com/gren/%CF%81%CE%B5%CE%BA%CF%8C%CF%81)

Still a cool name though :)

Add Additional return output to verify

Explore additional output to the verify command:

go run cmd/cli/main.go verify --public-key tests/test_public_key.key --signature tests/test_file.sig --artifact-path tests/test_file.txt  --artifact-sha 45c7b11fcbf07dec1694adecd8c5b85770a12a6c8dfdcf2580a2db0c47c31779
2020-12-07T12:55:31.987Z	INFO	pkg/client.go:76	Proof correct!

Consider what other fields / values may be useful, perhaps some sort of signed tree head perhaps?

docker-compose.yaml versioning

#184 introduces healthchecks to the docker-compose.yaml file. They contain the start_period option which is not supported for docker-compose file versions < 3.4. This might cause some errors when containers are created:

$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.mysql.healthcheck value Additional properties are not allowed ('start_period' was unexpected)
services.redis-server.healthcheck value Additional properties are not allowed ('start_period' was unexpected)
services.rekor-server.healthcheck value Additional properties are not allowed ('start_period' was unexpected)
services.rekor-server.build contains unsupported option: 'target'

A change in the version of this file might be necessary.

Unit tests

Start to implement

Implement Extra Data

As per discussions on slack. Implement ExtraData for storing of the URL.

After downloading the artifact the URL will be parsed into ExtraData in the addLeaf function

leaf := &trillian.LogLeaf{
    LeafValue: byteValue,
    ExtraData: extraData,
}

Add version subcommand

I've tried to upload a record for my software for testing and I had some issues. I installed rekor using go get -u -v github.com/sigstore/rekor/cmd/cli.

When I pulled the repository and rebuild the binary, everything worked.

It would be nice if --version or version subcommand (ideally both, aliased) could be implemented, so one can quickly check what version it's running. Even though for go get installations it will be difficult to get reasonable version without tagged release.

Project examples

We need an examples repo or folder under the an existing repo.

The examples should demonstrate how users can integrate with Rekors RestAPIs.

For example, this could be some golang examples or python (there is a python trillian library , although it might need an update)

incorrect location returned when object is created if apiKey is specified

The URL returned in the Location header after creating an object does not respect the potential inclusion of an API Key as a query parameter.

rekor git:(remove_hash) ./cli upload --artifact tests/test_file.txt --public-key tests/test_public_key.key --signature tests/test_file.sig 
Using config file: /Users/bcallawa/.rekor.yaml
Created entry at index 19, available at: http://localhost:3000/api/v1/log/entries?apiKey=DdsEjS6U5gsA6zwcSM9xCCv3t2atYhKb/b08416d417acdb0610d4a030d8f697f9d0a718024681a00fa0b9ba67072a38b5

In-toto manifest

Once #40 has landed, implement an in-toto type.

500 from /api/v1/log/proof if lastSize > log size

curl -v "https://api.rekor.dev/api/v1/log/proof?lastSize=1255" gives response:

{"code":500,"message":"Unexpected result from transparency log"}

when treeSize is 1254:

curl -v "https://api.rekor.dev/api/v1/log" gives response:

{"rootHash":"294ed76b6e1f8f17ef81732fa63d0e09d2c51b367f5f23f56fd0f914ab2c38ff","signedTreeHead":{"keyHint":"Ni+Oy6cvQyY=","logRoot":"AAEAAAAAAAAE5iApTtdrbh+PF++Bcy+mPQ4J0sUbNn9fI/Vv0PkUqyw4/xZrB2iBGKEpAAAAAAAACNYAAA==","signature":"MEQCIAh9SkSMSNdlCUIDTi4SX5c9p/G0jIjD3fvI4FpXEcgqAiA4c7zYi27yRchoBdKSCcHM3cB9N45Rc9Xx+adEvQF1PA=="},"treeSize":1254}

but gives a 200 response when lastSize = treeSize:

curl -v "https://api.rekor.dev/api/v1/log/proof?lastSize=1254"

{"hashes":["073970a07c978b7a9ff15b69fe15d87dfb58fd5756086e3d1fb671c2d0bd95c0", *snip*,
"da8602de4650d2a53862a65d3035204d96bcd7e25377574ff437249ab092bd5d"],"rootHash":"294ed76b6e1f8f17ef81732fa63d0e09d2c51b367f5f23f56fd0f914ab2c38ff"}

wrong path for module

go get: github.com/sigstore/rekor@none updating to
github.com/sigstore/[email protected]: parsing go.mod:
module declares its path as: github.com/projectrekor/rekor
but was required as: github.com/sigstore/rekor

Rekor CLI distribution

Once #29 is complete and we are happy with an initial implementation, we should think about how developers get the tool.

One method is go get.

Another is to release right here on github (and of course make an entry into the tlog)

OS Dist packaging (allow dnf install rekor or pacman rekor etc).

• Fedora
• Debian
• Arch Linux

Failed to upload an gzipped artifact

Following https://sigstore.dev/get_started/client/, I have the following output

$ gpg --armor -u [email protected] --output mysignature.asc --detach-sig myrelease.tar.gz
$ rekor upload --rekor_server http://localhost:3000 --signature mysignature.asc --public-key mypublickey.key --artifact "http://localhost:8000/myrelease.tar.gz"
2021/03/12 07:41:27 error retrieving external entities: openpgp: invalid signature: hash tag doesn't match

The reason why hash tag doesn't match is that the signature is for the payload myrelease.tar.gz not for its decompressed content. However, the rekor cli decompress the content automatically and then do the verification.

Public instance hyperlink returns 404

The hyperlink in README.md file api.recor.dev redirects to https://rekor.dev/api/v1/log/ but it returns 404 status code.

Fix up readme

Now that rekor-server has been merged into rekor, we need to update the readme to reference this and include steps for both running the server and the client.

Implement http RestAPI

Feature Request: Federated access

Reasons for doing so:

1. In order to prevent a single point of organizational failure some form of federated services would be useful.
2. Allows for network sharding to deal with temporary or permanent network sharding, to match the real world conditions more accurately.

Some options:
Either allowing for a store to point to another (i.e. RedHat could host a keystore that points to the KDE store). Some issues become people having to figure out which servers to use, much like DNS, likely leading to the use of defaults.

The other option is to use something like ActivityPub (or other gossip protocol) like Mastadon, where a signature could include which server it is posted to (so if a server is known to be untrusted everyone could choose to blacklist it) and know it would need to regather need signatures for software packages hosted from there.

If this already is in use, then maybe more adding documentation on what solution was chosen and how to use it.

arguments specified in environment variables/config file are not validated

$ REKOR_REKOR_SERVER=https://api.rekor.dev*444 ./cli loginfo                                     
2021/02/16 12:18:45 Get "https://api.rekor.dev*444/api/v1/log": dial tcp: lookup api.rekor.dev*444: no such host

This should have failed the command line arg parsing which detects invalid URLs before attempting to connect.

Rename x509 PKI to PKIX

I think this is probably closer to the "correct" name the way we're using these keys, since we don't actually verify a certificate chain.

We could either extend this type to take an optional cert chain to verify the public key against, or have two types of PKI here.

New Project Name

Signlog
Signsafe
Arbiter
Keysign
Hashsign
Treesign
Software transparency (like certificate transparency)
Lets sign
Agano (translation of ‘testament’)
Teist (translation of ‘testament’)
Nema (translation of ‘testament’)
Zavet (translation of ‘testament’)
Teken (afrikaans for ‘sign’)
Dentro (greek translation for 'tree')
Trie (a digital tree) https://en.wikipedia.org/wiki/Trie
Arianna, Ariane or Ariadne
Theseus
Lavra
Deadelus
Knossos
Hansel
Gretel
Hansel-Gretel

Please make a comment and I will add to the list.

Website & Documentation site

We are going a website and docs.

The site should be able to quickly inform a visitor:

• What is rekor / what its not
• How can they get started
• Can rekor do what they

This should be in the form of a single line deployment CLI command (think rustup.rs) and an FAQ.

We should also render openapi docs on to the site and have a backend sandbox that refreshes every x hours.

pubsub/bigquery

We could make some other types of (UNTRUSTED) feeds available to simplify integrations.

• All entries could get sent to a public pubsub/kafka topic for others to use.
• All entries could get synced to a public bigquery instance for ad-hoc querying/analysis.
• Publish daily/hourly (signed) snapshots and backups of the log to some kind of CDN.

Design: Allow for storing "signature promises"

cc @puiterwijk

This would look like an entry in Rekor that includes a digest of a file that will be signed, and the public key that will be used to sign that file.

The inclusion proof of that "promise entry" could then be included in the artifact that is signed, and then the final signature can be stored in rekor, which would include the "promise" as well.

Create helper function for GRPC connection handling

Create a specific function to manage grpc connections. Currently we are repeating the same code in each cobra call:

https://github.com/lukehinds/rekor/blob/0e6a269519cfda4999c19f16f418158d7da75fd4/cmd/get.go#L44-L51

These should all be moved out to a dedicated function

Public Rekor Ledger

A public ledger available. Including:

• Node.js data
• kernel data

Hard to find an existing entry

If you try to add an entry that already exists, we should return the index or UUID to that entry somehow.

Right now, if you try to add an entry we return an error, with no way to find the existing entry.

CLI Productionization:

Append Only

• figure out how to use the CLI to verify the server is append-only
• document that
• test that
• print raw proofs and show how to verify separately

Storing state during a "verify"

• store "high-water-mark" whenever you can
• never go back in time

Timestamps

• See if there's enough info to attest the time an entry was added.