google / trillian-examples

A place to store some examples which use Trillian APIs to build things.

Trillian examples

This repository contains example applications built on top of Trillian, showing that it's possible to apply transparency concepts to problems other than certificates. It also contains general-purpose components that can be used to strengthen the guarantees of a transparent ecosystem that already contains verifiable logs.

Currently the examples here are:

• binary_transparency/firmware: A demo showing how to apply transparency bring discoverability to device firmware updates, but the principles are also more generally applicable to all kinds of binaries/updates.
• helloworld: A simple example demonstrating the correct configuration of a Trillian log, personality, and client.
• sumdbaudit: Demonstration of an auditor for the GoLang SumDB module proxy, which clones a log and verifies the data in it.

The general-purpose components are:

• serverless: A suite of command-line tools for managing transparency logs whose state is entirely composed of on-disk files, along with examples of how to use GitHub/GitHub Actions to host & publicly serve the log.

Notable projects that have graduated from this repository to their own top-level repositories:

• witness
• distributor

There are two experimental deployments of the witness that have been deleted but are signposted here for archival reasons. Both of these tools can be retrieved by cloning this repository at git commit 793dcf1:

• usbarmory witness
• ethereum witness

These examples and components are not supported per-se, but the Trillian team will likely try to help where possible. You can contact them via the channels listed under Support on the Trillian repo.

trillian-examples's Issues

Error while testing

ImportError while importing test module '/home/runner/work/trillian-examples/trillian-examples/witness/ethereum/tests/test_witness_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
This error was generated while testing the code using python application workflow on github. Please remove the '-' sign from your directory names. I think that might be the problem.

Missing packages

Hi

Running make trillian results in lots of missing packages. Am I missing something or am I to go package by package and install the missing dependencies?

Thanks!

:~/go/src/github.com/google/trillian-examples/etherslurp$ make trillian
go get -u github.com/google/trillian
cd /home/paulmattei/go/src/github.com/google/trillian &&
go build ./server/trillian_log_server &&
go build ./server/trillian_log_signer &&
go build ./server/trillian_map_server
cmd/flags.go:23:2: cannot find package "bitbucket.org/creachadair/shell" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/bitbucket.org/creachadair/shell (vendor tree)
/usr/local/go/src/bitbucket.org/creachadair/shell (from $GOROOT)
/home/paulmattei/go/src/bitbucket.org/creachadair/shell (from $GOPATH)
server/cloudspanner_storage_provider.go:27:2: cannot find package "cloud.google.com/go/spanner" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/cloud.google.com/go/spanner (vendor tree)
/usr/local/go/src/cloud.google.com/go/spanner (from $GOROOT)
/home/paulmattei/go/src/cloud.google.com/go/spanner (from $GOPATH)
merkle/objhasher/objhasher.go:22:2: cannot find package "github.com/benlaurie/objecthash/go/objecthash" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/benlaurie/objecthash/go/objecthash (vendor tree)
/usr/local/go/src/github.com/benlaurie/objecthash/go/objecthash (from $GOROOT)
/home/paulmattei/go/src/github.com/benlaurie/objecthash/go/objecthash (from $GOPATH)
server/mysql_storage_provider.go:28:2: cannot find package "github.com/go-sql-driver/mysql" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/go-sql-driver/mysql (vendor tree)
/usr/local/go/src/github.com/go-sql-driver/mysql (from $GOROOT)
/home/paulmattei/go/src/github.com/go-sql-driver/mysql (from $GOPATH)
server/trillian_log_server/main.go:23:2: cannot find package "github.com/golang/glog" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/golang/glog (vendor tree)
/usr/local/go/src/github.com/golang/glog (from $GOROOT)
/home/paulmattei/go/src/github.com/golang/glog (from $GOPATH)
quota/mock_quota.go:9:2: cannot find package "github.com/golang/mock/gomock" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/golang/mock/gomock (vendor tree)
/usr/local/go/src/github.com/golang/mock/gomock (from $GOROOT)
/home/paulmattei/go/src/github.com/golang/mock/gomock (from $GOPATH)
storage/memory/log_storage.go:26:2: cannot find package "github.com/google/btree" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/google/btree (vendor tree)
/usr/local/go/src/github.com/google/btree (from $GOROOT)
/home/paulmattei/go/src/github.com/google/btree (from $GOPATH)
types/logroot.go:23:2: cannot find package "github.com/google/certificate-transparency-go/tls" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/google/certificate-transparency-go/tls (vendor tree)
/usr/local/go/src/github.com/google/certificate-transparency-go/tls (from $GOROOT)
/home/paulmattei/go/src/github.com/google/certificate-transparency-go/tls (from $GOPATH)
monitoring/prometheus/metrics.go:25:2: cannot find package "github.com/prometheus/client_golang/prometheus" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/prometheus/client_golang/prometheus (vendor tree)
/usr/local/go/src/github.com/prometheus/client_golang/prometheus (from $GOROOT)
/home/paulmattei/go/src/github.com/prometheus/client_golang/prometheus (from $GOPATH)
server/main.go:31:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
/usr/local/go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
/home/paulmattei/go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
monitoring/prometheus/metrics.go:26:2: cannot find package "github.com/prometheus/client_model/go" in any of:
/home/paulmattei/go/src/github.com/google/trillian/vendor/github.com/prometheus/client_model/go (vendor tree)
/usr/local/go/src/github.com/prometheus/client_model/go (from $GOROOT)
/home/paulmattei/go/src/github.com/prometheus/client_model/go (from $GOPATH)
Makefile:6: recipe for target 'trillian' failed
make: *** [trillian] Error 1

"registers:" SetLeaves() fails

When getting to the make mapper section, the SetLeaves() call fails

go run mapper/main.go --log_id=`cat logid` --map_id=`cat mapid`
2020/06/18 14:39:52 k: register ts: 2016-08-04 14:45:41 +0000 UTC
2020/06/18 14:39:52 key=register leaf=
evicting register -> &{map[entry-number:1 entry-timestamp:2016-08-04T14:45:41Z index-entry-number:1 item-hash:[sha-256:5fa9c4b569a71542c9c791203d54b6b94c125f189a9c529c63466a0b34cbe38c] key:register] [map[fields:[register text registry phase copyright fields] phase:beta register:register registry:cabinet-office text:Registers maintained by HM Government]]}
2020/06/18 14:39:52 SetLeaves() failed: rpc error: code = FailedPrecondition desc = revision must be > 0
exit status 1
make: *** [Makefile:45: mapper] Error 1

Add an example showing how to utilize Trillian for Key Compromise Transparency

@robstradling from Comodo has a great idea to bring clarity and consistency to the ecosystem on Key Compromises.

For example, recently a CA reseller compromised 23k keys, there have been discussions on:

• How should that key compromise have been demonstrated to the issuer?
• How to prove to the larger community as a whole that keys were compromised and give other CAs an opportunity to not use those keys?

Building something like this on Trillian would be relatively straightforward and would represent a great way to use Trillian and thus would be a good example.

This bug is an attempt to convince @robstradling to consider building an example of what he has in mind on top of Trillian :)

Serverless client inclusion command output not easy to use

If you want to externally verify a log inclusion proof, you need several pieces of information:

• tree size
• node index
• tree root hash
• array of node hashes
• hash or node contents of the node whose inclusion is being verified
• optionally, the log signature and witness signature(s)

The client inclusion command does make all of this information available, but it is not all in one place, and it is not in a format that is machine readable.

The hashes can be written to a file with the output_inclusion_proof option. This output has one hash per line, encoded with base64. So this information is machine-readable. However, it's not possible to use these hashes to do anything without the other information listed above.

The client inclusion command does output the tree size, the node index and the tree hash, but this is only available from the log output of the command, and is not in a format which is machine readable. Here is one example:

I1019 10:19:47.236496   30779 client.go:182] Local log state cache disabled
I1019 10:19:47.238259   30779 client.go:269] Leaf "leaf00009" found at index 9
I1019 10:19:47.238668   30779 client.go:298] Inclusion verified under checkpoint:
enclave-transparency-test
10
WpYbjsezEfgQanMdAKR4HYlHezFB8iYnG7z6JZ6VXAs=

Note that I could not find any current way to get client inclusion to output the signatures. It does verify the signatures, but external tooling might want to verify the signatures itself and not rely on the client command doing the signature verification.

It would be much easier for external tooling if the client inclusion command had the option to output all of the needed information in a single file in machine-readable format, such as JSON. The client program already has all of the needed information, it's just not outputting them in a way that's convenient for other programs to consume.

I can work on adding this output.

Enhance README.md to have a FT Map story nicely connected with the existing story

Current https://github.com/google/trillian-examples/tree/master/binary_transparency/firmware has a nice story till Terminal 666 - The Hacker:

This needs to be extended to include the FT-Map story!

Verification of db data

How do I verify the consistency of records in the database on a Trillian helloworld instance?
I manually changed the records in the database from the LeafData table. Then I ran TestIncl this function. It worked without error.How can I prevent this?

Use YAML for human-written config files

JSON is a reasonable compromise for machine-generated-but-human-inspectable stuff, but it's not brilliant for human-authored configs. YAML is much better at this, and in theory at least, is a super-set of JSON so perhaps any existing JSON configs will continue to work.

"registers" : Identical port use causes "bind: address already in use"

When running make tlsigner it fails from trying to bind to the same port as the already running logserver 8090

F0618 14:26:06.182534  880125 main.go:205] Server exited with error: listen tcp 127.0.0.1:8090: bind: address already in use                                                                                                                │
make: *** [Makefile:15: tlsigner] Error 1    

[Serverless] Split up witness PR handling

Currently, the experimental witness PR action code tries to do everything (validate and merge, etc.) on the PR branch itself, but this gets tricky when the PR is coming from a fork.

Instead, we should just validate that the incoming witness PR is good, allow it to merge, and then update checkpoint.witnessed on master.

[Serverless] Break out log and distributor roles

The serverless github action stuff supports both log and (nascent) witness distributor activity, these should be broken out so it's clear/easy to see how one might choose to setup and run one or both of these things both independently or even in the same repo if desired.

[Witness/Armory] Add some form of durable storage

Make tlsigner in etherslurp has port conflicts

Following the README for yields the following after running make tlsigner

vishalkuo@Vishal's Macbook Pro etherslurp (master)$ make tlsigner
cd /Users/vishalkuo/go_work/src/github.com/google/trillian && ./trillian_log_signer --logtostderr --force_master --http_endpoint=localhost:8092 --batch_size=1000 --sequencer_guard_window=0 --sequencer_interval=200ms
I0910 19:57:33.423265   48992 main.go:86] **** Log Signer Starting ****
W0910 19:57:33.424667   48992 main.go:113] **** Acting as master for all logs ****
I0910 19:57:33.424698   48992 mysql_quota_provider.go:44] Using MySQL QuotaManager
I0910 19:57:33.424922   48992 log_operation_manager.go:287] Log operation manager starting
I0910 19:57:33.425415   48992 main.go:161] RPC server starting on localhost:8090
I0910 19:57:33.425453   48992 main.go:145] HTTP server starting on localhost:8092
F0910 19:57:33.427225   48992 main.go:184] Server exited with error: listen tcp 127.0.0.1:8090: bind: address already in use
make: *** [tlsigner] Error 1

Since tlserver already starts on port 8090, tlsigner should use a different rpc port to avoid this conflict.

Makefile targets rungeth and initgeth get wrong datadir

Issue comes because $HOME collides with $H in Makefile and creates datadir=./OME/rinkeby

trillian-examples/etherslurp/Makefile

Lines 44 to 48 in 13b7e9c

	initgeth:: $G $E/rinkeby.json
	$G --datadir=$HOME/rinkeby init $E/rinkeby.json
	rungeth::
	$G --networkid=4 --datadir=$HOME/rinkeby --cache=1024 --syncmode=full --verbosity 3 --ethstats='yournode:Respect my [email protected]' --bootnodes=enode://a24ac7c5484ef4ed0c5eb2d36620ba4e4aa13b8c84684e1b4aab0cebea2ae45cb4d375b77eab56516d34bfbd3c1a833fc51296ff084b770b94fb9028c4d25ccf@52.169.42.101:30303 --rpc console

INFO [07-31|00:54:06.739] Database closed                          database=<GOPATH>/src/github.com/google/trillian-examples/etherslurp/OME/rinkeby/geth/chaindata
INFO [07-31|00:54:06.739] Stats daemon stopped

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl" (https fetch: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: EOF)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Re-add map demos removed in #257

Two examples of the map were removed in #257 due to a Trillian upgrade which removed the incremental DB-backed map. It would be well worth considering rewriting and adding back these examples to use the batchmap code.

Trillian and syslog centralized ?

So, many company centralized computer/servers logs using syslog-ng.
I’m looking a way to secure logs inside this centralized server in integrity. This logs as forensic value so we have to trust them…

If for example an administrator try to :
• Add log entries
• Remove log entries
• Edit log entries
after they are write on disk we need to find something that can help us to say « you can be sure this logs are the original once nobody touch them »

So, with syslog-ng we are able to write or own python destination.
We can imagine :
All logs when they are received by the syslog-ng centralize server before directly write log on disk,
We can wwrite metadata in trillian
like that :
{
"hash": "...", // hash_of_the_data
"timestamp": "08.27.17;+4fes",// syslog_timestamp
"source": "server.com", // syslog_sender_ip_or_host
"line_nb": "485" // current_position_in_files
}

And when auditor/security team would like to verify forensic logs files just ask to trillian if the forensic logs are coherent with it’s data.

Hope you understand what I mean ?
Do you think it’s a good idea ? Is it feasible ?
Sorry about my poor english
Thank you in advance for your answers

Some related content here :
https://news.ycombinator.com/item?id=25995034
https://transparency.dev/application/reliably-log-all-actions-performed-on-your-servers/

Don't pass secrets via flags

Looking at witness/golang/omniwitness/monolith/monolith.go, we realized that it takes the private key material via flags. This is a security risk as anyone on the same machine could get this via ps.

We should swap this over to use env variables or a file, or something. And also review other main files to make sure there aren't others like this.

Once I've eaten all my disk with etherslurp, how do I clean up? [EOM]

Witness can't recover from invalid proof supplied to TOFU update

I just got myself into a pickle - I accidentally misconfigured a witness for a compact-range based log, forgetting to set UseCompact: true in the witness config. The feeder happily sent down the TOFU checkpoint + proof, which the witness stored in the DB, but the next attempt to update failed with:

 main.go:97] Failed to feed: failed to update checkpoint: bad status response (500 Internal Server Error): "failed to update to new checkpoint: couldn't unmarshal proof: data should have trailing newline on last hash too\n"`

Eventually I figured out what I'd done wrong, and I updated the witness config, but the witness was borked for that log and the only way to recover without losing the state for the other logs being witnessed was to use the sqlite3 tool and manually delete the row corresponding to the borked log.

Provide simple example

The current examples are really good, but I think it would be useful to have a 'hello world' level example for those new to trillian.

Such as start a log server, signer, make a simple entry, retrieve the entry.

Typo in the Firmware Transparency Demo Doc

Doc location: https://github.com/google/trillian-examples/tree/master/binary_transparency/firmware

This line:
go run ./cmd/flash_tool/ --logtostderr --update_file=/tmp/update.ota --device_storage=/tmp/dummy_device # --force if it's the first time

miss the specification of the device. It shows the following error message.
F1217 18:17:28.270611 36381 flash_tool.go:53] device must be one of: 'dummy', 'armory'

We might want to add it as following. Or we might want to change the https://github.com/google/trillian-examples/blob/master/binary_transparency/firmware/cmd/flash_tool/flash_tool.go to have a default dummy device.

might want to change to:
go run ./cmd/flash_tool/ --logtostderr --update_file=/tmp/update.ota --device_storage=/tmp/dummy_device --device=dummy # --force if it's the first time

go get fails due to etcd path change

Per etcd-io/etcd#10044 (comment) (cited by etcd-io/etcd#10051), on a clean machine when I try to get the gossip hub the following happens:

$ go get github.com/google/trillian-examples/gossip/...
# github.com/google/trillian
go/src/github.com/google/trillian/trillian_admin_api.pb.gw.go:87:3: undefined: runtime.CamelCaseFieldMask
# github.com/coreos/etcd/clientv3
go/src/github.com/coreos/etcd/clientv3/auth.go:121:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthEnable
go/src/github.com/coreos/etcd/clientv3/auth.go:126:74: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthDisable
go/src/github.com/coreos/etcd/clientv3/auth.go:131:152: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
go/src/github.com/coreos/etcd/clientv3/auth.go:136:144: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
go/src/github.com/coreos/etcd/clientv3/auth.go:141:86: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserDelete
go/src/github.com/coreos/etcd/clientv3/auth.go:146:122: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserChangePassword
go/src/github.com/coreos/etcd/clientv3/auth.go:151:104: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGrantRole
go/src/github.com/coreos/etcd/clientv3/auth.go:156:80: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGet
go/src/github.com/coreos/etcd/clientv3/auth.go:161:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserList
go/src/github.com/coreos/etcd/clientv3/auth.go:166:106: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserRevokeRole
go/src/github.com/coreos/etcd/clientv3/auth.go:166:106: too many errors
# github.com/google/trillian/quota/etcd/quotapb
go/src/github.com/google/trillian/quota/etcd/quotapb/quotapb.pb.gw.go:152:3: undefined: runtime.CamelCaseFieldMask

Question about Firmware Transparency Design: update client (flash_tool) verifying consistency

Morning!

TLDR' I think we might want FT Log to respond with an inclusion promise (https://github.com/google/trillian/blob/master/docs/TransparentLogging.md#inclusion-proofs-vs-promises) called signed firmware timestamp (SFT similar to SCT in https://www.certificate-transparency.org/how-ct-works) to submission of valid firmware. And then for Update Client, verification of the SFT should be good enough instead of verification of consistency. Verification of consistency leads to availability issues and potential attack surface with URL of FT Log alone.

The following are the details. Because we are discussing the design, so I will use the language in the Claimant models.

From the description of the overview figure of the design page, the Update Client in the Device block is acting as the Believer role of both FIRMWARE and FIRMWARE_Log. And it is also acting as the Verifier role of FIRMWARE_Log by asking for consistency proof from FT Log and verifies the received proof. This is a 'Verify before Use' mode of believer as mentioned in https://github.com/google/trillian/blob/master/docs/claimantmodel/CoreModel.md#trust-but-verify. And Verify before Use mode is subject to availability issue. While Transparency is mostly used with the 'Trust but Verify' mode AFAIK. And also, for Update Client to verify the identity of FT Log, a URL might not be good enough (DNS poisoning etc). A fake FT log can lead to a Split view attack. So most likely a hard-coded certificate of FT Personality is required. On the other hand, if we let FT Log to respond with a signed firmware timestamp (SFT) to submission of valid firmware to FT Log and let Update Client verifies the SFT. We can solve the availability issue, the potential attack surface and can still achieve eventual discoverability. (Note that, in the current demo implementation of FT Log, publisher.go would submit firmware image and wait for proof of inclusion. Only after the publisher validates the inclusion of proof, would it proceed. So in this case, the MMD field in SFT could be deprecated. And signed firmware timestamp (SFT) might be better names signed firmware promise (SFP)? Just random thoughts.)

Since Update Client is a Believer of FIRMWARE_Log, so it might want to just believe FT_log (by verifying SFT) instead of still acting as Verifier. And since FIRMWARE vendor's monitor and Observer's FT Monitor are already acting as Verifier, so we should be able to assume any inconsistency of FT Log would be reported to Arbiter once found.

To put it simple, the Update Client needs to validate the blessing from two entities, one is the Vendor and the other is FT Log. And it shall not be the Update Client's duty to make sure these two entities are still working fine. It's FT Log's job to keep an eye on Vendor. And its Log monitor's (could be held by Vendor or third parties) job to watch FT Log. Putting the validation of FT Log working fine burden on Update Client does not seem to add much value. Since if the Vendor is benign, it would notice the misbehavior of FT Log and switch to another benign FT log source before it publishes an image.

If we want to defend against compromised FIRMWARE Vendor and compromised FT Log colluding together, then we need to add inclusion proof validation in Update Client and add multiple FT Log sources. See https://tools.ietf.org/html/draft-ietf-trans-threat-analysis-16#section-3.3.3. But this is out of scope for this issue.

Please let me know if I have any misunderstandings or incorrect assumptions!

ctclone Fails with Incomplete CT Log Responses

Issue Description

The ctclone tool encounters failures when Certificate Transparency (CT) logs return fewer entries than requested. This issue becomes problematic during periodic ctclone operations or at the ends of the logs. At certain points, the position in the cloned log may align in such a way that the CT log returns less than the expected number of entries.

Specific Example

An instance of this issue can be observed with Sectigo's Sabre 2025h2. When a "negotiated" page size of 256 is used, the log returns only 165 entries for a specific range. This response unexpectedly brings the position to 52992 (207 * 256), causing ctclone to fail with the following error:

worker 0: Retryable error getting data: "LeafFetcher.Batch(52827, 256): wanted 256 leaves but got 165"

Additional Context

Further details about this behavior in CT logs can be found in the discussion on Let's Encrypt's forum regarding the 'coerced get-entries' feature.

Important Note

The proposed solution covers only the cases where the actual log's page size is a multiple of the requested batch size. It is assumed that the negotiated batch size requested by ctclone aligns with this multiple. If the actual page size does not conform to this pattern, additional adjustments may be required.

Edited: formatting.

Delete the witness code

Witness code is being moved to https://github.com/transparency-dev/witness/

Once we're happy that all of the code is in its new home, and that we have cloud builds set up properly building docker images, etc, then the old code in this repo should be deleted.

Update merkle dep, and remove calls to visitor for leaves

See transparency-dev/merkle#20

Repo doesn't build

Current master branch fails to build (example). Possibly knock-on effect from google/trillian#1400 ?

Supporting the kernel.org transparency log

kernel.org maintains a transparency log in the form of a git repository and I was wondering how one should go about supporting this for the omnifeeder? Currently creating proofs is impractical as one would need to traverse the git repository and not all entries on the log is signed either. I'm a bit unsure about the usefulness of supporting git repositories like this in general?

Would it be better to throw the entries on a serverless implementation maybe?

https://git.kernel.org/pub/scm/infra/transparency-logs/gitolite/git/1.git/

I wrote up a monitor last year, but something more sound would probably be better :) https://tlog.linderud.dev/

Travis relies on MODPROXY=direct

The travis config uses MODPROXY=direct in order to work around cruftiness with 3rd party deps.
Once the deps are all fixed up then we should remove this workaround.

Related to google/trillian#1810

Replace deprecated `set-output` command with environment file

Describe the issue
In example config, set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information, see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Omniwitness Monolith docker image doesn't run on armv7

When running the monolith on an RPi:

docker run gcr.io/trillian-opensource-ci/omniwitness-monolith:latest                                                                                                                                      exec /bin/omniwitness: no such file or directory

docker run -it --entrypoint /bin/bash gcr.io/trillian-opensource-ci/omniwitness-monolith:latest
root@48793c53012d:/go# ldd /bin/omniwitness
	linux-vdso.so.1 (0x7eb89000)
	libc.musl-armv7.so.1 => not found

The Dockerfile is built by cloudbuild_docker and is clearly intended to be runnable on armv7 (see _DOCKER_BUILDX_PLATFORMS in the cloud build). However, the musl dependencies (installed here) appears not to be successfully installed in the arm image given the output from ldd above.