veracruz-project / veracruz

Main repository for the Veracruz privacy-preserving compute project, an adopted project of the Confidential Compute Consortium (CCC).

Veracruz: privacy-preserving collaborative compute

About

Veracruz is now an adopted project of the Confidential Compute Consortium (CCC).

Veracruz is a framework for defining and deploying collaborative, privacy-preserving computations amongst a group of mutually mistrusting individuals. Some potential use-cases for Veracruz include:

• Privacy-preserving collaborative machine learning,
• Privacy-preserving delegated computations from a computationally weak device to a more capable (but potentially untrusted) edge device or server,
• Secret auctions, elections or polls, surveys,
• ...and many more.

Veracruz uses strong isolation technology (a mixture of trusted hardware and high-assurance hypervisor-based isolation), along with remote attestation protocols, to establish a safe, "neutral ground" within which a collaborative computation takes place on an untrusted device. Concretely, Veracruz computations are special-purpose WebAssembly binaries, compiled against a small SDK which we provide. WebAssembly acts both as a sandbox, pinning down the behaviour of the program, and allows us to abstract over the different strong isolation technologies that we support.

To learn more about Veracruz, the motivation, design, use-cases, and so on, please read the Veracruz project wiki. The latest project news is available on the Veracruz project homepage.

To jump straight into the codebase, please read the growing number of guides we have on how to get started with Veracruz:

• Build instructions - Set up a build environment for Veracruz
• CLI Quickstart - Quickly run Veracruz with a prepared demo program

News

• May 2023: our paper "Private delegated computations using strong isolation" was accepted into IEEE Transactions on Emerging Topics in Computing.
• June 2022: we have released Veracruz 22.06 (see the veracruz-2206 Git tag). See release notes for notable changes in this release.
• May 2022: we have released Veracruz 22.05 (see the veracruz-2205 Git tag). See release notes for notable changes in this release.
• May 2022: we have released a technical report on the Veracruz and IceCap projects, hosted on Arxiv.
• April 2022: we have released Veracruz 22.04 (see the veracruz-2204 Git tag). See release notes for notable changes in this release.

Get involved!

The Veracruz team welcome new collaborators and contributions! We maintain a list of open issues, ideas for new features, and possible improvements that can be made to Veracruz in our issue tracker, but also welcome ideas for new features and improvements from contributors, too. Many issues in our issue tracker are marked as being suitable for new contributors to the project.

Veracruz maintains a public Slack channel for discussion about the project under the Confidential Compute Consortium's workspace. You can access this channel here. Anybody and everybody is welcome to join to meet the team and discuss the project!

We also have a weekly open Zoom meeting that you are welcome to join which is held every Thursday, 15:00 (BST) / 10:00 (CDT), and last for an hour. You can join with this link. Come along and meet the team, find out what everybody is working on, and discuss ideas for improving Veracruz!

Citing Veracruz

If you use Veracruz or otherwise wish to discuss the project, please cite the project as follows:

@techreport {
  author = {Brossard, Mathias and Bryant, Guilhem and El Gaabouri, Basma and Fan, Xinxin and Ferreira, Alexandre and Grimley-Evans, Edmund and Haster, Christopher and Johnson, Evan and Miller, Derek and Mo, Fan and Mulligan, Dominic P. and Spinale, Nick and van Hensbergen, Eric and Vincent, Hugo J. M. and Xiong, Shale},
  title = {Private delegated computations using strong isolation},
  institution = {Systems Research Group, Arm Research},
  doi = {https://doi.org/10.48550/arXiv.2205.03322},
  type = {Technical report},
  pages = {20},
  year = {2022}
}

License

Except where otherwise stated, this project's codebase is licensed under the MIT license.

veracruz's Issues

Add missing Rustdoc documentation

See #29 for a related issue.

Some Veracruz components are quite well commented with Rustdoc comments, whilst others are not. All modules and at least the exported functions and constants from a module should be equipped with Rustdoc comments.

Reimplement the Veracruz MVP host calls in libveracruz using WASI

For motivation, see Hugo's WASI proposal.

Add a build target to generate Rustdoc documentation files

There are Rustdoc comments sprinkled throughout the Veracruz codebase, but these are not currently being built into browsable documentation as part of the Veracruz build process. We should add a new build target to generate this documentation, which will make the project easier to understand for newcomers.

Implement Attestation as a Certificate Authorty in the Proxy Attestation Service

Requested feature
In order to reduce round-trip network traffic and complexity in the client, we should change the Proxy Attestation Service to act as a Certificate Authority.

Motivation
Our current proxy attestation is complicated and requires multiple back and forths from the client. If the client traffic can be reduced to a simple TLS handshake, this helps us scale to more network and power-constrained client device.

Add further tests

Veracruz has a number of whole-system integration tests which are usually hooked into our CI (see e.g. the TEST_PLAN.markdown file). However, more tests are always welcome, including further module-level/unit tests for the various subcomponents of Veracruz, the use of additional testing frameworks or tools such as fuzzers, and so on.

Provide documentation over contents of the policy file

Requested feature
Provide documentation over the contents of the policy file.

Motivation
The policy file is probably the main way users will interact/configure the system, it would be good to have some sort of standard reference users could refer to without having to dive into the codebase. This would also be a good way to show off some of the features of Veracruz.

Additional context
A template file exists here, which may be a useful starting point:
https://github.com/veracruz-project/veracruz/blob/main/test-collateral/policy.template

Related discussion #68

Add examples folder to demonstrate full Veracruz use cases

Create an examples folder where each example shows how to put all components together (program owner, data owner, policy, wasm binary ..), maybe examples can be driving by different Veracruz use cases.

Change veracruz-utils/src/nitro_enclave.rs to use vsocket

Requested feature
Change veracruz-utils/src/nitro_enclave.rs to use vsocket

Motivation
There's a bit of duplicated code, ans the file veracruz-utils/src/nitro_enclave.rs duplicates some of the functionality available in vsocket.
nitro_enclave.rs should be modified to use vsocket as much as possible. See the //TODO in nitro_enclave.rs

Command-line interface for user-facing components

Requested feature
Would be nice to have command-line interfaces for the various user-facing components of Veracruz: Tabasco, Sinaloa, and Veracruz

Motivation
A command-line interface would probably be the most natural way to run Tabasco and Sinaloa servers in production, and it may be an easier interface for new users to experiment with.

Additional context
Something like this is needed for #41, or any other non-rust clients.

Re-establish build and test CI support

With the move from a closed to an open project we are temporarily without CI support.

Remove Veracruz MVP host calls from Chihuahua

For motivation, see Hugo's WASI proposal.

Sonora enclave functions are not easily discoverable

Describe the bug
The SGX enclave entry functions for Sonora are not easily discoverable

To Reproduce
Try grepping for sinaloa_init_remote_attestation_enc in the project directory. You will find calls to it but not an implementation

Expected behaviour
To aide newcomers to the project, functions should be easy for them to find the implementations of.

Additional context
As part of the build process for SGX enclaves, the SGX tool sgx_edger8r is executed. It has an option --use-prefix, which adds the name of the source edl file to all of the generated symbols. This means that functions such as init_remote_attestation_enc are translated to sinaloa_remote_attestation_enc.

We should stop using the --use-prefix option and instead name the functions in the edl file and the source with the name they will be called with.

Should we rename the crates so the names make sense now?

Requested feature
Should we rename the Rust crates so the names are more descriptive of what the crate does?
Motivation
When the project started, we named the crates after Mexican states. This was mostly because naming is hard, things were in flux, and we wanted to avoid long discussions about what to name things.

Now that things are more stable, it would probably be more inviting to users and developers to provide names for the crates the are more descriptive.

Additional context
Ideas (these are just ideas and of course there's room for discussion):

Jalisco -> SGXRootEnclave
Sonora -> TrustZoneRootEnclave
Tabasco -> ProxyAttestationServer
Colima -> TransportProtocol
Durango -> VeracruzClient
Sinaloa -> VeracruzServer

I'm not sure about Baja, Mexico City, and Chihuahua. We may not want to rename everything at this point.

With a mind for future ideas I have about isolating the platform-specific changes from each other, we may want to have separate crates for the SGX-specific server, TrustZone-specific server, etc.

Ideas for these names: SGXServer, TrustZoneServer.

SgxRootEnclave and TrustZoneRootEnclave names need swapping

Describe the bug
As part of "the great renaming", the names of these two components were inadvertently swapped.

To Reproduce
Look at the repository.

Expected behaviour
What is currently called SgxRootEnclave should in fact be called TrustZoneRootEnclave, and vice versa.

Experiment with micro-controller support for Veracruz

At the moment Veracruz client code (Durango) is designed and built for use on an application-class microprocessor (e.g. an Arm A-core). However, there are potential use-cases where microcontrollers could "talk" directly to the trusted Veracruz runtime. To support this, we need:

1. Some compelling use-cases to act as guides for implementation.
2. A version of Durango that is usable on a resource-constrained device.
3. Additional examples, tied to the use-cases highlighted above.

Code refactoring and tidying

We welcome any refactoring or cleanup of the Veracruz code-base.

Rename the Veracruz components to something more descriptive

As discussed in #26, the various Veracruz components are named after Mexican states. Whilst this worked in the early stages of the project, when we had relatively few components to worry about, the project has now grown to the point where we have a large number of components, each with a Mexican name which is not tied to the purpose of that component. When working on the project full-time, it is hard to keep track of the purpose of each component, and newcomers will undoubtedly have it worse. The consensus, when this issue was discussed, was that these names should now change to something more descriptive, with the suggestions (still open to change or refinement):

• Jalisco -> SGXRootEnclave
• Sonora -> TrustZoneRootEnclave
• Tabasco -> ProxyAttestationServer
• Colima -> TransportProtocol
• Durango -> VeracruzClient
• Sinaloa -> VeracruzServer
• Chihuahua -> ExecutionEngine
• Baja -> SessionManager
• MexicoCity -> RuntimeManager
• Tlaxcala -> WASMChecker

Note that this change will touch almost everything in the Veracruz codebase:

• All crate names need updating, and any references to crates in the Veracruz source,
• All Makefiles and build scripts,
• All documentation (e.g. the COMPONENTS.markdown file, as well as the contents of the project wiki),
• All comments in the various source files.

Pull out the state machine from the execution engine.

The state machine for controlling the Veracruz becomes more complex given the streaming extension. The implementation of the machine is currently mixed with the execution engine (Chihuahua). This becomes a barrier for maintenance. Separately, the state machine relies on the execution engine to set the correct state via explicit setters. Here a better solution here is to pull out the state machine as a more self-contained file or component with a unified API, next([Condition]) for example.

Add support for PSA Crypto to the Veracruz host ABI

There are several interesting use-cases for Veracruz that require the use of cryptographic functionality by the executing WASM program inside the enclave (for example: deniable encryption). At the moment, the Veracruz host ABI does not present any cryptography-related functionality, which forces programmers wishing to use cryptographic to link a cryptography library into WASM program directly. This is both error prone and bloats the size of the WASM binary. Instead, the Veracruz host ABI could be extended to provide common cryptographic functionality, built around Arm's PSA Crypto API, and make this available to the WASM program. We would need:

1. Some candidate use-cases to drive development,
2. A candidate ABI design,
3. Implementation of the required cryptographic primitives in the trusted Veracruz runtime,
4. Extension of libveracruz to expose the PSA Crypto API to the WASM program.

Add support for AWS Nitro Enclaves

Requested feature
Add support for AWS Nitro Enclaves

Motivation
We currently support SGX and TrustZone. AWS recently launched their Nitro Enclaves. We should add support for them.

Additional context
AWS launched Nitro Enclaves last week. Details are online (https://aws.amazon.com/ec2/nitro/nitro-enclaves/)

RFC: Use of Hadolint for Dockerfile linting.

Requested/Suggested feature
Integrate Hadolint into the CI to ease Dockerfile maintenance.

Motivation
With recent work on the Dockerfile for the updated CI, there has been a few respins of the PR that might have been avoided with a Hadolint pass to pick the changes up in one pass, and to pick up any regressions.

Hadolint is available as a binary package, and is also cross-platform compatible with arm64v8 if necessary.

Additional context
Project Oak uses Hadolint to keep their Dockerfiles squeaky clean.

Document how to use Veracruz under seL4

Veracruz seL4 support depends on code originating in a different Arm research project ("IceCap"). Yet, this support is mentioned in various places (e.g. the README.markdown file in this repository). Some documentation is needed to explain how Veracruz can be run with IceCap, e.g. how to obtain IceCap sources, how to build, and so on.

Add more examples to the Veracruz SDK

Veracruz has a host of example programs available in the sdk/examples directory. These are used for integration testing, but also show off what sorts of things Veracruz is capable of. More examples, especially in areas not well-covered by our current set of examples, are welcome.

Documentation about how to link AWS CodeBuild.

Put documentation about how to link AWS CodeBuild in buildspec.yml.
This will help other maintainers to understand the process.

Host Rustdoc documentation on the project website

Requested feature
The Rustdoc documentation present in the code should be automatically generated and hosted somewhere accessible, e.g. the Veracruz project website.

Motivation
At the moment, we have patchy Rustdoc coverage (there's another, existing issue open to try to improve this) but no way of browsing what documentation exists easily. To do this at present, you have to first generate the documentation build manually, and then open a browser at target/doc/package_name/index.html to see the generated documentation. It would be better to just have this done automatically and the documentation hosted on the user's behalf.

Additional context
See also issue #29 which aims to add build targets for Rustdoc to the project Makefiles.

Need to restore random generation of the enclave name

Describe the bug
Long, long ago, the enclave name used as the name for the TLS server was randomly generated.
While implementing support for TrustZone, it was hardcoded to "ac40a0c" to simplify things (and the random number generator wasn't working).
It needs to be changed back to being randomly generated.
This is not a security issue (it would be perfectly valid for it to remain static), but it's bad optics.

Also, we should remove the "get_enclave_name" functionality and instead embed the name in the attestation tokens (this has never been done, so.. new feature, yay!)

To Reproduce
Every time the enclave starts, it just hard codes it to the above value.

Expected behaviour
Each boot of the enclave should generate a new, different name.
Now, on the simulated TrustZone platform, it is likely to always produce the same value, because the RNG is seeded with the same value each time. But for SGX (and Nitro, in the near future), the value should be different every time.

Screenshots, terminal output, or logs
N/a

Environment, Git commit hash, and configuration
n/a

Additional context
n/a

Add compile-time assertions to check the relative bit width of integer types

Various bits of the Veracruz codebase are quite opinionated about the relative sizes of various Rust types, e.g. conversions between usize and u64 are quite common. Technically speaking, these conversions can fail at runtime due to usize not being a fixed-width type in Rust, and these conversions mark one of the few places we still have calls to expect() and unwrap() in the runtime's code-base.

We should codify these assumptions that we make about Rust's integer type bit widths using static assertions, that capture these assumptions at runtime using a library like e.g. static_assertions.

Improve the project wiki

Most of the old README.markdown file for the project has now been moved into the project wiki. However, this documentation can be improved in several ways:

• Some planned pages itemised on the Home page are still missing, so please help to add them.
• Consider adding new pages and linking them from home.
• A lot of the content is optimised towards explaining the project to the CCC technical committee, and not necessarily to newcomers to the project wanting to start contributing. Please change/rewrite the content as you see fit.
• Various useful bits of information, about e.g. common build and test failures, and things like that, should be collected somewhere. If you come across a build/test problem consider adding the problem and its solution to the wiki.

Remove the get_enclave_name and get_enclave_cert functions from the Sinaloa trait

Requested feature
Currently, the Sinaloa trait has two functions who's sole purpose is for testing: get_enclave_name and get_enclave_cert.

These functions should (eventually) be removed, and each of the structs implementing the trait should be changed.

Motivation
It was valuable in the beginning to have tests that wouldn't get mired down in the attestation flows. These functions were added to allow test code to query the enclave for the certificate and name, without going through the attestation process. This short-circuits the attestation process, which is certainly something we don't want anyone doing in a real deployment.

The question now is, do we still think we need the no_attestation tests? Or can we convert them to with_attestation, which would allow us to remove these dangerous functions?

Isolate platform specific changes to their own crates

Requested feature
Currently, all supported isolate technologies are supported by the Sinaloa and Mexico City crates, with features enabled or disabled to support the specific technology.

This has resulted in a confused build process. It also means that the work of adding support for future technologies will create merge conflicts and further complicate the building of these crates.

We should create separate crates for the non-secure side of each of the supported technologies, as well as for the secure-side. This would effectively split out the platform-specific parts of Sinaloa and Mexico City into multiple, platform-specific crates.

Motivation

The current use of features results in code and build processes that are hard to understand and hard to follow. Also, development of support for new platforms is likely to create difficult merge conflicts with other work.

It is desirable to have new platform support to be added (either inside this repository or outside of it) independently of the rest of the work happening on the project.

Additional context
This is likely to have knock-on effects elsewhere in the code base. For example, veracruz-utils has features to enable support for various platforms. It might be beneficial to have a utils-common, and a utils-* crate for each supported platform (or at least those that need platform-specific utilities).

Remove the hack where we pass keys through PSA Crypto to get the public key out

Requested feature
In several places in the code, we load the asymmetric key pair into PSA Crypto and then immediately extract the public key from PSA Crypto. This is done because none of the other crypto libraries support extracting the public component from the keypair. We should find a better way to do this.

Motivation
This is super hacky, and it embarrasses me that I wrote it, but... someone else should fix it.

Additional context
You can find one example of this in nitro-root-enclave/src/main.rs, near let_device_public_key =
There are other places this is done as well (jalisco and sonora also do this, I think)

Push Veracruz Docker image to Docker hub

Push the Docker Veracruz image to Docker Hub to make it easier to test and run Veracruz examples.

Improve readme and repository presentation

Make repository readme lighter and move detailed documentation to a special docs/ folder.

Improve Veracruz Dockerfile

Improve the Dockerfile to make it build faster and make the produced image lighter.

Add build target for `cargo clippy` in Veracruz Makefile

At the moment we do not have support for the Clippy linter in the Veracruz Makefile. Some of Clippy's lint groups would be useful to check on the Veracruz codebase, especially those related to correctness, performance, and idiomatic Rust code style. In the past, I quickly experimented with Clippy on Veracruz but found that lint warnings in our code were swamped by warnings originating from our dependencies. Apparently, this cannot be turned off (see e.g. here). However, maybe this situation has changed, or maybe there's some way of filtering the lint warnings to produce warnings originating from only our code, or similar, which should be investigated.

Create a Veracruz homepage/blog

Create a dedicated Veracruz homepage, distinct from the README.markdown file in the repository, as a proper landing page for the Veracruz project. A blog featuring the latest Veracruz news may also be a good idea, following feedback from @geky on how effective this was for mbed.

We should check in Cargo.lock files

Requested feature
We should check in Cargo.lock files to enable repeatable builds
Motivation
We've had a number of problems arise because dependencies were updated that broke our build process. Checking in the Cargo.lock files would lock us to specific versions, which should get rid of these problems.

Additional context

This isn't really a problem with the dependencies, per se. The dependencies should check for compatibility unless they are incrementing a major release number. However, we are stuck running on a fairly old nightly build. Unless we are able to update to a more recent stable build, it's unreasonable to expect our dependencies to check against our compiler version before releasing.

Thus, checking in our Cargo.lock files feels like the right move. It's possible that, if we ever get on a stable build version, we could change our minds on this.

Add WASI host-calls to Veracruz trusted runtime

For motivation, see Hugo's WASI proposal.

Add support for developing Veracruz programs in C/C++

At the moment Veracruz programs must be developed in Rust. This is not a restriction imposed by the Veracruz framework itself, but rather by libveracruz in the SDK, which is a Rust library. By developing a C analogue of the current libveracruz we will be able to support more source languages for Veracruz development.

Move from TLS1.2 to TLS1.3

At the moment we are using the RusTLS Rust TLS library with TLS1.2 for protected communications between the trusted Veracruz runtime and various principals involved in the computation. We should consider adopting TLS1.3 instead.

Streaming support for Veracruz

At the moment Veracruz has a "batch-oriented" model of computation wherein data is provisioned in its entirety into the trusted Veracruz runtime, which is read by the WASM binary to produce a result, which is then retrievable by the Results Receiver. Whilst this model of computation suffices for certain classes of programs of interest, and with relatively small datasets, it breaks down when:

1. Datasets start to approach the upper ranges of an isolate's resource limits (e.g. Intel SGX's EPC size, or Arm TrustZone's secure memory partition size),
2. Dealing with computational problems where results should be produced gradually, or results should be produced as soon as possible.

In these case, a streaming model of computation is more suitable, where a dataset is fed gradually into the isolate, processed, in chunks, and then a result is fed out gradually in return. To do this properly, we need:

1. Some example use-cases to drive the design and implementation of the enhancement,
2. An analysis of common streaming frameworks to understand the design-space,
3. An implementation of streaming support in the trusted Veracruz runtime,
4. The addition of new features to libveracruz to reflect this new support,
5. Some reflection of which model of computation will be used in the Global Policy,
6. An implementation of the example use-cases, described above, to be used in integration tests.

Need to develop example client code

As it currently stands, the only code that we have that interacts with the Durango client library is test code.
This is not very user-friendly, as the test code is not easy to follow.

We should probably create a series of example crates that give users templates on how to create and use client software.

Setup supporting Veracruz repositories

The main Veracruz repository depends on various forked libraries and Docker files hosted in repositories under the main veracruz-project organization. Some of these (e.g. the Veracruz Docker image repository) are important enough that they should have a dedicated README explaining what they are for, and pointing new users to somewhere that explains how to use them (either somewhere in that repository, or e.g. on the wiki in this main repository).

Update BUILD_INSTRUCTIONS.markdown to match latest Docker developments

Describe the bug
The BUILD_INSTRUCTIONS.markdown is hard to follow, especially in light of our recent Docker changes.

To Reproduce
Follow the BUILD_INSTRUCTIONS.markdown file in conjunction with the contents of README.md in veracruz-docker-image.

Expected behaviour

A few points of inconsistency, or things that could be clarified, stand out:

1. The Intel IAS service does not give you anything that it describes as a "token", yet this is mentioned in the BUILD_INSTRUCTIONS.markdown file. Rather, a primary and a secondary key are generated on your behalf.
2. It's not clear whether you should sign up to a linkable or an unlinkable EPID account from the instructions, or whether this matters.
3. The process for building the Docker container now differs from what is described in the build instructions, and is more accurately reflected in the README.md file in veracruz-docker-image.

Remove duplication of the AWS Nitro Root certificate

Requested feature
Remove duplication of the AWS Nitro Root enclave certificate in the code.

Motivation
The certificate is currently hard-coded in two locations in the code: In the tabasco server, and in the nitro-root-enclave.
This value is not expected to change, ever (the certificate's expiry is 2049, and if we need to change it, something VERY VERY BAD has happened at AWS).
However, having it in two locations is messy and unnecessary.

Add support for MCU clients

Requested feature
Add support for MCU clients with minimal resources running on a Cortex-M class device

Motivation
Currently, the client library is written without constrained devices in mind, so it is not optimized for size and performance.
We see use-cases where low end Cortex-M devices running an embedded RTOS would like to offload computation without having to trust the owner of the device being offloaded to.

Additional context
We currently use RustLS, which is not optimized for low end devices. Should probably use MbedTLS as the TLS library.

Add support for JITing Veracruz in Intel SGX isolates

At the moment, WASM JITting is only supported in freestanding-chihuahua and our experimented seL4 backend. This should be extended to supporting JITting under Intel SGX, by compiling wasmtime against the Intel SGX SDK libraries.

Change Markdown files to match Github's expectations

1. Github expects the SECURITY_DISCLOSURE.markdown file to be called SECURITY.markdown.
2. The code-of-conduct should be moved out of CONTRIBUTING.markdown and put in a dedicated file.

Remove remaining uses of Python 2

Requested feature
Remove any remaining uses of Python 2 in the Veracruz codebase.

Motivation
Python 2 is now not supported. I think our one remaining use of it is in generating policy files from the policy file templates. This tool should either be ported across to Python 3 or rewritten in Rust.

Additional context
N/A.

Remove RUNTIME_MANAGER_HASH from nitro-root-enclave/src/main.rs

Requested feature
Remove MEXICO_CITY_HASH from nitro-root-enclave/src/main.rs. I don't think it necessary any more.

Motivation
I think it was added as a hack, but I think it's no longer needed.

Additional context
Before attestation was fully functional, we had to add the hash of mexico city to the nitro root enclave manaully. I think it's handled correctly now and this should be removed.