Edit 2022-10-17: This issue is being slightly repurposed to decide requirements and/or guidelines on how artifacts are automatically verified to meet the SLSA requirements.
Tasks:
Context: #130
Original post
There was a lot of discussion in #37 that seems to indicate there's still an apparent disconnect on policy (who produces/owns it) and evaluation (when/where to evaluate that policy).
I think, for the most part, we actually agree with one another. There are just some slight misunderstandings that are confusing us.
I think the following are the open questions:
- Who creates and owns the policy?
- Should we have a separate set or recommendations on what a good policy looks like?
- Should those requirements and recommendations be part of the SLSA levels (e.g. SLSA 3P) or separate from it?
- What use cases are we targeting?
A. complete end-to-end: producer creates and artifact & provenance and those make their way all the way to the consumer who then checks the artifact & provenance against their own policy
B. early evaluation: producer creates and artifact & provenance and at some point of trust (e.g. code signing, ACL'd folder, repo upload) that artifact & provenance against some other policy
- What do people mean when they talk about 'resource'?
Does that sound right?
My opinion on what the answers to these questions are:
- Who creates and owns the policy?
Some of the discussion in #37 has changed my mind.
So, now my opinion is "anyone that cares can create a policy".
The question itself is a bit of a misnomer. There isn't necessarily one single policy that should be evaluated. There could be many policies created and owned by different people. Anyone that cares about the artifact can create their own policy. The artifact producer themselves can create a policy if they want, but it's not required. Downstream users can use the producer's policy if they wish (and there's a way for them to get it securely), but they can also create their own policy (perhaps using create on first use as has been discussed in #37).
One thing we do have to watch out for when there are lots of policies is what happens if/when the software creators refactor how their code is built. E.g. changing the name of the Makefile target, switching repos, etc...
- Should we have a separate set of recommendations on what a good policy looks like?
Yes, I think so. I think having a separate clear set of requirements for policy would help. We have this internally and I think it's helpful.
- Should those requirements and recommendations be part of the SLSA levels (e.g. SLSA 3P) or separate from it?
I think it can be helpful to include them as a part of the SLSA levels, but I don't feel that strongly about it. A reason to do this is that it would be very easy for someone to create a bad policy that doesn't actually provide as much security benefit as they might like. E.g. "Any artifact built by GitHub Actions SLSA 3 builder". Since GitHub Actions is public a malicious actor could create whatever malicious artifact they want with GitHub Actions, and get the policy evaluator to accept that. There'd still be some protection since after the fact an audit could be done to see who built that thing and what source it came from (which might deter some, but not all actors).
- What use cases are we targeting?
I think we should target both A and B. A is the grand vision of where we want the entire industry to go. It might be easier to achieve in some cases than B, but in others it might be harder. B is a bit more limited but allows for adoption in cases that might otherwise be harder to implement. E.g. It would be much easier to protect the signing operation of an iOS app with SLSA than to add support for evaluating SLSA provenance into iOS.
- What do people mean when they talk about 'resource'?
This is just the name of the 'thing' or operation that is being protected by a policy.
Example: PyYAML
Let's assume that policy is evaluated in two places:
- GitHub Artifact Publishing Service (GAPS) which checks policy before uploading a package to a repo. This service manages the credentials needed to upload to the repo for the user.
- An upgraded PyPI client that checks policy before installing a package it got from PyPI.
For both 1 and 2 the systems need to know what policy to evaluate at any given time.
GAPS publishes lots of stuff, so it needs to lookup the policy for the thing being published. It might do this by looking up the policy in map based on the name of the thing it's publishing "pypi:pyyaml". How this policy is created and managed is up to GAPS.
The upgraded PyPI client, meanwhile, installs lots of packages. If it creates policy on first use it still needs to store that policy somewhere and it needs to be able to distinguish the policy for PyYAML from the policy for urllib3. So it might have a database where it can lookup policies based on the name of the thing it's installing. E.g. "pyyaml" or "urllib3".
The policy being evaluated by GAPS and the PyPI service could very well be different and that's totally fine.
It may be that some policies don't have an explicit resource, perhaps it's just some configuration flag that you add to a tool that only does one task. Perhaps when you configure GAPS for your project it doesn't used named policies it just has you provide the policy file. That's fine. But it's still probably the case that some thing (even if not explicit) is being protected by a policy, and that thing is what we call the resource.