Action Cache Verification

[sluongng]

I want to split this issue from #133 to discuss about improvements to Action Cache verification specifically.

As folks are already aware, Action Cache is a Key-Value store that maps a cache key (i.e, an Action's digest) to an Action Result message, which in turn points to the outputs of said action. Currently, there is no way to validate if the given Key-Value mapping is "correct". To put this differently, clients kinda have to "blindly" trust the content of the Action Cache, and server implementations often secure the Action Cache by enforcing write (UpdateActionCache) permissions to only trusted clients.

As the next major step for remote-api v2, I want to explore the solutions where we can increase the trust for Action Cache entries for multi-tenancy use cases.

Specifically, I want to enable a client to trust and use Action Cache entries created by a third party through verification. There are a few prior arts, such as:

1. Git commit signing

• https://git-scm.com/docs/gitformat-signature
• https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

2. OCI container signing with SigStore

• https://docs.sigstore.dev/cosign/signing/signing_with_blobs/
• https://git-scm.com/docs/gitformat-signature

3. Nix packaging shared cache

• https://nix.dev/tutorials/nixos/binary-cache-setup.html#generate-a-signing-key-pair

4. Go Module Checksum database

• https://go.dev/ref/mod#checksum-database
• https://research.swtch.com/tlog
• https://github.com/google/trillian

5. Google OSS Rebuild

• https://github.com/google/oss-rebuild

[EdSchouten]

Bonanza has replaced the Action Cache with a mechanism where you can tag objects in the CAS. So the 'action digest' is a key, and you can use that to create tags that point to some CAS object. So in my case I store the equivalent of ActionResult in the CAS. Tags can be signed. Protocol can be found here:

https://github.com/buildbarn/bonanza/blob/main/pkg/proto/storage/tag/tag.proto

[sluongng]

So to establish the basics, I think it is worth stating the questions:

1. Which data do we want to sign? What NOT to sign

The sign payload seems to be a combination of the ActionCache Key (usually a Digest of an Action message), together with a sanitized ActionResult.

There is ActionResult.execution_metadata with a bunch of execution-related data that I am not sure if they should be signed. There is also auxiliary_metadata inside the execution_metadata that can be used for plenty of things. I am not entirely sure which among these metadata is worth keeping during signing: is it the entire execution metadata, or do we sign without the auxiliary ones?

2. What signing key should we support? How do clients/servers discover the public keys to verify the signature?

I think the API should avoid taking an opinion on this, at least during the early version of AC signing spec. That means clients and servers are free to choose which public key to use to verify the signature.

The signature should be formatted in a way that, upon parsing, the client could easily use a popular verifier (GPG, GPGSM, SSH-KEYGEN) to validate the payload. That means we usually expect the first line to be

-----BEGIN SSH SIGNATURE-----

or

-----BEGIN SIGNED MESSAGE-----

or something similar.

3. Where do we store the signature of an Action Cache entry?

The storage location should make it easy for a client to verify the entry. We want to avoid making the client call a second RPC just to download the signature of the AC entry. I believe this is necessary to encourage the use of AC signing and improve the supply chain security of the whole system.

And the best way to do that would be to include the signature as part of the existing ActionResult message returned from GetActionResult rpc. This likely means that if clients/servers decided to store the ActionResult as-is in CAS, there will be less deduplication because the ActionCache Key will be part of the signature calculation.

I do believe multiple Actions to share the same ActionResult is relatively uncommon, though. And even if it happens, an ActionResult proto message is fairly small vs most CAS objects and should have minimal impact. A storage solution can always opt to store the signature separately to dedupe things if needed.

[EdSchouten]

So to establish the basics, I think it is worth stating the questions:

1. Which data do we want to sign? What NOT to sign

The sign payload seems to be a combination of the ActionCache Key (usually a Digest of an Action message), together with a sanitized ActionResult.

There is ActionResult.execution_metadata with a bunch of execution-related data that I am not sure if they should be signed. There is also auxiliary_metadata inside the execution_metadata that can be used for plenty of things. I am not entirely sure which among these metadata is worth keeping during signing: is it the entire execution metadata, or do we sign without the auxiliary ones?

To me it's not evident why such a distinction needs to be made. A worker may produce an ActionResult in its entirety. It can therefore sign the ActionResult as a whole. Why not just marshal the ActionResult Protobuf message and throw the bytes into a signing algorithm?

The signature should be formatted in a way that, upon parsing, the client could easily use a popular verifier (GPG, GPGSM, SSH-KEYGEN) to validate the payload. That means we usually expect the first line to be

-----BEGIN SSH SIGNATURE-----

or

-----BEGIN SIGNED MESSAGE-----

or something similar.

That sounds unnecessary. Even formats like JWT just append the signature in the form of raw bytes, without any container structure around it. The algorithm used by the public key implies the format of the signature.

3. Where do we store the signature of an Action Cache entry?

The storage location should make it easy for a client to verify the entry. We want to avoid making the client call a second RPC just to download the signature of the AC entry. I believe this is necessary to encourage the use of AC signing and improve the supply chain security of the whole system.

And the best way to do that would be to include the signature as part of the existing ActionResult message returned from GetActionResult rpc. This likely means that if clients/servers decided to store the ActionResult as-is in CAS, there will be less deduplication because the ActionCache Key will be part of the signature calculation.

I do believe multiple Actions to share the same ActionResult is relatively uncommon, though. And even if it happens, an ActionResult proto message is fairly small vs most CAS objects and should have minimal impact. A storage solution can always opt to store the signature separately to dedupe things if needed.

We all know that the current design of GetActionResult() is a complete pain in the back. The fact that it returns the ActionResult directly instead of using a separate GetActionResultResponse message makes it all very inflexible.

I would suggest adding a GetActionResult2() or something that fixes this mistake. While you're at it, you can clean up the mess named 'inlining'.