Support QoS for writes

[gdbelvin]

Goal: support large backfills and migrations without interrupting live user traffic.

Current state:

1. The sequencer just happens to read from lower numbered logIDs completely before advancing to read anything from other logIDs.
2. Because the logIDs associated with a domainID can be dynamically provisioned by the admin API, clients should not have to know about logIDs.
3. We currently distribute load among logIDs. -- Something that does not play well with point Add a ProofNode protobuf to flesh out Merkle proofs #2.

Thoughts on design:
a) Perhaps clients should specify the maximum merge delay they want for their writes
b) The server could accept / reject these requests based on current load
c) The server could convert these requests into deadlines and use that for prioritization.
d) The current logID based system would force some kind of rough bucketing strategy

[mhutchinson]

If you want to support processing messages not necessarily in the order they are sent to the KT input queue via gRPC, then you'll need some mechanism to stack-rank messages for processing.

I'm assuming a goal is that KT evolves to to the point where you can run KeyTransparency-as-a-service (i.e. multi-tenant). That seems to be the reasoning behind multiple directory IDs. This rules out running client-defined prioritization code within the KT server.

So there is the requirement that a writer to KT can pass in advice on the QoS tier for a message being written, and that KT can interpret this advice in a well-defined way which is consistent across all of its tenants. This advice could be:

• log ID
• explicit enum
• a timestamp that the message should be merged before

There are a number of strategies that could be implemented given each of these implementation paths. These are listed in order of how decoupled they are from the implementation. Where to pick on this spectrum isn't clear without zooming out to consider the whole of the KT API when considered as a service. Happy to advise on this if my experience is useful, but I won't be able to justify prioritizing time to lead this work given my current focus.