Create aeolus_snapshot_proxy.proto

[jacob-curley-fnal]

An RPC to get all active alarms from AEOLUS. To be implemented by the acnet alarms bridge service and called from the unified alarms service on startup

[mguzman04]

This rpc means that there will be 2 data paths from the alarm bridge? One from the bridge to redis and one through gRPC. Would it make sense for the bridge to re-send through redis when this rpc is invoked?

[jacob-curley-fnal]

I mean, we already have to make the call. We either get the data we want in the response to the call, or we get an OK back from the call and then get the data from Redis. The second option means our Redis stream will likely need to know how to detect that existing ACNET alarms are stale, clear the cache, and repopulate with the new values from Redis. Alternatively, this call can be part of a dedicated startup process before we begin ingesting from Redis, so we only have one mode of ingesting from Redis

[mguzman04]

To be clear, I don't disagree with this implementation just playing devil's advocate. My understanding is that at startup we subscribe to redis and don't listen to old messages. The alarms bridge would only push active alarms either through the rpc call or through redis.

I'm naively thinking that going through redis would be simpler because that data path is already established. I could be wrong though

[jacob-curley-fnal]

No prob, glad for the feedback! Don't want to do something that's more complicated than necessary.

The problem is that Redis is not a reliable place to find the current, full set of alarming ACNET devices. Items in there have either a TTL or can get evicted if the stream hits max capacity. There could be a device in alarm where its notification has been kicked out of the Redis stream due to other devices adding messages on top of it. The only way to get the full, current picture is to ask AEOLUS (or manually poll all the ACNET devices, I guess). The alarms service is already going to have to be in a special mode when it boots, as that's the only time it will ever consume messages from Kafka (to get the EPICS states), so not too much of a stretch to add an extra "we only do this on boot" behavior. Once we get the current picture from both the AEOLUS (proxied through the alarms bridge) and Kafka, we then shift into "running" mode and consume new alarms from Redis, which are either ACNET or EPICS.

Does that clarify?

[mguzman04]

No further questions

[bstover74]

No prob, glad for the feedback! Don't want to do something that's more complicated than necessary.

The problem is that Redis is not a reliable place to find the current, full set of alarming ACNET devices. Items in there have either a TTL or can get evicted if the stream hits max capacity. There could be a device in alarm where its notification has been kicked out of the Redis stream due to other devices adding messages on top of it. The only way to get the full, current picture is to ask AEOLUS (or manually poll all the ACNET devices, I guess). The alarms service is already going to have to be in a special mode when it boots, as that's the only time it will ever consume messages from Kafka (to get the EPICS states), so not too much of a stretch to add an extra "we only do this on boot" behavior. Once we get the current picture from both the AEOLUS (proxied through the alarms bridge) and Kafka, we then shift into "running" mode and consume new alarms from Redis, which are either ACNET or EPICS.

Does that clarify?

Just to clarify, we agreed long ago that alarm-bridge-acnet would be the proxy between the alarms service and AEOLUS to get both the initial active alarms and alarms updates. I agree it would be cleaner to use a hybrid approach like you already mentioned to send initial active alarms through RPC and then get updates through our Redis stream based on start up or restart sequencing controlled by our alarms service.

[bstover74]

Looks good.