Document best practices for Eventual Consistency

[niko5]

I am wondering if there are any best practices for implementing the UI where there is eventual consistency with Neos.EventSourcing and Neos.Flow.

At the moment with immediate consistency I post a Command, from a Fluid form, which is handled by the Command Action Controller this either loads the aggregate directly (or using a CommandHandler) completes the request then I am redirect to the query side which loads the page with the updated view. This all works fine when everything is synchronous.

However when moving to asynchronous listeners and projections what should one do? Here are some ideas of mine at the moment.

1. You could do the same as above but put a delay time with the redirect (defaults to 0) which may give the read model a chance to update - but this is effectively blocking.
2. A slightly more sophisticated way is to perhaps get the aggregate id and version number prior to making the call on the aggregate. This gets stored in the read model (somehow) and then the wait on the redirect until the aggregate version has increased or we time out .
3. Don't have a redirect and return a Response object from the Action Controller with either OK response (200) with the Aggregate id and version prior to the update (the command is executed synchronously) or an error say 406 if Aggregate constraints are violated. This will then leave it for the UI to decide what to do eg it decides what view to call again perhaps using the the Aggregate version and id to make sure it is unto date.
   ...

These are just some ideas that I am kicking around

[bwaidelich]

@niko5 Thanks for this. I took the freedom to rephrase the title and format your message slightly

I have some more ideas floating around and will try to formulate them asap

[albe]

Hi niko, thanks for the question! Though this is not specific to Neos.EventSourcing/Flow, it is a very fundamental question to any Event Sourced/Eventually Consistent system and here are some answers and possible solutions:

1. You could do the same as above but put a delay time with the redirect (defaults to 0) which may give the read model a chance to update - but this is effectively blocking.

Well, yes, but no, not really. Really, don't do this. It doesn't solve the "problem" at all and makes your whole application less responsive and wonky.

2. A slightly more sophisticated way is to perhaps get the aggregate id and version number prior to making the call on the aggregate. This gets stored in the read model (somehow) and then the wait on the redirect until the aggregate version has increased or we time out.

You are on a good track here. You anyway need the (new) aggregate version number after a command in order to implement optimistic concurrency checks, so it's a good thing to return that version from the command handler. The same aggregate version can be used to query your read model with a constraint "at least version X". The main issue here comes with multi-aggregate ReadModels. Once you solved this you can then do this query in different ways - either (long) polling or even via websockets and push. Note here, that this query is the GET request you redirect to after your command handler, e.g. acme/tasks/userform/?v=41. It could return immediately displaying a "processing..." placeholder and then do the polling for the updated data on the client, or just wait itself in a loop until the readmodel is at the expected version (the UX suffers though, because it might take forever until the read model reaches that version and the request will just time out).

Here are some other approaches to the issue at hand, that are maybe non-obvious:

• Do not return to the same page where your user will see the data he entered before again. Does your user even want that? Maybe you're in a process that involves multiple steps, so just move him forward.

• Or just show a "Congratulations!" / "Thank you!" screen that affirms his previous action and gives him details on what he is supposed to expect next. Maybe he'll have to wait for an e-mail or something else that is async in nature anway.

• Is it even necessary that your user sees the most up-to-date data or is it enough if he just sees the "most recent available data"? Maybe just make the "version" of the data visible to the user, either by a timestamp of "last edited" or the actual "version number". He can then either reload the page, or maybe you do some polling in the client to automatically update the page once new data is available. That would also suddenly make the page work in a multi-user environment nicely.

[bwaidelich]

One very common requirement is the redirect to a newly created resource (e.g. signup).
The tricky part is that only the Read side knows whether it is "ready".
For that some kind of blocking mechanism is needed.
I didn't find any literature about that yet and I guess it really depends on the use case, but here are some options I can think of:

1. avoid the immediate display (like @albe suggested) – but keep in mind that even a "thank you, click here to go to your newly generated profile" screen might not work if the user clicks too fast.
2. carry the stream version (or sequence number) to the read model and implement s.th. like $someFinder->findByIdWithVersion($id, $version) (like suggested above)
3. display client data until it's consistent (that's what Facebook and stackoverflow did for comments apparently: render the comment on the client side and replace it once available on the server side)
4. implement blocking on the write side. We did that for the ES CR (for now) where we allow commands to be sent so that they wait until certain projectors handled all the events that were dispatched by the corresponding command

[niko5]

Another option that I am interested in is Server-Send events and this is for the situation where there is a multi user web app that requires realtime background updates. However I am not sure if this is possible with Flow.

[bwaidelich]

Server-Send events

you mean sth like websockets?

[niko5]

Not web sockets Server Sent is a one way from server to client - seems easier than web sockets and fits quiet well with event sourcing . so I am thinking effectively of a controller which is a projector pushing events to the http client. Here is a link on server sent events

[bwaidelich]

Ah, thanks for clarifying. The foundation would be similar though: A mechanism that blocks until some projection is up-to-date. But sever sent events would make a nice usecase of course!

[bwaidelich]

FYI: (somewhat hacky) example for a ServerSent HTTP component: https://gist.github.com/bwaidelich/045a53c11d4b49e2a1a849185819493a

[niko5]

Thanks 😊 I will take a look and test it out 😄