timeout

[biskit]

i have a call that's initiated via SyncRead() that has to create some data before returning. hence it issues a SyncPropose() call once a while results in either the SyncRead resulting in a timeout or other SyncPropose calls resulting in a timeout (in the same shardID)

is there any sample on how to hande timeout's correctly?

[lni]

sorry but I am not sure whether I understand your issue. could you please provide a concrete example on the sequence of calls you want. thanks.

[biskit]

the ratio of my reads to writes is around 100:1. some of these reads have to do a write (as they generate data) and that has to be rafted. it's all fine until there's a long write transaction taking place and many reads hammer the same shard for getting an opportunity to write

the one holding the lock and currently writing (the long transaction) suddenly takes much longer to complete than if the reads did not hammer it requesting a lock for it to some writing

it seems to me like there's some kind of a spin lock situation but I could be wrong. i am writing what I am observing and happy to provide more info though, being able to replicate with a sample might be a lot of work...

[ys-d]

I suspect the reason is that each shard is handled by a single coroutine.
If a time-consuming write request occupies that coroutine, subsequent requests scheduled to that shard will be blocked and queued.

[lni]

the ratio of my reads to writes is around 100:1. some of these reads have to do a write (as they generate data) and that has to be rafted. it's all fine until there's a long write transaction taking place and many reads hammer the same shard for getting an opportunity to write

For a read op that requires write, can you provide the exact sequence of dragonboat API calls? Thanks.

[biskit]

i think what's happening is bound to happen if the running SyncPropose is executing a long transaction that cannot be broken down to smaller ones and still be idempotent

i have set the timeout for the read initiated SyncPropose to be smaller than the write initiated one so that the caller will atleast get a timeout or or some other response that will make them retry their request

this seems like a good compromise. like to hear your thoughts on it

[lni]

which type of StateMachine is used in your code? is it IStateMachine?

please note that IStateMachine uses a RWMutex internally to guard write and read accesses, while the other two types don't use such RWMutex. IConcurrentStateMachine is called concurrent for this reason. please see the godoc comments in the statemachine folder for more details.

[biskit]

IOnDiskStateMachine

[lni]

i have set the timeout for the read initiated SyncPropose to be smaller than the write initiated one

so you have SyncRead() that might start SyncPropose() and SyncPropose() calls that might start SyncPropose() as well?

what do you actually mean "write initiated one". thanks.

[lni]

I believe -

1. you can't initiate a SyncPropose() in your SM's Update() method (i.e. SyncPropose() initiated by another SyncPropose()), as both of those two SyncPropose() calls will have to wait for the other one to complete first.

2. you can initiate a SyncPropose() in your SM's Lookup() method (i.e. SyncPropose() initiated by SyncRead()).

3. you can initiate a SyncRead() in your SM's Lookup() method (i.e. SyncRead() initiated by SyncRead())

4. you can initiate a SyncRead() in your SM's Update() method (i.e. SyncRead() initiated by SyncPropose())

when you look deeper -

2 and 3 are okay, because Lookup() is always executed in user goroutine, they won't block the completion of SyncPropose() and SyncRead().

1 is not possible, it will always cause the SyncPropose() initiated in SM's Update() method to timeout.

4 is okay, as the SyncRead() will be able to complete even when the SM execution goroutine is busy (running your SM's Update() method).

Note that the above analysis assumes that all initial operations are initiated by users code from user goroutine.

[biskit]

so you have SyncRead() that might start SyncPropose() and SyncPropose() calls that might start SyncPropose() as well?

i have a SyncRead() initiate a SyncPropose(). i don't have a SyncPropose() call another SyncPropose()

what do you actually mean "write initiated one". thanks.

what I mean is when the code does an SyncPropose() (compared to code doing a SyncRead() that calls SyncPropose()) I have a higher context timeout value than in the latter case

[biskit]

I believe -

1 you can't initiate a SyncPropose() in your SM's Update() method (i.e. SyncPropose() initiated by another SyncPropose()), as both of those two SyncPropose() calls will have to wait for the other one to complete first.
2 you can initiate a SyncPropose() in your SM's Lookup() method (i.e. SyncPropose() initiated by SyncRead()).
3 you can initiate a SyncRead() in your SM's Lookup() method (i.e. SyncRead() initiated by SyncRead())
4 you can initiate a SyncRead() in your SM's Update() method (i.e. SyncRead() initiated by SyncPropose())

mine is case 2

[lni]

i have a SyncRead() initiate a SyncPropose()

this should be fine.

if you read nodehost.go on how SyncRead() is implemented, it is pretty straight forward. It waits for the ReadIndex operation to complete first and then call SM's Lookup() from your user goroutine (the one which called SyncRead()). In your case SyncPropose() is called inside SM's Lookup(). This should not be anything different from calling SyncPropose() directly from your user goroutine.

[lni]

i think what's happening is bound to happen if the running SyncPropose is executing a long transaction that cannot be broken down to smaller ones and still be idempotent

Is your above mentioned "the running SyncPropose" initiated by SyncRead()?

What can be useful here is to describe what is exactly happening, probably using a minimum example if possible, please include -

• the sequence of events
• how many concurrent SyncPropose() and SyncRead() calls.
• how those SyncPropose() and SyncRead() calls interact with each other including their dependencies.
• when hit with timeout, does it always happen or just some % of calls end up in timeout.

[biskit]

Is your above mentioned "the running SyncPropose" initiated by SyncRead()?
No

I'm sorry to make you iterate with questions... i believe i have described the case pretty clearly however i have not provided a demonstrator. i believe the issue is clear:

• a function calls SyncPropose() - a write call
• this executes a long transaction
• another function calls SyncRead() - a read call
• this inturn calls SyncPropose() because it needs to create some data as part of the call
• since they are on the same shard the second call to SyncPropose() waits for first one to finish
• i set different context timeouts because the user expects the read call to finish faster one way or other (data or error response)
• the SyncRead() caller returns timeouts when the first one is in a long transaction

i come to understand there's nothing wrong with dragonboat behavior because SyncPropose is mutex locked

[lni]

@biskit

Thanks a lot for the full details. I got initially confused as I kind of totally forgot how ReadIndex works and my hallucination version of ReadIndex caused me to feel really strange on how your SyncRead() could also timeout.

In your case, assuming your first SyncPropose() and SyncRead() are issued by two goroutines at roughly the same time -

if your proposal is committed before the ReadIndex (started by your SyncRead()) operation's read of the commit value, you will see your SyncRead() to timeout, as the ReadIndex protocol have to wait for that proposal to be applied (i.e. for your long transaction to complete) first.

if your proposal is committed after the ReadIndex operation's read of the commit value, your ReadIndex is likely to complete successfully allowing your SM's Lookup() to initiate that SyncPropose(). That second write is likely to be committed after your first SyncPropose(), this will in turn cause your second write to timeout as the SM's Update() method have to apply your long transaction first because such sequential update rule is required by the Raft protocol.

The way how you set timeout seems to be fine to me. I'd probably suggest to review your design to divide that long transactions into smaller pieces if possible.