Feat/cache distributed lock - #27
Open
swsarancodes wants to merge 2 commits into
Open
Conversation
ThreadedMotoServer binds 0.0.0.0 when given port=0, and every moto-backed test fixture built its endpoint_url straight from that bind address. Linux treats 0.0.0.0 as reachable from a client socket; Windows does not, so every SQS/S3/KMS/SES/SNS/Secrets-Manager test using this pattern hung or failed with EndpointConnectionError on Windows.
Adds cross-instance mutual exclusion to the cache abstraction - useful for
leader election, cron dedup across replicas, and idempotent job claiming,
none of which cloudrift previously had an answer for.
async with cache.lock("invoice:42:close"):
... # only one process/replica runs this at a time
Implemented once in _RedisMixin (cache/base.py), so all three Redis
backends - StandaloneRedisBackend, AWSElastiCacheBackend,
AzureRedisCacheBackend - get it for free, matching how every other Redis
command in this codebase is implemented once and shared.
Design notes:
- Every acquisition gets a random fencing token (SET key token NX PX ttl).
Release and extend are conditional on that token via WATCH/MULTI
compare-and-mutate, not a bare DEL/PEXPIRE - so a caller can only ever
act on a lock it currently holds, never one that expired and was
re-acquired by someone else. This closes the classic bug naive
set-then-delete locks have.
- WATCH/MULTI instead of a Lua script (EVAL) so the same code path works
against managed Redis offerings that restrict scripting and against
fakeredis in tests (fakeredis has no EVAL support at all).
- auto_extend=True (default) runs a background watchdog that refreshes the
TTL at roughly ttl/3 for as long as the `async with` block is running,
so a critical section that runs longer than expected doesn't have its
lock expire and get stolen mid-operation. The watchdog is cancelled the
moment the block exits, on success or exception.
- extend_lock()/release_lock() are exposed directly for callers managing a
lock's lifetime by hand instead of via the context manager.
- Backends that can't offer atomic conditional writes fail loudly with
NotImplementedError, matching the existing fail-loud convention for
abstraction gaps (see Azure Service Bus delete()).
12 new tests cover: release-on-exit, release-on-exception, contention,
blocking handoff between two waiters, lock-key/cache-key namespace
isolation, fencing-token rejection of a stale holder, idempotent release,
manual extend, auto-extend surviving a critical section longer than its
base ttl, watchdog cancellation on release, and the default
NotImplementedError contract. All run against fakeredis - no real Redis
needed.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds distributed locking to the cache using Redis, so only one process can run a critical section at a time.
Includes safe token-based release/extend, auto TTL refresh, manual lock controls, and 12 new tests covering contention, expiry, exceptions, and watchdog behavior.
Also fixes a Windows test issue where moto fixtures were connecting to
0.0.0.0instead of127.0.0.1.