LDEV-6327 rewrite near cache as map+queue (also fixes LDEV-4413 write-side aliasing)

[zspitzer]

The bugs

Two related correctness bugs in the redis cache near-cache layer:

• LDEV-4413: cacheGet returns a shared reference to the cached value. Callers (or other concurrent readers) can mutate it, corrupting the cached state. David Rogers's PR copy NearCacheEntry values as-if they had been materialized from cache #16 fixes the read side; the write side (concurrent puts of mutable objects) was unfixed.
• LDEV-6327: When two puts to the same key sit in the deque waiting to drain, the head-first scan in Storage.get returns the older entry. Reads can see stale writes for up to the drain window — deterministic 78/100 repro on master.

This PR also resolves

Ticket	Issue
LDEV-4413	Live ref aliasing
LDEV-6327	Stale-wins on duplicate puts
LDEV-2135	Session vars lost after thread.join(), sessionCluster=true
LDEV-4408	application action="update" losing session vars
LDEV-6046 Redis path	sessionInvalidate() not clearing Redis
GH #13	Same as LDEV-4413
GH #5 concern #2	Same as LDEV-6046 Redis path

The root cause

Storage was the wrong data structure:

ConcurrentLinkedDeque<NearCacheEntry> entries;

NearCacheEntry get(byte[] bkey) {
    Object[] arr = entries.toArray();      // allocates Object[] of every pending entry
    for (Object obj: arr) {                 // O(N) byte[] equality scan
        NearCacheEntry e = (NearCacheEntry) obj;
        if (equals(e.getByteKey(), bkey)) return e;
    }
    return null;
}

Every single cacheGet allocated an Object[] snapshot of the deque and linear-scanned it. With the deque holding 1000 pending entries, that's ~50µs of pure CPU per get — slower than just hitting Redis. The near cache was actively slowing things down vs not having one.

JFR on master, full bench suite:

• Arrays.copyOf(Object[]) — 59% of CPU (inside deque.toArray)
• Storage.equals(byte[], byte[]) — 25% of CPU
• ConcurrentLinkedDeque.toArrayInternal — 9% of CPU
• Object[] — 95% of all allocation pressure

The fix

Replace the deque with a map keyed by content-equality wrapper, plus a separate queue for drain order:

ConcurrentHashMap<ByteArrayWrapper, NearCacheEntry> entries;  // O(1) lookup, overwrites on duplicate put
ConcurrentLinkedQueue<ByteArrayWrapper> drainQueue;           // FIFO for the drain thread

Storage.put:

• Eagerly serialise val → byte[] at call time. Captures an immutable snapshot before the caller can mutate. Closes LDEV-4413 write-side.
• entries.put(wkey, entry) overwrites the prior entry. Closes LDEV-6327 — there's no stale entry left to find.
• drainQueue.offer(wkey) schedules the write to Redis.

Storage.get:

• entries.get(wrapper). O(1). No Object[]. No scan.

Drain thread:

• Polls drainQueue, re-reads the latest entry from the map (a newer put may have overwritten), writes pre-serialised bytes directly to Redis via a new putBytes overload (no re-serialise), conditional-removes from the map.

Three commits, telling the story in order:

1. David Rogers — add failing test (PR copy NearCacheEntry values as-if they had been materialized from cache #16)
2. David Rogers — copy NearCacheEntry values as-if they had been materialized from cache (PR copy NearCacheEntry values as-if they had been materialized from cache #16, fixes LDEV-4413 read side)
3. (this PR) — LDEV-6327 replace near cache deque scan with map+queue (fixes LDEV-6327 + LDEV-4413 write side)

David's authorship preserved on his commits.

The evidence

Correctness (deterministic testbed scenarios, 16-thread contention):

	master	PR #16 only	this PR
LDEV-4413 read aliasing (50 trials)	22 fail	0 fail	0 fail, 50 pass
LDEV-4413 write aliasing (50 trials)	50 fail	50 fail	0 fail, 50 pass
LDEV-6327 stale-wins (100 trials)	78 stale	91 stale	0 stale, 100 pass

Throughput (ABBA, 100k cycles, 16 threads, full suite):

Test	master ops/s	this PR ops/s	Δ
put-get-different	1,370	108,542	+6510%
put-get-same	4,421	112,829	+2452%
duplicate-put	4,815	83,795	+1640%
hot-key-duplicate-put	18,714	110,232	+489%
get-burst	2,679	10,261	+283%
large-struct	62,602	73,119	+17%
put-small	99,436	113,000	+14%
hot-key-put	101,736	114,261	+12%
hot-key-write-burst	86,342	96,277	+12%
get-hot	87,592	88,009	+0.5%

No regressions anywhere. The multipliers measure the bug-path tax in isolation; the synthetic 100k+ ops/s assumes the JVM is dedicated to cache calls. In production, expect lower per-op CPU, ~10× less GC pressure, and better tail latency under cache contention — not literal 60× application-level speedup.

JFR on the same suite confirms master spends ~93% of CPU on the deque-scan path; this PR has 0% there. Full bench suite finishes in 55s on this PR vs 185s on master.

Tests: extension test suite (./test.bat) — 27/27 pass on Lucee 7.1, 22/22 on Lucee 7.0. PR #16's own regression test (NearCacheEntriesAreCopiedOnReadPriorToWriteCommit) still passes.

Trade-offs

• cachePut now serialises on the caller thread instead of the drain thread. Necessary for the write-side fix — the snapshot must be taken before the caller can mutate. Bench shows pure-put throughput goes up anyway, because the drain stops being a bottleneck.
• Storage.put now throws IOException on serialise failure. Master swallowed these in the drain thread. Fail-fast is correct; signature unchanged at the public Cache#put boundary which already declares throws IOException.
• One ByteArrayWrapper allocated per cacheGet for the map lookup. ~16 bytes, JIT likely stack-allocates via escape analysis. Trivial vs the Object[] snapshot of 1000 entries this replaces.