Read cache growth

[esteinig]

There is some missing uncache instruction that is not considered - the read cache grows over time. It looks like in the off target viral unblock percent, that the unblocking is not as effective as it could be (for the Monkeypox genome) - is this related?

[esteinig]

Seems to happen more pronounced when specifying target regions hmmm

[esteinig]

There may be a small number of reads that get issued a Proceed decision, but have already completed and no further chunks are returned from the control server. These may get stuck in the cache - checked the chunk sizes of cache-stuck reads and they are all within minimum/maximum chunk size bounds. This means that no further decisions - including uncache decisions - are made for these reads (they are also not resubmitted for basealling because no request with the correct identifier retrieves them from the cache)...

[esteinig]

I think these pop up in the off-target pass read summary when using a two-way test (test_viruses) and targeting a complete sequence, since these reads are short (342 bp median, 670 bp mean) and numbers match roughly - they would get written out by Icarust

[esteinig]

Might need the timed read cache again...

[esteinig]

This is crucial to address. I re-implemented the moka cache with lifetime of entries to TTL 5 minutes and TTI 3 minutes for periodic clearing of cache, otherwise cache overflow could become a problem in long running experiments. Introduces a slight penalty to unblocked percentage in test_virus but did not incur latency cost compared to the ArcMutex HashMap cache

Looks like somewhat unacceptable latency in the test_human with TMEM41B target config... hmmm... gotta test same setting with HashMap cache... (1024 pores)

So weird - in the HashMap implementation median bp is much smaller (319 vs 400 bp) but the reads passed for off target are 1/3 higher (912 vs 612) with as much bases increase plus less percent off-target unblocked (- 1%) and 5 on-target reads vs 8 compared to the Moka run. So this means that the Moka version makes some unblock decisions later (leading to longer median/mean unblocked lengths). Yet the HashMap version makes decisions quicker, but for some does not come to a decision in time and passes these reads/lets them exceed max chunk size. So weird... does that make sense? Would that mean the HashMap ArMutex locking might be too slow for these reads, letting them through entirely whereas the Moka version catches them in time because its caching is more efficient?

I think for experiment efficiency maybe we should not optimize for latency then due to read caching interfering - latency/unblocked read length may not be a good measure of this?

Same experiment with 512 pores - still same patterns i.e. longer unblocked but not as drastic a difference (292 HM vs 314 MK median, 319 vs. 354 mean), higher percent unblocked (97.03 HM vs 98.17 MK) and 1/3 less off-target reads through...

Need more testing to see what's going on here.

[esteinig]

Going with the Moka cache for now - necessary for long runs - can consider a custom cache cleaning for HashMap implementation, but Moka seems simpler for now. Will need more testing.

[esteinig]

• move cache to post-basecalling to store reads not signal for sequential passes (Ryan's excellent suggestion)