HATracker: Add a local cache warmup on startup

[SungJin1212]

This PR introduces local cache warmup logic on HA tracker startup that fetches all keys from the KV store and warms the local cache.

Previously, whenever the Distributor started, it suffered from initial cold cache misses. Since the local map was empty, the HATracker treated every incoming request as a new entry. This caused unnecessary CAS operations to the KV store even for existing valid keys.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[friedrichg]

Do you see distributors in clusters with large number of pairs using more memory after this change? Do we need a flag for this?

[SungJin1212]

@friedrichg
I added a benchmark for syncKVStoreToLocalMap under various key size.
I ran a benchmark and the results indicate that the overhead is negligible even at a very large scale (10000 keys, about 18MB).

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/ha
cpu: Apple M4 Max
BenchmarkHATracker_syncKVStoreToLocalMap
BenchmarkHATracker_syncKVStoreToLocalMap/keys=100
BenchmarkHATracker_syncKVStoreToLocalMap/keys=100-14         	   10740	    113366 ns/op	  203197 B/op	    2532 allocs/op
BenchmarkHATracker_syncKVStoreToLocalMap/keys=1000
BenchmarkHATracker_syncKVStoreToLocalMap/keys=1000-14        	    1174	    973137 ns/op	 1834239 B/op	   24289 allocs/op
BenchmarkHATracker_syncKVStoreToLocalMap/keys=10000
BenchmarkHATracker_syncKVStoreToLocalMap/keys=10000-14       	     130	   9065987 ns/op	18199241 B/op	  240955 allocs/op
PASS

[friedrichg]

@SungJin1212 Thanks, this is great work.

[friedrichg]

I am not sure this is needed. as this is called before running the loop and starting the service.

[friedrichg]

Same here