Skip to content

querier: support series batching from Store Gateways #7203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
SungJin1212 wants to merge 7 commits into
base: master
Choose a base branch
from
Open

querier: support series batching from Store Gateways #7203

SungJin1212 wants to merge 7 commits into from

Conversation

@SungJin1212
Copy link
Member

@SungJin1212 SungJin1212 commented Jan 12, 2026 •
edited
Loading

Add -querier.store-gateway-series-batch-size flag to configure the maximum number of series to be batched in a single gRPC response message from Store Gateways.

Benchmark results

BenchmarkThanosBucketStores_SeriesBatch

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/storegateway
cpu: Apple M4 Max
BenchmarkThanosBucketStores_SeriesBatch
BenchmarkThanosBucketStores_SeriesBatch/series_100
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=1
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=1-14         	    3273	    315173 ns/op	  260090 B/op	    4560 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=10
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=10-14        	    4335	    265844 ns/op	  234986 B/op	    3017 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=100
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=100-14       	    4653	    255788 ns/op	  203221 B/op	    2768 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=1000
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=1000-14      	    4700	    253746 ns/op	  210531 B/op	    2768 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=10000
BenchmarkThanosBucketStores_SeriesBatch/series_100/batchSize=10000-14     	    4394	    265009 ns/op	  286143 B/op	    2769 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_1000
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=1
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=1-14        	     656	   1782411 ns/op	 2282108 B/op	   41688 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=10
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=10-14       	     819	   1447217 ns/op	 2057748 B/op	   26274 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=100
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=100-14      	     886	   1326311 ns/op	 1763974 B/op	   23841 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=1000
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=1000-14     	     832	   1423374 ns/op	 2280276 B/op	   23535 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=10000
BenchmarkThanosBucketStores_SeriesBatch/series_1000/batchSize=10000-14    	     825	   1429912 ns/op	 2378003 B/op	   23535 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_10000
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=1
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=1-14       	      63	  18745491 ns/op	22518443 B/op	  423704 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=10
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=10-14      	      76	  15242607 ns/op	20108438 B/op	  268911 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=100
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=100-14     	      85	  13425516 ns/op	17205850 B/op	  244663 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=1000
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=1000-14    	      93	  12481578 ns/op	20406169 B/op	  241365 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=10000
BenchmarkThanosBucketStores_SeriesBatch/series_10000/batchSize=10000-14   	      81	  14098264 ns/op	18610246 B/op	  240819 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100000
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=1
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=1-14      	       6	 187283375 ns/op	245824413 B/op	 4249119 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=10
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=10-14     	       7	 154968667 ns/op	220216622 B/op	 2685126 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=100
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=100-14    	       8	 135575479 ns/op	190494851 B/op	 2443041 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=1000
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=1000-14   	       8	 125515792 ns/op	214022162 B/op	 2409687 allocs/op
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=10000
BenchmarkThanosBucketStores_SeriesBatch/series_100000/batchSize=10000-14  	       8	 128369026 ns/op	202337341 B/op	 2405048 allocs/op
PASS

BenchmarkParquetBucketStore_SeriesBatch

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/storegateway
cpu: Apple M4 Max
BenchmarkParquetBucketStore_SeriesBatch
BenchmarkParquetBucketStore_SeriesBatch/series_100
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=1
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=1-14         	    1928	    603369 ns/op	  514838 B/op	    6232 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=10
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=10-14        	    2332	    517278 ns/op	  483278 B/op	    4700 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=100
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=100-14       	    2413	    517015 ns/op	  448317 B/op	    4451 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=1000
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=1000-14      	    2323	    520959 ns/op	  469221 B/op	    4453 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=10000
BenchmarkParquetBucketStore_SeriesBatch/series_100/batchSize=10000-14     	    2230	    524052 ns/op	  627970 B/op	    4457 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_1000
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=1
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=1-14        	     578	   1995044 ns/op	 3731692 B/op	   47843 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=10
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=10-14       	     760	   1566651 ns/op	 3467065 B/op	   32516 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=100
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=100-14      	     817	   1480604 ns/op	 3138868 B/op	   30065 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=1000
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=1000-14     	     769	   1538239 ns/op	 3697080 B/op	   29755 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=10000
BenchmarkParquetBucketStore_SeriesBatch/series_1000/batchSize=10000-14    	     764	   1733054 ns/op	 3849235 B/op	   29757 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_10000
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=1
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=1-14       	      70	  19578657 ns/op	35824360 B/op	  463283 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=10
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=10-14      	      74	  13880345 ns/op	33793557 B/op	  309923 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=100
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=100-14     	      94	  10923364 ns/op	30925422 B/op	  285539 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=1000
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=1000-14    	     100	  10938252 ns/op	38345685 B/op	  282740 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=10000
BenchmarkParquetBucketStore_SeriesBatch/series_10000/batchSize=10000-14   	      92	  12059209 ns/op	33399931 B/op	  282462 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100000
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=1
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=1-14      	       7	 154174429 ns/op	321060982 B/op	 4616102 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=10
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=10-14     	       9	 117896125 ns/op	299043656 B/op	 3087348 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=100
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=100-14    	      10	 104118350 ns/op	269843749 B/op	 2840424 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=1000
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=1000-14   	      12	  98614431 ns/op	288224993 B/op	 2810725 allocs/op
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=10000
BenchmarkParquetBucketStore_SeriesBatch/series_100000/batchSize=10000-14  	      10	 100187800 ns/op	280080162 B/op	 2806721 allocs/op
PASS

Process finished with the exit code 0

Which issue(s) this PR fixes:
Fixes #7202

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@SungJin1212
fix lint
7331d3c 
Signed-off-by: SungJin1212 <[email protected]>
@SungJin1212
Add flag to experimental features
a753795 
Signed-off-by: SungJin1212 <[email protected]>
yeya24
yeya24 reviewed Jan 12, 2026
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to upgrade to thanos-io/thanos#8629 to include it as a fix

CHANGELOG.md Outdated
* [FEATURE] Distributor: Add a per-tenant flag `-distributor.enable-type-and-unit-labels` that enables adding `__unit__` and `__type__` labels for remote write v2 and OTLP requests. This is a breaking change; the `-distributor.otlp.enable-type-and-unit-labels` flag is now deprecated, operates as a no-op, and has been consolidated into this new flag. #7077
* [FEATURE] Querier: Add experimental projection pushdown support in Parquet Queryable. #7152
* [FEATURE] Ingester: Add experimental active series queried metric. #7173
* [FEATURE] Querier: Add `-querier.store-gateway-series-batch-size` flag to configure the maximum number of series to be batched in a single gRPC response message from Store Gateways. #7203
Copy link
Contributor

@yeya24 yeya24 Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just call it as an enhancement not a new feature

@yeya24
Copy link
Contributor

yeya24 commented Jan 12, 2026

Batched server response is not supported in parquet gateway, correct? I don't think we use the batchableServer which does the batching

yeya24
yeya24 approved these changes Jan 13, 2026
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yeya24 yeya24 yeya24 approved these changes

Assignees

No one assigned

Labels

component/querier size/XL type/feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Store Gateway: Enable series response batching

2 participants

@SungJin1212 @yeya24