Use prometheus/tsdb in ingester.

[tomwilkie]

This PR is the first attempt at using prometheus/tsdb in the ingester.

There is lots missing/TODO:

• Add series limits to TSDB to match existing Cortex limits.
• Upstream TSDB changes for exposing chunks on series interface.
• Figure out how to do per user metrics for TSDB (number of series, ingestion rate etc).
• Ensure TSDB errors for things like out-of-order etc are being handled correctly.
• Figure out how to handle transfers for TSDB.
• Figure out how to do upgrades.
• Implement flushing TSDB to object store
• Allow index entries to point to sub-objects
• Benchmark Bigtable vs in-object-store index.

[codesome]

I can pick TSDB changes to be made 1 by 1, starting with series limit.

[codesome]

Limit series in TSDB: prometheus-junkyard/tsdb#617

[cstyan]

Figure out how to do per user metrics for TSDB (number of series, ingestion rate etc). Could we achieve this by just having a separate TSDB data dir per user ID?

Edit: I think we could then extend that to just make it easy to get the # of active series in TSDB (looks like right now we just have the gauge metric) rather than having to add a limit to TSDB itself, and then Cortex can ask TSDB for that value when checking if there's room for more series for a given user?

[codesome]

@cstyan Yes, after I opened the PR in TSDB to have limits on number of series, it looks unnecessary to have it in TSDB. We should be able to handle that in cortex (with or without TSDB per user ID).
Also, wouldn't it be too many database instances to handle if we have TSDB per user ID?

[bwplotka]

Where is this coming from? I cannot see this on tip of TSDB 🤔 nor 0.8.0 which go mod vendors here.

Is this manually modifed TSDB code in vendor? Cannot see any replace directive on go.mod as well so you don't use fork as well...

[tomwilkie]

I just added it here; opening a TSDB PR now.

[tomwilkie]

prometheus-junkyard/tsdb#659

[tomwilkie]

Closing in favour of #1695.