feat: Introduce sst file format for btree global index #49

ChaomingZhangCN · 2026-01-07T05:36:40Z

Purpose

This PR only covers basic reading and writing of SST format files, data compression encoding and caching are not considered. Too much code in the PR will make it difficult to review, so I will continue work for this in the future.

DONE:

cache interface
memory slice & memory slice input & memory slice output
sst format file write & read
segment based bloomfilter & bitset

TODO:

block cache: lru?
compression: lz4, zstd, etc.(simd implementaion?)

Tests

./build/debug/paimon-common-sst-file-format-test
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from SstFileIOTest
[ RUN ] SstFileIOTest.TestSimple
[ OK ] SstFileIOTest.TestSimple (2 ms)
[----------] 1 test from SstFileIOTest (2 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (2 ms total)
[ PASSED ] 1 test.

./build/debug/paimon-common-test
[----------] 1 test from BitSetTest
[ RUN ] BitSetTest.TestBitSet
[ OK ] BitSetTest.TestBitSet (0 ms)
[----------] 1 test from BitSetTest (0 ms total)

[----------] 5 tests from BloomFilterTest
[ RUN ] BloomFilterTest.TestOneSegmentBuilder
[ OK ] BloomFilterTest.TestOneSegmentBuilder (0 ms)
[ RUN ] BloomFilterTest.TestEstimatedHashFunctions
[ OK ] BloomFilterTest.TestEstimatedHashFunctions (0 ms)
[ RUN ] BloomFilterTest.TestBloomNumBits
[ OK ] BloomFilterTest.TestBloomNumBits (0 ms)
[ RUN ] BloomFilterTest.TestBloomNumHashFunctions
[ OK ] BloomFilterTest.TestBloomNumHashFunctions (0 ms)
[ RUN ] BloomFilterTest.TestBloomFilter
[ OK ] BloomFilterTest.TestBloomFilter (2 ms)
[----------] 5 tests from BloomFilterTest (2 ms total)

lxy-9602 · 2026-01-08T12:34:00Z

src/paimon/common/io/cache/cache.cpp

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2024-present Alibaba Inc.
+ *


Please update the copyright year from 2024 to 2026 for new files.

lxy-9602 · 2026-01-08T12:38:30Z

src/paimon/common/io/cache/cache.h

+class CacheValue {
+ public:
+    explicit CacheValue(std::shared_ptr<MemorySegment>& segment) : segment_(segment) {}
+


const std::shared_ptr& segment may be better.

lxy-9602 · 2026-01-08T12:40:13Z

src/paimon/common/io/cache/cache.h

+        std::function<std::shared_ptr<CacheValue>(const std::shared_ptr<CacheKey>&)> supplier) = 0;
+
+    virtual void Put(const std::shared_ptr<CacheKey>& key, std::shared_ptr<CacheValue>& value) = 0;
+


const std::shared_ptr& value may be better.

lxy-9602 · 2026-01-08T12:44:28Z

src/paimon/common/io/cache/cache_key.cpp

+    size_t seed = 0;
+    seed ^=
+        std::hash<std::string>{}(file_path_) + 0x9e3779b97f4a7c15ULL + (seed << 6) + (seed >> 2);
+    seed ^= std::hash<int64_t>{}(position_) + 0x9e3779b97f4a7c15ULL + (seed << 6) + (seed >> 2);


Consider converting 0x9e3779b97f4a7c15ULL to a static constant

lxy-9602 · 2026-01-08T12:47:47Z

src/paimon/common/io/cache/cache_manager.cpp

+    std::function<Result<MemorySegment>(const std::shared_ptr<CacheKey>&)> reader) {
+    auto& cache = key->IsIndex() ? index_cache_ : data_cache_;
+    auto supplier = [&reader](const std::shared_ptr<CacheKey>& k) -> std::shared_ptr<CacheValue> {
+        auto segment = reader(k).value();


Capturing &reader by reference may risk memory leaks. May consider capturing by value instead. Also ensure Result is .ok() before calling .value() to avoid undefined behavior.

lxy-9602 · 2026-01-08T13:47:57Z

src/paimon/common/sst/block_aligned_type.h

+        return BlockAlignedType::UNALIGNED;
+    } else {
+        throw std::invalid_argument("Invalid block aligned type: " + std::to_string(v));
+    }


Also, may return Result<BlockAlignedType>

lxy-9602 · 2026-01-08T13:49:42Z

src/paimon/common/sst/block_cache.h

+    BlockCache(std::string& file_path, std::shared_ptr<InputStream>& in,
+               std::shared_ptr<MemoryPool>& pool, std::unique_ptr<CacheManager>& cache_manager)
+        : file_path_(file_path), in_(in), pool_(pool), cache_manager_(std::move(cache_manager)) {}
+


may pass const shared_ptr& or unique_ptr&&

lxy-9602 · 2026-01-08T14:02:48Z

src/paimon/common/sst/block_entry.h

+ public:
+    BlockEntry(std::shared_ptr<MemorySlice>& key, std::shared_ptr<MemorySlice>& value)
+        : key_(key), value_(value) {}
+


Consider making BlockEntry a struct if it only has Get methods.

lxy-9602 · 2026-01-09T05:46:11Z

src/paimon/common/sst/block_handle.h

+
+ public:
+    static constexpr int32_t MAX_ENCODED_LENGTH = 9 + 5;
+


Can you add a comment explaining the value of MAX_ENCODED_LENGTH?

lxy-9602 · 2026-01-09T05:48:21Z

src/paimon/common/sst/block_handle.cpp

+    auto output = std::make_shared<MemorySliceOutput>(MAX_ENCODED_LENGTH, pool);
+    output->WriteVarLenLong(offset_);
+    output->WriteVarLenLong(size_);
+    return output->ToSlice();


I'm not sure if reading uses an int followed by a long (ReadVarLenLong for offset and ReadVarLenInt for size), while writing uses two longs. Is this correct? Consider adding a test to verify.

lxy-9602 · 2026-01-09T05:58:20Z

src/paimon/common/sst/block_iterator.cpp

+std::unique_ptr<BlockEntry> BlockIterator::Next() {
+    if (!HasNext()) {
+        throw std::invalid_argument("no such element");
+    }


May use bad status instead of exceptions.

lxy-9602 · 2026-01-09T06:07:18Z

src/paimon/common/sst/block_iterator.cpp

+    }
+
+    return false;
+}


Please add a test for BlockReader and BlockIterator.

lxy-9602 · 2026-01-09T06:09:37Z

src/paimon/common/sst/block_reader.h

+class BlockIterator;
+
+/** Reader for a block. */
+class BlockReader : public std::enable_shared_from_this<BlockReader> {


Consider using /// for class or interface comments to maintain a consistent format.

lxy-9602 · 2026-01-09T06:20:36Z

src/paimon/common/sst/block_reader.cpp

+        auto data = block->Slice(0, index_offset);
+        auto index = block->Slice(index_offset, index_length);
+        return std::make_shared<UnAlignedBlockReader>(data, index, comparator);
+    }


Can 5 be a static constant rather than a magic number? Or add comments for it.

lxy-9602 · 2026-01-09T06:23:57Z

src/paimon/common/sst/block_trailer.cpp

+std::string BlockTrailer::ToString() const {
+    return "BlockTrailer{compression_type=" + std::to_string(compression_type_) + ", crc32c_=0x" +
+           std::to_string(crc32c_) + "}";
+}


May add a conversion to ensure the return value is a hexadecimal string for crc32c.

lxy-9602 · 2026-01-09T06:29:51Z

src/paimon/common/sst/block_writer.cpp

+    if (aligned_) {
+        int currentSize = endPosition - startPosition;
+        if (aligned_size_ == 0) {
+            aligned_size_ = currentSize;


current_size & end_position & start_position

lxy-9602 · 2026-01-09T06:50:08Z

src/paimon/common/sst/sst_file_writer.h

+    BlockWriter* IndexWriter() const {
+        return index_block_writer_.get();
+    }
+


Test interface can be private.

lxy-9602 · 2026-01-09T06:51:27Z

src/paimon/common/sst/sst_file_writer.h

+    std::shared_ptr<Bytes> last_key_;
+
+    const std::shared_ptr<MemoryPool>& pool_;
+


Ensure that obtaining a shared_ptr reference does not introduce any risk of memory leaks？

lxy-9602 · 2026-01-09T06:54:04Z

src/paimon/common/sst/sst_file_writer.cpp

+    if (!handle.ok()) {
+        return handle.status();
+    }
+


May use PAIMON_ASSIGN_OR_RAISE() (ps: Ensure that the macro PAIMON_ASSIGN_OR_RAISE always includes the return type not auto ).

lxy-9602 · 2026-01-09T07:09:12Z

src/paimon/common/utils/bloom_filter.h

+
+ private:
+    static constexpr int32_t BYTE_SIZE = 8;
+


Can the existing BloomFilter64 and BitSet classes in paimon/common/utils/ meet your requirements for bloom filter file index?

lxy-9602 · 2026-01-09T07:10:40Z

src/paimon/common/io/cache/cache_key.h

+
+    bool IsIndex() override;
+
+    int64_t Position();


Get method can be const.

Copilot

Pull request overview

This PR introduces the SST (Sorted String Table) file format infrastructure for the btree global index. It implements core read/write functionality for SST files including memory management utilities (MemorySlice, MemorySliceInput/Output), data structures (BloomFilter, BitSet), block-based storage (BlockWriter/Reader, BlockIterator), and a basic cache interface. The implementation focuses on fundamental functionality, deferring compression encoding and LRU caching to future work.

Changes:

Added memory slice abstractions for efficient I/O operations on memory segments
Implemented SST file format with block-based storage including writer, reader, and iterator components
Introduced BloomFilter and BitSet data structures for efficient key lookups
Added cache interface with placeholder NoCache implementation

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
src/paimon/common/utils/bloom_filter.{h,cpp}	Segment-based Bloom filter implementation for probabilistic key membership testing
src/paimon/common/utils/bloom_filter_test.cpp	Comprehensive tests for Bloom filter including hash functions and multi-segment scenarios
src/paimon/common/utils/bit_set.{h,cpp}	Memory segment-backed bit set implementation used by Bloom filter
src/paimon/common/utils/bit_set_test.cpp	Tests for BitSet operations including set, get, and clear
src/paimon/common/sst/sst_file_writer.{h,cpp}	SST file writer with block and Bloom filter serialization
src/paimon/common/sst/sst_file_reader.{h,cpp}	SST file reader with point and range query support using block iterators
src/paimon/common/sst/sst_file_io_test.cpp	Integration test for SST file write and read with Bloom filter validation
src/paimon/common/sst/block_writer.{h,cpp}	Block writer supporting aligned/unaligned key-value pair storage
src/paimon/common/sst/block_reader.{h,cpp}	Block reader with support for both aligned and unaligned block formats
src/paimon/common/sst/block_iterator.{h,cpp}	Iterator for sequential and binary search access to block entries
src/paimon/common/sst/block_handle.{h,cpp}	Metadata structure for block location and size with serialization
src/paimon/common/sst/block_entry.h	Simple key-value pair container for block entries
src/paimon/common/sst/block_trailer.{h,cpp}	Block metadata including CRC32 checksum and compression type
src/paimon/common/sst/block_cache.{h,cpp}	Block cache for reading SST blocks with position-based caching
src/paimon/common/sst/block_aligned_type.h	Enumeration for aligned vs unaligned block storage modes
src/paimon/common/sst/bloom_filter_handle.h	Metadata for Bloom filter location in SST files
src/paimon/common/memory/memory_slice.{h,cpp}	Slice abstraction over memory segments with comparison operators
src/paimon/common/memory/memory_slice_input.{h,cpp}	Input stream over memory slices with variable-length integer support
src/paimon/common/memory/memory_slice_output.{h,cpp}	Output stream over memory slices with auto-resize capability
src/paimon/common/io/cache/cache.{h,cpp}	Cache interface with NoCache placeholder implementation
src/paimon/common/io/cache/cache_key.{h,cpp}	Position-based cache key with hash support
src/paimon/common/io/cache/cache_manager.{h,cpp}	Cache manager coordinating data and index caches
src/paimon/CMakeLists.txt	Build configuration updates for new SST and memory components

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/paimon/common/utils/bit_set.cpp

src/paimon/common/io/cache/cache.h

src/paimon/common/sst/block_writer.h

src/paimon/common/sst/sst_file_writer.cpp

src/paimon/common/sst/block_cache.h

src/paimon/common/utils/bit_set.h

src/paimon/common/sst/block_cache.h

src/paimon/common/sst/block_handle.cpp

src/paimon/common/utils/bit_set.cpp

…paimon-cpp into sst-file-format

ChaomingZhangCN added 3 commits January 5, 2026 15:36

feat: introduce sst file format for btree global index

f01566f

feat: introduce sst file format for btree global index

3a8f42d

fix bloom filter

96ec401

ChaomingZhangCN changed the title ~~feat: introduce sst file format for btree global index~~ feat: sst file format for btree global index Jan 7, 2026

ChaomingZhangCN added 2 commits January 7, 2026 13:54

minor fix

69e1434

fix compile

95e2012

ChaomingZhangCN changed the title ~~feat: sst file format for btree global index~~ feat: Introduce sst file format for btree global index Jan 7, 2026

ChaomingZhangCN and others added 13 commits January 7, 2026 16:55

minor fix

e0c876e

fix compile

73bc0c2

Merge branch 'main' into sst-file-format

0fdc643

add virtual

3ffcf50

address

e08c162

Merge branch 'main' into sst-file-format

592df79

address

573f843

address

a5c8c53

fix tests

a56c96c

fix tests

757d212

fix tests

a0cb2a7

fix tests

bfa93f4

fix tests

dfe0527

lxy-9602 reviewed Jan 9, 2026

View reviewed changes

lucasfang requested a review from Copilot January 12, 2026 01:40

Merge branch 'main' into sst-file-format

a389fb0

Copilot started reviewing on behalf of lucasfang January 12, 2026 01:40 View session

Copilot AI reviewed Jan 12, 2026

View reviewed changes

ChaomingZhangCN added 3 commits January 12, 2026 19:28

address

4e43fe4

Merge branch 'sst-file-format' of https://github.com/ChaomingZhangCN/…

384e1da

…paimon-cpp into sst-file-format

clang lint

672d0ff

clang lint

779440c

ChaomingZhangCN requested a review from lxy-9602 January 12, 2026 12:47

Merge branch 'main' into sst-file-format

0916877

		std::function<std::shared_ptr<CacheValue>(const std::shared_ptr<CacheKey>&)> supplier) = 0;

		virtual void Put(const std::shared_ptr<CacheKey>& key, std::shared_ptr<CacheValue>& value) = 0;

		std::shared_ptr<Bytes> last_key_;

		const std::shared_ptr<MemoryPool>& pool_;

feat: Introduce sst file format for btree global index #49

Are you sure you want to change the base?

feat: Introduce sst file format for btree global index #49

Uh oh!

Conversation

ChaomingZhangCN commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

DONE:

TODO:

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChaomingZhangCN commented Jan 7, 2026 •

edited

Loading

lxy-9602 Jan 9, 2026 •

edited

Loading