Skip to content
BranchesTags

Repository files navigation

Nezha: A Key-Value Separated Distributed Store with Optimized Raft Integration

License Go 1.24+ RocksDB ICDE 2026 arXiv

High-Performance Distributed Key-Value Storage System with Key-Value Separation Optimized Raft Consensus Protocol

Nezha is an innovative distributed key-value storage system that deeply integrates key-value separation technology with the Raft consensus protocol, significantly reducing redundant persistence operations while providing scalable throughput and strong consistency guarantees. By redesigning the persistence strategy and introducing a tiered garbage collection mechanism, the system dramatically improves read and write performance while maintaining Raft's safety properties.

Quick Start

Docker (no installation required):

docker run -d --name nezha --network host \
  -v nezha-data:/app/data dyucong/nezha:latest \
  -address 127.0.0.1:3088 -internalAddress 127.0.0.1:30881 -peers 127.0.0.1:30881

From source (Ubuntu 20.04+):

bash scripts/setup-env.sh && source ~/.bashrc
./scripts/run-node.sh

Key Features

  • KVS-Raft Protocol: Innovative integration of key-value separation into the Raft consensus protocol
  • Optimized Persistence Strategy: Reduces value write operations from at least 3 times to just 1 time
  • Raft-Aware Garbage Collection: Adaptive GC framework that balances read and write performance
  • Three-Phase Request Processing: Ensures correct request handling during GC operations
  • Strong Consistency Guarantee: Maintains Raft's safety properties and linearizability
  • High Performance: Average throughput improvements of 460.2% (PUT), 12.5% (GET), 72.6% (SCAN) over standard Raft+RocksDB

System Architecture

Nezha adopts a three-layer architectural design with deep optimization of consensus and storage layers:

1. Application Layer

  • Provides standard key-value storage interfaces including Put, Get, Scan
  • Compatible with existing system APIs
  • Supports multiple access patterns

2. Consensus Layer (KVS-Raft)

  • Implements Raft protocol integrated with key-value separation
  • Values are stored directly in Raft logs with unified persistence
  • State machine stores only lightweight offsets for enhanced performance

3. Storage Layer

  • Three-module storage management: Active Storage, New Storage, Final Compacted Storage
  • Raft-aware garbage collection mechanism with dynamic storage space optimization
  • Hash index + sequential storage optimization for both point and range query performance

Deployment Methods

Nezha provides two deployment options:

Method 1: Source Code Compilation

Quick Setup (Recommended)

For Ubuntu 20.04+, use the one-command setup script:

bash scripts/setup-env.sh
source ~/.bashrc

This installs Go 1.24, RocksDB, and all dependencies automatically (~1 minute).

Then start the node:

./scripts/run-node.sh

Manual Setup

1. Go Environment (1.24+)

wget https://go.dev/dl/go1.24.0.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go1.24.0.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
go version

2. System Dependencies

sudo apt-get update
sudo apt-get install -y gcc g++ make git \
    librocksdb-dev \
    libsnappy-dev zlib1g-dev libbz2-dev \
    liblz4-dev libzstd-dev libgflags-dev

3. Configure CGO Environment Variables

echo 'export CGO_CFLAGS="-I/usr/include"' >> ~/.bashrc
echo 'export CGO_LDFLAGS="-L/usr/lib/x86_64-linux-gnu -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd"' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu' >> ~/.bashrc
source ~/.bashrc

4. Download Project Dependencies

cd Nezha
go mod download

Build

go build -o nezha ./cmd/nezha/

Run

Single Node

./nezha \
  -address 127.0.0.1:3088 \
  -internalAddress 127.0.0.1:30881 \
  -peers 127.0.0.1:30881 \
  -data ./data

Or using go run directly (no build step required):

go run ./cmd/nezha/ \
  -address 127.0.0.1:3088 \
  -internalAddress 127.0.0.1:30881 \
  -peers 127.0.0.1:30881 \
  -data ./data

Multi-Node Cluster (3 nodes on separate machines)

# Node 1 (run on machine with IP1)
./nezha -address IP1:3088 -internalAddress IP1:30881 \
        -peers IP1:30881,IP2:30881,IP3:30881 -data ./data

# Node 2 (run on machine with IP2)
./nezha -address IP2:3088 -internalAddress IP2:30881 \
        -peers IP1:30881,IP2:30881,IP3:30881 -data ./data

# Node 3 (run on machine with IP3)
./nezha -address IP3:3088 -internalAddress IP3:30881 \
        -peers IP1:30881,IP2:30881,IP3:30881 -data ./data

Parameter Reference

Parameter Description Required
-address Client-facing address and port Yes
-internalAddress Raft internal communication address and port Yes
-peers Comma-separated Raft addresses of all cluster nodes Yes
-data Data storage directory (default: .) No

Data Directory Structure

After startup, Nezha creates the following layout under the specified -data directory:

<data>/
└── data/
    ├── dbfile/
    │   └── keyIndex/       # LevelDB key-to-offset index
    └── valuelog/
        └── RaftState.log   # Unified Raft log + value store

Method 2: Docker Container

No compilation required. Pull the pre-built image from Docker Hub and run immediately.

Prerequisites

Run (Single Command)

docker run -d \
  --name nezha \
  --network host \
  -v nezha-data:/app/data \
  dyucong/nezha:latest \
  -address 127.0.0.1:3088 \
  -internalAddress 127.0.0.1:30881 \
  -peers 127.0.0.1:30881

Mac / Windows users: --network host is Linux-only. Use port mapping instead:

docker run -d \
  --name nezha \
  -p 3088:3088 -p 30881:30881 \
  -v nezha-data:/app/data \
  dyucong/nezha:latest \
  -address 0.0.0.0:3088 \
  -internalAddress 0.0.0.0:30881 \
  -peers 127.0.0.1:30881

Using Docker Compose

cd deploy/docker
docker-compose up -d

Management

docker logs nezha -f      # Follow logs
docker stop nezha         # Stop
docker start nezha        # Start again
docker rm -v nezha        # Remove container and data

Or use the management script:

cd deploy/docker
./manage.sh start    # Start node
./manage.sh stop     # Stop node
./manage.sh restart  # Restart node
./manage.sh logs     # Follow logs
./manage.sh status   # View status
./manage.sh clean    # Remove all containers, volumes, and image

Build from Source (Optional)

If you want to build the image yourself instead of using the pre-built one:

cd deploy/docker
./manage.sh build   # ~2 min (uses apt librocksdb, no source compilation)

Performance Testing

Experimental Environment

Benchmarks were conducted on a 3-node cluster, each node equipped with:

Component Specification
CPU Intel Xeon E5-2603 v3 (12 cores, 2.4 GHz)
Memory 64 GB DRAM
Storage 2 TB SSD
OS Ubuntu 20.04.4 LTS
Network 10 Gigabit Ethernet

Dataset: 100 GB, key size 10 B, value size 1 KB–256 KB, Zipf access distribution.

PUT (Random Write)

go run ./cmd/bench/randwrite_goroutine/ \
  -cnums 100 -dnums 39062 -vsize 256000 \
  -servers 127.0.0.1:3088

GET (Zipf Distribution Read)

go run ./cmd/bench/zipf_read/ \
  -cnums 100 -dnums 10000 \
  -servers 127.0.0.1:3088

SCAN (Range Scan)

go run ./cmd/bench/scan_pro/ \
  -cnums 1 -dnums 4 \
  -servers 127.0.0.1:3088

Benchmark Parameter Reference

Parameter Description Example
-cnums Number of concurrent clients 100
-dnums Number of operations 39062
-vsize Value size in bytes 256000
-servers Comma-separated server addresses 127.0.0.1:3088

Project Structure

Nezha/
├── go.mod / go.sum            # Go module
│
├── cmd/                       # Executables
│   ├── nezha/                 # The node binary: flags -> kvstore.Config, run until SIGTERM
│   ├── bench/                 # Benchmark tools, one main per directory (see bench/README.md)
│   └── ycsb/                  # YCSB workloads A, D, E, F and load scripts
│
├── internal/                  # Libraries private to this module
│   ├── kvstore/               # The node: Raft-replicated KV store with key-value separation
│   │   ├── server.go          # KVServer, Config wiring (New), lifecycle (Run)
│   │   ├── config.go          # Config and the -system presets
│   │   ├── service.go         # Client-facing gRPC service (Put/Get/Scan)
│   │   ├── apply.go           # Apply loop: committed entries -> store rows
│   │   ├── read.go            # Read paths over store, value log and sorted files
│   │   ├── gcloop.go          # GC trigger
│   │   ├── gc_first.go        # First GC round (value-log rewrite into a sorted file)
│   │   ├── gc_merge.go        # Later GC rounds (merge into the sorted file)
│   │   ├── recovery.go        # Crash recovery: state file, log replay, GC resume
│   │   ├── lsmraft.go         # LSM-Raft baseline (SSTable shipping to followers)
│   │   ├── inlinecache.go     # Inline small-value cache (AVP)
│   │   └── sparseindex.go     # Sparse index over the sorted value file
│   ├── raft/                  # Raft: election, replication, value log as Raft log,
│   │                          #   compaction, persistence, recovery, SSTable transport
│   ├── pool/                  # gRPC connection pool
│   └── util/                  # Logging and random key/value generators
│
├── api/                       # gRPC protocol definitions and generated code
│   ├── kvrpc/                 # Client-server RPC (Put/Get/Scan)
│   └── raftrpc/               # Raft inter-node RPC (RequestVote/AppendEntries)
│
├── scripts/                   # Operations and experiment scripts
│   ├── setup-env.sh           # One-command environment setup
│   ├── run-node.sh            # One-command node startup
│   ├── bench/                 # Performance comparison drivers
│   ├── test/                  # End-to-end and regression checks
│   ├── multinode/             # Multi-node correctness and failover drivers
│   └── lib/                   # Shared shell helpers
│
├── deploy/docker/             # Dockerfile, docker-compose.yml, manage.sh
├── .github/workflows/         # CI: build and push the Docker image
└── tla/                       # TLA+ specifications of the KVS-Raft protocol

Troubleshooting

CGO Linking Error

# Verify environment variables are set
echo $CGO_CFLAGS
echo $CGO_LDFLAGS
echo $LD_LIBRARY_PATH

# Verify RocksDB is installed
ls /usr/lib/x86_64-linux-gnu/librocksdb*

Port Already in Use

# Check what is using the ports
ss -tulpn | grep -E "3088|30881"

# Kill occupying process if needed
sudo kill $(lsof -t -i:3088)

Node Fails to Start

# Check firewall
sudo ufw status
sudo ufw allow 3088/tcp
sudo ufw allow 30881/tcp

Citation

This work was published at the 42nd IEEE International Conference on Data Engineering (ICDE 2026), Montreal, QC, Canada, May 4–8, 2026. If you use Nezha in your research, please cite:

Yangyang Wang, Yucong Dong, Ziqian Cheng, and Zichen Xu. "Nezha: A Key-Value Separated Distributed Store with Optimized Raft Integration." In Proceedings of the 42nd IEEE International Conference on Data Engineering (ICDE 2026), pp. 2503–2516. IEEE, 2026. https://doi.org/10.1109/ICDE65706.2026.00187

@inproceedings{wang2026nezha,
  author    = {Yangyang Wang and Yucong Dong and Ziqian Cheng and Zichen Xu},
  title     = {Nezha: A Key-Value Separated Distributed Store with Optimized Raft Integration},
  booktitle = {Proceedings of the 42nd {IEEE} International Conference on Data Engineering ({ICDE} 2026)},
  address   = {Montreal, QC, Canada},
  pages     = {2503--2516},
  publisher = {IEEE},
  year      = {2026},
  doi       = {10.1109/ICDE65706.2026.00187}
}

Contact

  • Issues: GitHub Issues
  • Email: Contact project maintainers through GitHub

About

Nezha, a prototype distributed storage system that innovatively integrates key-value separation with Raft to provide scalable throughput in a strong consistency guarantee.

Resources

Readme
Activity

Stars

8 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases

Packages

Contributors

Languages