fix: report blobstore ENOSPC by c3y1huang · Pull Request #76 · longhorn/spdk

c3y1huang · 2026-07-21T08:44:23Z

Which issue(s) this PR fixes:

Issue longhorn/longhorn#1061

What this PR does / why we need it:

See RCA.

Special notes for your reviewer:

None

Additional documentation or context

None

Copilot

Pull request overview

This PR improves how out-of-space conditions in the SPDK blobstore are surfaced and diagnosed, ensuring thin-provisioned write failures are reported as NVMe CAPACITY EXCEEDED (instead of a generic device error) and reducing log spam when the lvstore runs out of clusters. It also adds EC (erasure-coded bdev) diagnostics improvements to better attribute and throttle child I/O failure logging, aligned with the referenced RCA.

Changes:

Blobstore: add a one-shot “out of clusters” WARNLOG gate per full→free→full cycle.
Lvol: translate blobstore -ENOSPC into NVMe status GENERIC / CAPACITY_EXCEEDED.
EC bdev: add safe slot attribution via a published base-bdev pointer cache plus per-slot failure counters to throttle repeated failure logs; add unit test coverage for the new gate logic.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/unit/lib/bdev/vbdev_lvol.c/vbdev_lvol_ut.c	Adds a unit test asserting ENOSPC maps to NVMe CAPACITY EXCEEDED and updates blob write stubs to inject errors.
test/unit/lib/bdev/ec/bdev_ec.c/bdev_ec_ut.c	Adds a unit test for the per-slot failure log throttling gate behavior.
module/bdev/lvol/vbdev_lvol.c	Maps blobstore `-ENOSPC` completion to `spdk_bdev_io_complete_nvme_status(... CAPACITY_EXCEEDED)`.
module/bdev/ec/bdev_ec.c	Publishes/clears cached base-bdev pointers on open/close paths; resets per-slot failure counter on successful hot-swap; adds more detailed submit-failure logs and exports failure counters in JSON.
module/bdev/ec/bdev_ec_io.c	Adds slot mapping via cached base-bdev pointers and throttled logging on failed child I/O completion and submit failures.
module/bdev/ec/bdev_ec_internal.h	Introduces `base_bdevs[]`, `child_io_failures[]`, and helper inlines for pointer publishing and log gating.
lib/blob/blobstore.h	Adds `out_of_clusters_warned` to the blobstore state (protected by `used_lock`).
lib/blob/blobstore.c	Implements the warn-once gate for ENOSPC cluster allocation and clears it when clusters are released.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

c3y1huang · 2026-07-23T12:52:39Z

While testing for benchmarking, found a bug: the module's optimal_io_boundary causes the bdev layer to split every write to one stripe before it arrives; this means the full-stripe condition will not be met. I will include the fix in this PR.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

derekbit

In general, LGTM

c3y1huang self-assigned this Jul 21, 2026

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch 2 times, most recently from 21509fa to d8356de Compare July 21, 2026 10:14

c3y1huang requested a review from Copilot July 21, 2026 10:14

Copilot started reviewing on behalf of c3y1huang July 21, 2026 10:15 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread lib/blob/blobstore.c

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch from d8356de to 1337c63 Compare July 21, 2026 10:30

c3y1huang requested a review from Copilot July 21, 2026 10:31

Copilot started reviewing on behalf of c3y1huang July 21, 2026 10:32 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread lib/blob/blobstore.c Outdated

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch from 1337c63 to 4238a68 Compare July 21, 2026 11:10

c3y1huang requested a review from Copilot July 21, 2026 11:11

Copilot started reviewing on behalf of c3y1huang July 21, 2026 11:11 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread module/bdev/ec/bdev_ec.c

Comment thread lib/blob/blobstore.c

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch from 4238a68 to 08c9c6c Compare July 21, 2026 11:42

c3y1huang requested a review from Copilot July 21, 2026 11:44

Copilot started reviewing on behalf of c3y1huang July 21, 2026 11:44 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

c3y1huang mentioned this pull request Jul 21, 2026

[FEATURE] V2 Data Engine Sharding - Experimental longhorn/longhorn#1061

Closed

c3y1huang requested a review from Copilot July 22, 2026 08:06

Copilot started reviewing on behalf of c3y1huang July 22, 2026 08:07 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread module/bdev/lvol/vbdev_lvol.c Outdated

Comment thread lib/lvol/lvol.c

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch from d935ccb to 8eaabbc Compare July 23, 2026 01:31

c3y1huang requested a review from Copilot July 23, 2026 01:35

Copilot started reviewing on behalf of c3y1huang July 23, 2026 01:35 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread lib/lvol/lvol.c Outdated

c3y1huang requested a review from Copilot July 23, 2026 01:47

Copilot started reviewing on behalf of c3y1huang July 23, 2026 01:47 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread test/unit/lib/bdev/ec/bdev_ec.c/bdev_ec_ut.c

c3y1huang requested a review from Copilot July 23, 2026 01:59

c3y1huang requested a review from Copilot July 24, 2026 03:59

Copilot started reviewing on behalf of c3y1huang July 24, 2026 04:00 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

c3y1huang marked this pull request as ready for review July 24, 2026 04:05

c3y1huang added 21 commits July 27, 2026 08:43

bdev/ec: log hard submit failures on the write path

666b33b

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: cache base bdev pointers for safe slot lookup

4c49f39

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: rate-limit per-slot child I/O failure logs

60937c4

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

blob: warn once when the blobstore runs out of clusters

af1af20

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/lvol: complete thin-provision ENOSPC as NVMe CAPACITY EXCEEDED

8e1cc7d

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

fix(lvol): compute num_md_pages in 64-bit and reject overflow

5417e9e

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(lvol): boundary unit test for md-pages overflow

e91dbb3

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: report resize no-op as idempotent success

44b5771

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(bdev/ec): cover resize precondition errors

fa33bb4

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

fix(lvol): pass validated opts copy to setup_lvs_opts

a2a8708

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: add write-path defer and NOMEM observability counters

77c7f35

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: park stripe-conflicted writes instead of NOMEM bounce

85d09ac

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(bdev/ec): cover stripe-conflict wait queue

985d310

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: add iov slice helper for multi-strip fan-out

172e2ef

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(bdev/ec): cover iov slice helper

ee8567d

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: fan out healthy reads wider than one strip

a9d0bf7

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(bdev/ec): cover healthy read fan-out

87270bc

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: handle degraded reads spanning multiple chunks

12768fd

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: accept multi-chunk RMW writes

48eaefd

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

bdev/ec: split I/O at stripe boundaries to enable full-stripe writes

354a071

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

test(bdev/ec): cover stripe-boundary split and degraded read routing

df37696

longhorn/longhorn-1061 Signed-off-by: Chin-Ya Huang <chin-ya.huang@suse.com>

c3y1huang force-pushed the 1061-v2-sharding-kbench-failure branch from 5ab116d to df37696 Compare July 27, 2026 00:44

derekbit approved these changes Jul 27, 2026

View reviewed changes

derekbit merged commit 324634b into longhorn:longhorn-v25.09 Jul 27, 2026
2 checks passed

c3y1huang deleted the 1061-v2-sharding-kbench-failure branch July 27, 2026 03:54

Uh oh!

Conversation

c3y1huang commented Jul 21, 2026

Which issue(s) this PR fixes:

What this PR does / why we need it:

Special notes for your reviewer:

Additional documentation or context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

c3y1huang commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

derekbit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

c3y1huang commented Jul 23, 2026 •

edited

Loading