Skip to content

Make fdbbackup query exit non-zero when it fails - #14125

Merged
saintstack merged 2 commits into
apple:mainfrom
saintstack:dev/stack/fdbbackup-query-exit-code
Oct 22, 2026
Merged

saintstack merged 2 commits into
apple:mainfrom
saintstack:dev/stack/fdbbackup-query-exit-code

Conversation

@saintstack

@saintstack saintstack commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

fdbbackup query reports failures as a JSON object on stdout and then returns normally. stopAfter() therefore yields a present Optional, and main() leaves the exit status at FDB_EXIT_SUCCESS.

Every failure path is affected, including "no restorable files set found for specified key ranges" and the catch-all for an arbitrary Error. Automation that checks the exit code — the normal thing for shell automation to trust — reads those failures as successes.

Fix

queryBackup now returns its own exit status through the fstatus future. That future was already declared and already inspected in main() for exactly this purpose, but nothing had ever assigned it.

The six failure paths co_return FDB_EXIT_ERROR; the success path returns FDB_EXIT_SUCCESS.

The fstatus check is also widened so an absent value maps to FDB_EXIT_ERROR. Absent means stopAfter() absorbed a fatal error rather than the command choosing a status — reachable here because the restore-timestamp resolution at the top of queryBackup sits outside the function's try.

Why not just throw

Throwing from the failure paths would produce the right exit code via the existing f check, but stopAfter() would also raise a SevError trace event and print Fatal Error: ... to stderr for an ordinary "no restorable files set" answer. That is the wrong severity for a routine negative result, and it breaks the one useful property this subcommand has: stdout is a single parseable JSON document and nothing else.

Note that submitBackup and submitDBBackup already throw actor_cancelled specifically to reach stopAfter's quiet branch, so the tension with stopAfter's severity is pre-existing.

Testing

  • ninja fdbbackup backup_tests clean.

  • BackupTests: 21/21, exit 0.

  • Failure path, against a container URL that does not resolve:

    $ fdbbackup query -d 'file:///tmp/nope' --query-restore-version 1; echo "exit=$?"
    {"key_ranges_filter":"","destination_container":"file:///tmp/nope","error":"Backup Container URL invalid"}
    exit=1
    

    Previously exit=0.

  • The success path returning 0 has not been exercised against a real backup container. It is co_return FDB_EXIT_SUCCESS with FDB_EXIT_SUCCESS == 0.

Caveat

stopAfter leaves its result at the initialized T() for please_reboot, please_reboot_delete, actor_cancelled and local_config_changed. For int that is 0, so a cancelled query still exits 0 and the new absent-value branch does not cover it. This matches the pre-existing behaviour through the f path and does not matter for a one-shot CLI, but it means this change should not be described as "any failure now exits non-zero".

Compatibility

fdbbackup query previously exited 0 on failure and now exits 1. Anything relying on the old behaviour — for example treating a non-zero exit as "the binary itself broke" rather than "the query failed" — will see a change. The JSON on stdout is unchanged in both shape and content.

No fdbbackup query invocation exists in fdb-kubernetes-tests.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@saintstack saintstack added the Backup_v3 Range Partitioned Backup label Sep 24, 2026
@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

Every failure path printed a JSON error object and returned normally, so
stopAfter() yielded a present Optional and main() kept FDB_EXIT_SUCCESS.
Automation checking the exit code read failures as successes.

queryBackup now reports its status through the fstatus future, which was
already declared and inspected for this but never assigned. An absent value
means stopAfter() absorbed a fatal error, so it maps to FDB_EXIT_ERROR.

Throwing instead would emit SevError and a stderr line for an ordinary "no
restorable files set" answer, breaking stdout as a single parseable document.
@saintstack
saintstack force-pushed the dev/stack/fdbbackup-query-exit-code branch from a6690f8 to 477ff5f Compare October 13, 2026 20:38
@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@saintstack saintstack closed this Oct 14, 2026
@saintstack saintstack reopened this Oct 14, 2026
@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@foundationdb-ci

This comment has been minimized.

@akshay-v-kaushik

akshay-v-kaushik commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

@saintstack LGTM for the most part. I had two thoughts:

  1. Should we pin the exit codes with two tests? fdbbackup/tests/ has no query coverage.
  2. timeKeeperVersionFromDatetime is the only await outside the try. Should we consider moving it inside would make that path consistent if you think it's worth it

…-resolution errors

dir_backup_tests now runs `fdbbackup query` against an existing backup
(exit 0) and a missing one (non-zero). The success check is skipped for
encrypted backups because query cannot be given an encryption key and
fails with "Restore error" on the manifest.

Move the restore-timestamp resolution inside queryBackup's try so a
failure there is reported as a JSON error with FDB_EXIT_ERROR like every
other failure, instead of escaping to stopAfter().
@saintstack

Copy link
Copy Markdown
Contributor Author

Should we pin the exit codes with two tests? fdbbackup/tests/ has no query coverage.
timeKeeperVersionFromDatetime is the only await outside the try. Should we consider moving it inside would make that path consistent if you think it's worth it

Good review. Done. Please take another look @akshay-v-kaushik

@akshay-v-kaushik akshay-v-kaushik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 3f8db96
  • Duration 0:35:29
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: 3f8db96
  • Duration 0:38:47
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 3f8db96
  • Duration 0:57:51
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 3f8db96
  • Duration 1:01:32
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: 3f8db96
  • Duration 1:05:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 3f8db96
  • Duration 1:47:49
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: 3f8db96
  • Duration 2:27:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack
saintstack merged commit 355077b into apple:main Oct 22, 2026
10 checks passed
@saintstack

Copy link
Copy Markdown
Contributor Author

Thanks for the review @akshay-v-kaushik

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backup_v3 Range Partitioned Backup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants