Skip to content

[#13459] improvement(ci): Split build and Backend IT into sharded sub-jobs - #13495

Draft
yuqi1129 wants to merge 4 commits into
apache:mainfrom
yuqi1129:improve/13459-ci-sharding
Draft

yuqi1129 wants to merge 4 commits into
apache:mainfrom
yuqi1129:improve/13459-ci-sharding

Conversation

@yuqi1129

@yuqi1129 yuqi1129 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Define build and Backend IT shards in dev/ci/test-shards.sh, with a catch-all others shard for new modules.
  • Run build (docker, others) and Backend IT (hive, lakehouse, others) shards in parallel matrices; retain main's separate core unit/H2/MySQL/PostgreSQL lanes.
  • Merge JaCoCo reports from build shards together with the existing core-lane coverage, and give Backend IT report artifacts unique names.
  • Assign Iceberg experimental tests to the lakehouse shard; preserve the complete cron/manual Backend suite with its original 120-minute timeout.

Why are the changes needed?

The existing build and Backend IT jobs approach or exceed their 120-minute timeout because Docker-tagged test tasks serialize behind the shared environment lock. Sharding keeps the existing test coverage while shortening the critical path. The measurements and shard plan are in #13459.

Fix: #13459

Does this PR introduce any user-facing change?

No API or configuration changes. CI job names, scheduling, and report artifacts change.

How was this patch tested?

  • Five regression tests via python3 dev/ci/test_test_shards.py, wired into the build workflow: shard ownership, catch-all exclusions, dedicated core lanes, Iceberg experimental tests, complete cron suite, and invalid selectors.
  • Mutation check: removing the lakehouse experimental task makes the regression test fail.
  • bash -n, actionlint, YAML parsing, all 29 shell blocks in the affected workflows, and git diff --check.
  • Verified that main's core-test and coverage jobs are preserved verbatim after resolving the merge.
  • Full Gradle/Docker suites were not run locally; GitHub CI will validate execution of the matrices.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Code Coverage Report

Overall Project 71.7% 🟢
Files changed No Java source files changed -

Module Coverage
aliyun 19.74% 🔴
api 53.51% 🟢
authorization-common 85.96% 🟢
authorization-ranger 14.44% 🔴
aws 53.54% 🟢
aws-bundle 0.0% 🔴
azure 32.1% 🔴
azure-bundle 0.0% 🔴
catalog-common 28.55% 🔴
catalog-fileset 82.06% 🟢
catalog-glue 71.83% 🟢
catalog-hive 83.63% 🟢
catalog-jdbc-common 45.99% 🟢
catalog-jdbc-doris 86.88% 🟢
catalog-jdbc-mysql 81.8% 🟢
catalog-jdbc-postgresql 83.89% 🟢
catalog-jdbc-starrocks 79.16% 🟢
catalog-kafka 75.21% 🟢
catalog-lakehouse-generic 78.48% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 85.93% 🟢
catalog-lakehouse-paimon 84.23% 🟢
catalog-model 77.99% 🟢
cli 44.62% 🟢
client-java 78.32% 🟢
client-java-runtime 0.0% 🔴
common 58.59% 🟢
core 86.29% 🟢
filesystem-hadoop3 76.48% 🟢
flink 18.07% 🔴
flink-common 53.95% 🟢
flink-runtime 0.0% 🔴
gcp 32.2% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 17.84% 🔴
hive-metastore-common 62.3% 🟢
hive-metastore3-libs 58.8% 🟢
iceberg-aliyun-bundle 0.0% 🔴
iceberg-common 66.16% 🟢
iceberg-rest-server 77.0% 🟢
idp-basic 86.77% 🟢
integration-test-common 0.0% 🔴
jobs 66.9% 🟢
lance-common 36.14% 🔴
lance-rest-server 69.51% 🟢
lineage 59.39% 🟢
optimizer 83.86% 🟢
optimizer-api 42.56% 🟢
server 90.89% 🟢
server-common 82.42% 🟢
spark 60.86% 🟢
tencent 81.78% 🟢
trino-connector 62.11% 🟢

@yuqi1129 yuqi1129 self-assigned this Sep 24, 2026
…the aggregate result jobs

- Fold the `client` Backend IT shard into `others`. Both are short, and the
  merged shard still finishes before the slowest build shard, so PR feedback
  time is unchanged while each backend runs one job less.
- Remove `build-result` and `BackendIT-result`. No branch protection rule
  requires them, so they only added checks.
@yuqi1129
yuqi1129 marked this pull request as ready for review September 24, 2026 12:41
Copilot AI lite review requested due to automatic review settings September 24, 2026 12:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@yuqi1129

Copy link
Copy Markdown
Contributor Author

The whole CI can be finished within an hour, but it will add an extra 9 CI pipelines. I'm not sure whether it's worth it.cc @jerryshao

@nevzheng

Copy link
Copy Markdown
Collaborator

@roryqi @jerryshao PTAL. Efficient CI / Merge Times accelerate OSS development!

@yuqi1129

Copy link
Copy Markdown
Contributor Author

follow-up: #13471

nevzheng pushed a commit to nevzheng/gravitino that referenced this pull request Oct 12, 2026
Wire coreUnitTest/coreH2Test/coreMySQLTest/corePostgreSQLTest into a
core-test matrix job in build.yml directly, so the PR no longer depends
on the Backend IT sharding in apache#13495 (dev/ci/test-shards.sh). The build
job skips :core:test and uploads its JaCoCo reports; a coverage job
merges them with the four core lane .exec files after
core-test-contract reconciles the lanes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019TX7XCnuCMGSJ7xX2iubTH
nevzheng pushed a commit to nevzheng/gravitino that referenced this pull request Oct 13, 2026
Wire coreUnitTest/coreH2Test/coreMySQLTest/corePostgreSQLTest into a
core-test matrix job in build.yml directly, so the PR no longer depends
on the Backend IT sharding in apache#13495 (dev/ci/test-shards.sh). The build
job skips :core:test and uploads its JaCoCo reports; a coverage job
merges them with the four core lane .exec files after
core-test-contract reconciles the lanes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019TX7XCnuCMGSJ7xX2iubTH
@nevzheng
nevzheng self-requested a review October 21, 2026 16:13
@yuqi1129
yuqi1129 marked this pull request as draft October 22, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Restructure build and Backend IT CI into sharded sub-jobs

3 participants