From 4753b13c78a8f0131fecba550af49b8a07bdc08b Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 16:58:54 +0800 Subject: [PATCH 01/18] Add breeze verify to show CI-required local verification for changed files The self-review checklist in AGENTS.md (step 7) asks contributors to find the tests selective-checks would run and run them locally. The only tool for that is `breeze ci selective-check`, which looks at a single commit and prints GitHub Actions outputs, so a contributor still has to read the workflow files to turn those outputs into commands. This adds a developer-facing, dry-run command that feeds the working-tree diff to the same SelectiveChecks logic and prints the commands for the default CI matrix cell, with the rest explicitly left to CI. --- dev/breeze/doc/03_developer_tasks.rst | 40 ++++ dev/breeze/doc/images/output_verify.svg | 116 +++++++++++ dev/breeze/doc/images/output_verify.txt | 1 + dev/breeze/src/airflow_breeze/breeze.py | 1 + .../commands/developer_commands_config.py | 10 + .../commands/verify_commands.py | 103 ++++++++++ .../airflow_breeze/utils/verification_plan.py | 146 +++++++++++++ dev/breeze/tests/test_verification_plan.py | 192 ++++++++++++++++++ dev/breeze/tests/test_verify_commands.py | 84 ++++++++ 9 files changed, 693 insertions(+) create mode 100644 dev/breeze/doc/images/output_verify.svg create mode 100644 dev/breeze/doc/images/output_verify.txt create mode 100644 dev/breeze/src/airflow_breeze/commands/verify_commands.py create mode 100644 dev/breeze/src/airflow_breeze/utils/verification_plan.py create mode 100644 dev/breeze/tests/test_verification_plan.py create mode 100644 dev/breeze/tests/test_verify_commands.py diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst index 56b8bbbb8ba76..a7b43e1a8d1d4 100644 --- a/dev/breeze/doc/03_developer_tasks.rst +++ b/dev/breeze/doc/03_developer_tasks.rst @@ -316,6 +316,46 @@ If the provider name is ``apache-airflow-providers-cncf-kubernetes``, it will be Note: For building docs for apache-airflow-providers index, use ``apache-airflow-providers`` as the short hand operator. +Finding out what CI will run for your change +-------------------------------------------- + +``breeze verify`` reads the files changed against the target branch (the merge-base of +``--base-ref`` and ``HEAD``, plus untracked files), runs the same selective-checks logic CI uses +and prints the commands you would have to run locally to cover the default CI matrix cell +(default Python, sqlite backend). It is a dry-run: nothing is executed. The only non-zero exit is a +usage error such as a base ref git cannot resolve. + +.. code-block:: bash + + breeze verify + breeze verify --json + breeze verify --base-ref upstream/main + +Each row says what kind of check it is, whether it runs on the host or needs Docker and the +CI image (``breeze``), and the exact command. Use ``--json`` for machine-readable output with +the same fields. + +When a change touches CI tooling or dependency files, selective checks make CI run the full suite +and the table grows accordingly. ``breeze verify`` says so under the table. Run what covers your change +locally and leave the rest to CI. + +Local verification is never the full CI matrix. The table covers one matrix cell: the default +Python version on sqlite. Other Python versions, Postgres and MySQL, lowest-dependency runs, +Kubernetes, Helm, e2e suites, the provider compatibility matrix and ARM runners only run in CI. +Three more caveats. Selective checks compare ``pyproject.toml`` contents between ``HEAD`` and ``HEAD^`` +only, so dependency changes that are uncommitted or in earlier commits of your branch are not detected +as such (test selection is unaffected, only the dependency-bump checks are). Untracked files count for +test selection but ``prek`` only sees tracked files, so ``git add`` new files before running the prek +row. Packaging steps CI runs around some tests (building and twine-checking the Task SDK and +airflow-ctl wheels, regenerating the Python API client from its own repository) are not listed. + +These are all available flags of ``verify`` command: + +.. image:: ./images/output_verify.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_verify.svg + :width: 100% + :alt: Breeze verify + Running static checks --------------------- diff --git a/dev/breeze/doc/images/output_verify.svg b/dev/breeze/doc/images/output_verify.svg new file mode 100644 index 0000000000000..2a89a4b6733fc --- /dev/null +++ b/dev/breeze/doc/images/output_verify.svg @@ -0,0 +1,116 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Command: verify + + + + + + + + + + +Usage:breeze verify[OPTIONS] + +List the local verification CI would require for the current changes. Nothing is executed. + +╭─ Verify flags ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--base-refGit ref the change will be merged into. Changed files are computed from its merge-base with HEAD.       │ +│[default: main](TEXT)│ +│--json    Print machine-readable JSON instead of a table.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--verbose-vPrint verbose information about performed steps.│ +│--help   -hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ + + + + diff --git a/dev/breeze/doc/images/output_verify.txt b/dev/breeze/doc/images/output_verify.txt new file mode 100644 index 0000000000000..96e6cfe0ba225 --- /dev/null +++ b/dev/breeze/doc/images/output_verify.txt @@ -0,0 +1 @@ +742710f4e88c9adb0a1a83385e15dc4f diff --git a/dev/breeze/src/airflow_breeze/breeze.py b/dev/breeze/src/airflow_breeze/breeze.py index be8b8bae1ae7d..34b4a8d5d6509 100755 --- a/dev/breeze/src/airflow_breeze/breeze.py +++ b/dev/breeze/src/airflow_breeze/breeze.py @@ -46,6 +46,7 @@ from airflow_breeze.commands.setup_commands import setup_group # noqa: E402 from airflow_breeze.commands.testing_commands import testing_group # noqa: E402 from airflow_breeze.commands.ui_commands import ui_group # noqa: E402 +from airflow_breeze.commands import verify_commands # noqa: E402, F401 main.add_command(testing_group) main.add_command(kubernetes_group) diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py index 0c38fb7efa92b..1206a3c493730 100644 --- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py @@ -28,6 +28,7 @@ "cleanup", "generate-migration-file", "doctor", + "verify", ], } DEVELOPER_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = { @@ -400,6 +401,15 @@ ], }, ], + "breeze verify": [ + { + "name": "Verify flags", + "options": [ + "--base-ref", + "--json", + ], + }, + ], "breeze doctor": [ { "name": "Auto-healing of breeze", diff --git a/dev/breeze/src/airflow_breeze/commands/verify_commands.py b/dev/breeze/src/airflow_breeze/commands/verify_commands.py new file mode 100644 index 0000000000000..5af1d69769c25 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/commands/verify_commands.py @@ -0,0 +1,103 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import contextlib +import io +import json +import sys + +import click +from rich.markup import escape +from rich.table import Table + +from airflow_breeze.branch_defaults import AIRFLOW_BRANCH, DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH +from airflow_breeze.commands.common_options import option_verbose +from airflow_breeze.commands.main_command import main +from airflow_breeze.global_constants import GithubEvents +from airflow_breeze.utils.console import get_console +from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH +from airflow_breeze.utils.run_utils import run_command +from airflow_breeze.utils.shared_options import get_verbose +from airflow_breeze.utils.verification_plan import build_local_verification_plan + + +def get_changed_files_against(base_ref: str) -> tuple[str, ...]: + def _git(*args: str) -> list[str]: + result = run_command( + ["git", *args], capture_output=True, text=True, check=False, cwd=AIRFLOW_ROOT_PATH + ) + if result.returncode != 0: + raise click.ClickException( + f"git {args[0]} failed for base ref {base_ref!r}: {result.stderr.strip()}" + ) + return result.stdout.splitlines() + + merge_base = _git("merge-base", base_ref, "HEAD")[0] + changed = _git("diff", "--name-only", merge_base) + _git("ls-files", "--others", "--exclude-standard") + return tuple(sorted(set(changed))) + + +@main.command( + name="verify", + help="List the local verification CI would require for the current changes. Nothing is executed.", +) +@click.option( + "--base-ref", + default=AIRFLOW_BRANCH, + show_default=True, + help="Git ref the change will be merged into. Changed files are computed from its merge-base with HEAD.", +) +@click.option("--json", "as_json", is_flag=True, help="Print machine-readable JSON instead of a table.") +@option_verbose +def verify(base_ref: str, as_json: bool): + from airflow_breeze.utils.selective_checks import SelectiveChecks + + # SelectiveChecks narrates its decisions on stdout; keep stdout for the result only. + with contextlib.redirect_stdout(sys.stderr if get_verbose() else io.StringIO()): + changed_files = get_changed_files_against(base_ref) + sc = SelectiveChecks( + files=changed_files, + commit_ref="HEAD", + default_branch=AIRFLOW_BRANCH, + default_constraints_branch=DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH, + github_event=GithubEvents.PULL_REQUEST, + ) + result = build_local_verification_plan(sc, changed_files, base_ref) + if as_json: + print(json.dumps(result, indent=2)) + return + console = get_console() + console.print( + f"[info]{len(changed_files)} changed file(s) against {base_ref}, " + f"CI default Python {result['default_python_version']}[/]\n" + ) + table = Table(title="Run locally to match CI", show_lines=False) + for column in ("kind", "runs_in", "required", "command"): + table.add_column(column) + for item in result["items"]: + table.add_row(item["kind"], item["runs_in"], str(item["required"]).lower(), escape(item["command"])) + console.print(table) + if result["full_tests_needed"]: + console.print( + "\n[warning]CI runs the full suite for this change because it touches CI tooling or dependency " + "files. Run what covers your change locally and leave the rest to CI.[/]" + ) + console.print( + f"\n[warning]Covers Python {result['default_python_version']} on sqlite only; other Python versions, " + "Postgres, MySQL, the provider compatibility matrix and CI-only jobs run in CI.[/]" + ) diff --git a/dev/breeze/src/airflow_breeze/utils/verification_plan.py b/dev/breeze/src/airflow_breeze/utils/verification_plan.py new file mode 100644 index 0000000000000..ccc8f58ed9058 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/utils/verification_plan.py @@ -0,0 +1,146 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import json +import shlex +from dataclasses import asdict, dataclass +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from airflow_breeze.utils.selective_checks import SelectiveChecks + + +@dataclass(frozen=True) +class VerificationItem: + kind: str + command: str + runs_in: str + required: bool = True + + +# SelectiveChecks flag -> (kind, command, runs_in) of the CI job(s) that flag gates. +# Mirrors .github/workflows/ci-amd.yml and basic-tests.yml; "breeze" means the command needs +# Docker and the CI image, "host" means only host tooling. +FLAG_COMMANDS: dict[str, tuple[tuple[str, str, str], ...]] = { + "run_mypy_providers": (("mypy", "prek run --stage manual mypy-providers --all-files", "breeze"),), + "has_migrations": (("schema", "prek run --stage manual migration-round-trip --all-files", "breeze"),), + "run_task_sdk_tests": (("unit", "breeze testing task-sdk-tests", "breeze"),), + "run_airflow_ctl_tests": (("unit", "breeze testing airflow-ctl-tests", "breeze"),), + "run_scripts_tests": (("unit", "uv run --project scripts pytest scripts/tests/", "host"),), + "run_ui_tests": ( + ("ui", "cd airflow-core/src/airflow/ui && pnpm install --frozen-lockfile && pnpm test", "host"), + ( + "ui", + "cd airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui && pnpm install --frozen-lockfile && pnpm test", + "host", + ), + ), + "run_api_codegen": (("schema", "breeze testing python-api-client-tests", "breeze"),), + "run_go_sdk_tests": (("unit", "cd go-sdk && go test ./...", "host"),), + "run_java_sdk_tests": ( + ("unit", "cd java-sdk && ./gradlew test", "host"), + ("docs", "breeze build-docs --sdk-docs-only --sdk=java", "breeze"), + ), + "run_ts_sdk_docs": (("docs", "breeze build-docs --sdk-docs-only --sdk=typescript", "breeze"),), + "run_breeze_integration_tests": ( + ("unit", "cd dev/breeze && uv run --locked pytest -m integration_tests", "host"), + ), +} + +# Gated CI jobs that need a cluster, a prod image or an e2e stack; classified only, never printed. +NOT_RUNNABLE_LOCALLY: frozenset[str] = frozenset( + { + "run_kubernetes_tests", + "run_helm_tests", + "run_kustomize_overlays_tests", + "run_ui_e2e_tests", + "run_task_sdk_integration_tests", + "run_airflow_ctl_integration_tests", + "run_system_tests", + "run_remote_logging_s3_e2e_tests", + "run_remote_logging_elasticsearch_e2e_tests", + "run_remote_logging_opensearch_e2e_tests", + "run_event_driven_e2e_tests", + "run_java_sdk_e2e_tests", + "run_go_sdk_e2e_tests", + "run_openlineage_e2e_tests", + "run_openlineage_e2e_compat_tests", + "run_ts_sdk_e2e_tests", + } +) + + +def build_prek_item(sc: SelectiveChecks, base_ref: str) -> VerificationItem: + if sc.basic_checks_only: + command = f"SKIP_BREEZE_PREK_HOOKS=true SKIP={sc.skip_prek_hooks} prek run --from-ref {shlex.quote(base_ref)} --to-ref HEAD" + return VerificationItem("prek", command, "host") + return VerificationItem("prek", f"SKIP={sc.skip_prek_hooks} prek run --all-files", "breeze") + + +def build_unit_test_items(group: str, test_types_json: str | None) -> list[VerificationItem]: + test_types = " ".join(chunk["test_types"] for chunk in json.loads(test_types_json or "null") or []) + if not test_types: + return [] + prefix = f"breeze testing {group}-tests" + return [ + VerificationItem( + "unit", + f'{prefix} --run-in-parallel --run-db-tests-only --parallel-test-types "{test_types}"', + "breeze", + ), + VerificationItem( + "unit", + f'{prefix} --use-xdist --skip-db-tests --no-db-cleanup --backend none --parallel-test-types "{test_types}"', + "breeze", + ), + ] + + +def build_local_verification_plan( + sc: SelectiveChecks, changed_files: tuple[str, ...], base_ref: str +) -> dict[str, Any]: + items = [build_prek_item(sc, base_ref)] + if sc.run_unit_tests: + items += build_unit_test_items("core", sc.core_test_types_list_as_strings_in_json) + if not sc.skip_providers_tests: + items += build_unit_test_items("providers", sc.providers_test_types_list_as_strings_in_json) + for flag, commands in FLAG_COMMANDS.items(): + if getattr(sc, flag): + items += [VerificationItem(*command) for command in commands] + if sc.docs_build: + items.append( + VerificationItem("docs", f"breeze build-docs {sc.docs_list_as_string}".rstrip(), "breeze") + ) + items.append(VerificationItem("unit", "cd dev/breeze && uv run --locked pytest", "host")) + if dists := " ".join(sorted(json.loads(sc.shared_distributions_as_json))): + items.append( + VerificationItem( + "unit", + f"for d in {dists}; do (cd shared/$d && uv run --group dev pytest) || exit 1; done", + "host", + ) + ) + # CI runs this with `|| true`, so it never fails a PR. + items.append(VerificationItem("ui", "breeze ui check-translation-completeness", "host", required=False)) + return { + "base_ref": base_ref, + "default_python_version": sc.default_python_version, + "full_tests_needed": sc.full_tests_needed, + "changed_files": list(changed_files), + "items": [asdict(item) for item in items], + } diff --git a/dev/breeze/tests/test_verification_plan.py b/dev/breeze/tests/test_verification_plan.py new file mode 100644 index 0000000000000..018d2d6da26dd --- /dev/null +++ b/dev/breeze/tests/test_verification_plan.py @@ -0,0 +1,192 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import inspect +import json +import shlex +from functools import cached_property +from unittest.mock import Mock + +import pytest + +from airflow_breeze.global_constants import DEFAULT_PYTHON_MAJOR_MINOR_VERSION, GithubEvents +from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH +from airflow_breeze.utils.selective_checks import SelectiveChecks +from airflow_breeze.utils.verification_plan import ( + FLAG_COMMANDS, + NOT_RUNNABLE_LOCALLY, + build_local_verification_plan, + build_unit_test_items, +) + +NEUTRAL_COMMIT = "938f0c1f3cc4cbe867123ee8aa9f290f9f18100a" +# Exported by SelectiveChecks but no workflow job is gated on them (build-info pass-through only). +NOT_A_CI_JOB = { + "run_amazon_tests", + "run_api_tests", + "run_ol_tests", + "run_python_scans", + "run_javascript_scans", +} + + +def _selective_checks(files: tuple[str, ...], default_branch: str = "main") -> SelectiveChecks: + return SelectiveChecks( + files=files, + commit_ref=NEUTRAL_COMMIT, + github_event=GithubEvents.PULL_REQUEST, + pr_labels=tuple(), + default_branch=default_branch, + ) + + +def _mock_selective_checks(**flags: object) -> Mock: + sc = Mock(spec=SelectiveChecks) + for flag in (*FLAG_COMMANDS, "run_unit_tests", "docs_build", "basic_checks_only"): + setattr(sc, flag, False) + sc.skip_providers_tests = True + sc.skip_prek_hooks = "identity" + sc.shared_distributions_as_json = "[]" + sc.default_python_version = DEFAULT_PYTHON_MAJOR_MINOR_VERSION + sc.full_tests_needed = False + for flag, value in flags.items(): + setattr(sc, flag, value) + return sc + + +@pytest.mark.parametrize("flag", sorted(FLAG_COMMANDS)) +def test_each_flag_maps_to_its_commands(flag: str): + result = build_local_verification_plan(_mock_selective_checks(**{flag: True}), (), "main") + commands = [item["command"] for item in result["items"]] + assert commands == [ + "SKIP=identity prek run --all-files", + *(command for _, command, _ in FLAG_COMMANDS[flag]), + "cd dev/breeze && uv run --locked pytest", + "breeze ui check-translation-completeness", + ] + assert [item["required"] for item in result["items"]][-1] is False + assert all(item["required"] for item in result["items"][:-1]) + + +@pytest.mark.parametrize( + ("basic_checks_only", "expected"), + [ + (False, ("prek", "SKIP=identity prek run --all-files", "breeze")), + ( + True, + ( + "prek", + "SKIP_BREEZE_PREK_HOOKS=true SKIP=identity prek run --from-ref main --to-ref HEAD", + "host", + ), + ), + ], +) +def test_prek_command_follows_basic_checks_only(basic_checks_only: bool, expected: tuple[str, str, str]): + result = build_local_verification_plan( + _mock_selective_checks(basic_checks_only=basic_checks_only), (), "main" + ) + prek = result["items"][0] + assert (prek["kind"], prek["command"], prek["runs_in"]) == expected + + +def test_docs_only_change_mirrors_the_ci_cell(): + files = ("airflow-core/docs/index.rst",) + sc = _selective_checks(files) + result = build_local_verification_plan(sc, files, "main") + assert [item["command"] for item in result["items"]] == [ + f"SKIP={sc.skip_prek_hooks} prek run --all-files", + "breeze build-docs apache-airflow", + "cd dev/breeze && uv run --locked pytest", + "for d in " + + " ".join(sorted(json.loads(sc.shared_distributions_as_json))) + + "; do (cd shared/$d && uv run --group dev pytest) || exit 1; done", + "breeze ui check-translation-completeness", + ] + + +def test_core_change_splits_db_and_non_db_cells(): + files = ("airflow-core/src/airflow/models/dag.py",) + sc = _selective_checks(files) + core_types = " ".join(c["test_types"] for c in json.loads(sc.core_test_types_list_as_strings_in_json)) + result = build_local_verification_plan(sc, files, "main") + unit = [ + item["command"] for item in result["items"] if item["command"].startswith("breeze testing core-tests") + ] + assert unit == [ + f'breeze testing core-tests --run-in-parallel --run-db-tests-only --parallel-test-types "{core_types}"', + "breeze testing core-tests --use-xdist --skip-db-tests --no-db-cleanup --backend none " + f'--parallel-test-types "{core_types}"', + ] + + +def test_every_selective_checks_run_flag_is_classified(): + run_flags = { + name + for name, value in inspect.getmembers(SelectiveChecks) + if isinstance(value, cached_property) + and (name.startswith("run_") or name in ("docs_build", "has_migrations")) + } + classified = set(FLAG_COMMANDS) | NOT_RUNNABLE_LOCALLY | {"run_unit_tests", "docs_build"} | NOT_A_CI_JOB + assert run_flags - classified == set(), ( + "new SelectiveChecks flag: add it to FLAG_COMMANDS, NOT_RUNNABLE_LOCALLY or NOT_A_CI_JOB" + ) + assert classified - run_flags == set(), "classified flag no longer exists on SelectiveChecks" + + +@pytest.mark.parametrize("group", ["core", "providers"]) +def test_unit_test_commands_use_the_same_flags_as_the_ci_script(group: str): + ci_script = AIRFLOW_ROOT_PATH / "scripts" / "ci" / "testing" / "run_unit_tests.sh" + ci_flag_sets = { + frozenset(shlex.split(line)[3:]) + for line in ci_script.read_text().splitlines() + if line.strip().startswith(f"breeze testing {group}-tests") + } + for item in build_unit_test_items(group, json.dumps([{"description": "x", "test_types": "Always"}])): + tokens = shlex.split(item.command)[3:] + cut = tokens.index("--parallel-test-types") + assert frozenset(tokens[:cut] + tokens[cut + 2 :]) in ci_flag_sets, item.command + + +def test_empty_diff_prints_only_the_always_items(): + sc = _selective_checks(()) + commands = [item["command"] for item in build_local_verification_plan(sc, (), "main")["items"]] + assert ( + commands[0] + == f"SKIP_BREEZE_PREK_HOOKS=true SKIP={sc.skip_prek_hooks} prek run --from-ref main --to-ref HEAD" + ) + assert not any(command.startswith("breeze testing") for command in commands) + + +def test_release_branch_drops_providers_tests(): + files = ("airflow-core/src/airflow/models/dag.py",) + result = build_local_verification_plan( + _selective_checks(files, default_branch="v3-1-test"), files, "v3-1-test" + ) + commands = [item["command"] for item in result["items"]] + assert any(command.startswith("breeze testing core-tests") for command in commands) + assert not any(command.startswith("breeze testing providers-tests") for command in commands) + + +def test_prek_command_quotes_base_ref(): + result = build_local_verification_plan( + _mock_selective_checks(basic_checks_only=True), (), "branch; echo unexpected" + ) + assert result["items"][0]["command"].endswith( + "prek run --from-ref 'branch; echo unexpected' --to-ref HEAD" + ) diff --git a/dev/breeze/tests/test_verify_commands.py b/dev/breeze/tests/test_verify_commands.py new file mode 100644 index 0000000000000..545b9a2fafb26 --- /dev/null +++ b/dev/breeze/tests/test_verify_commands.py @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import json +from subprocess import CompletedProcess +from unittest.mock import patch + +import pytest +from click.testing import CliRunner + +from airflow_breeze.commands.verify_commands import verify + + +@patch( + "airflow_breeze.commands.verify_commands.get_changed_files_against", + return_value=("airflow-core/docs/index.rst",), +) +def test_json_output_keeps_stdout_parseable(mock_files): + result = CliRunner().invoke(verify, ["--json"], catch_exceptions=False) + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert set(payload) == { + "base_ref", + "default_python_version", + "full_tests_needed", + "changed_files", + "items", + } + assert payload["changed_files"] == ["airflow-core/docs/index.rst"] + assert set(payload["items"][0]) == {"kind", "command", "runs_in", "required"} + assert payload["items"][0]["kind"] == "prek" + + +@patch( + "airflow_breeze.commands.verify_commands.run_command", + return_value=CompletedProcess( + args=[], returncode=128, stdout="", stderr="fatal: Not a valid object name" + ), +) +def test_unknown_base_ref_is_a_clean_usage_error(mock_run): + result = CliRunner().invoke(verify, ["--base-ref", "no-such-branch"]) + assert result.exit_code == 1 + rendered = " ".join(result.output.replace("│", " ").split()) + assert "git merge-base failed for base ref 'no-such-branch': fatal: Not a valid object name" in rendered + + +@patch( + "airflow_breeze.commands.verify_commands.get_changed_files_against", + return_value=("airflow-core/docs/index.rst",), +) +def test_selective_checks_narration_is_hidden_unless_verbose(mock_files): + result = CliRunner().invoke(verify, [], catch_exceptions=False) + assert result.exit_code == 0 + assert "FileGroupForCi" not in result.output + assert "FileGroupForCi" not in result.stderr + + +@pytest.mark.parametrize( + ("files", "expanded"), + [ + (("airflow-core/docs/index.rst",), False), + (("dev/breeze/src/airflow_breeze/breeze.py",), True), + ], +) +def test_full_suite_expansion_is_explained(files: tuple[str, ...], expanded: bool): + with patch("airflow_breeze.commands.verify_commands.get_changed_files_against", return_value=files): + result = CliRunner().invoke(verify, [], catch_exceptions=False) + assert result.exit_code == 0 + assert ("CI runs the full suite for this change" in " ".join(result.output.split())) is expanded From d442e0a599e3d5bda856490ac0b9c883204df726 Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 18:45:34 +0800 Subject: [PATCH 02/18] Print the CI reproduction banner on stderr so breeze verify --json stays parseable In GitHub Actions breeze appends a "how to reproduce locally" banner after every command. It went to stdout, so any command whose stdout is a machine contract, like `breeze verify --json`, produced JSON followed by the banner. The banner is a log aid, so stderr is the right stream for it. --- dev/breeze/src/airflow_breeze/utils/reproduce_ci.py | 4 ++-- dev/breeze/tests/test_reproduce_ci.py | 2 +- dev/breeze/tests/test_verify_commands.py | 11 +++++++++-- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py b/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py index 4a429f556ab8d..21603b5f3827f 100644 --- a/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py +++ b/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py @@ -27,7 +27,7 @@ from rich.markup import escape from airflow_breeze.global_constants import APACHE_AIRFLOW_GITHUB_REPOSITORY -from airflow_breeze.utils.console import get_console +from airflow_breeze.utils.console import get_stderr_console from airflow_breeze.utils.run_utils import commit_sha # Options that are side-effect-only or not meaningful for reproduction (safety net; @@ -211,7 +211,7 @@ def print_local_reproduction(commands: list[ReproductionCommand]) -> None: lines.append(shlex.join(command.argv)) rendered = "\n".join(lines) ruler = "─" * 80 - console = get_console() + console = get_stderr_console() console.print(f"\n[warning]{ruler}[/]") console.print("[warning]HOW TO REPRODUCE LOCALLY[/]\n") console.print(f"[info]{escape(rendered)}[/]\n", soft_wrap=True) diff --git a/dev/breeze/tests/test_reproduce_ci.py b/dev/breeze/tests/test_reproduce_ci.py index 13d9c1b58e626..b762db6c8a755 100644 --- a/dev/breeze/tests/test_reproduce_ci.py +++ b/dev/breeze/tests/test_reproduce_ci.py @@ -114,7 +114,7 @@ def test_build_checkout_reproduction_commands_plain_sha(monkeypatch): assert commands[0].argv == ["git", "checkout", "def456"] -@mock.patch("airflow_breeze.utils.reproduce_ci.get_console", autospec=True) +@mock.patch("airflow_breeze.utils.reproduce_ci.get_stderr_console", autospec=True) def test_print_local_reproduction_renders_copyable_commands(mock_get_console, monkeypatch): monkeypatch.setenv("CI", "true") monkeypatch.setenv("GITHUB_ACTIONS", "true") diff --git a/dev/breeze/tests/test_verify_commands.py b/dev/breeze/tests/test_verify_commands.py index 545b9a2fafb26..6856093a8bc84 100644 --- a/dev/breeze/tests/test_verify_commands.py +++ b/dev/breeze/tests/test_verify_commands.py @@ -17,6 +17,7 @@ from __future__ import annotations import json +import re from subprocess import CompletedProcess from unittest.mock import patch @@ -25,12 +26,18 @@ from airflow_breeze.commands.verify_commands import verify +ANSI = re.compile(r"\x1b\[[0-9;]*m") + @patch( "airflow_breeze.commands.verify_commands.get_changed_files_against", return_value=("airflow-core/docs/index.rst",), ) -def test_json_output_keeps_stdout_parseable(mock_files): +def test_json_output_keeps_stdout_parseable(mock_files, monkeypatch): + # GitHub Actions makes breeze print a "how to reproduce" banner after every command. + monkeypatch.setenv("CI", "true") + monkeypatch.setenv("GITHUB_ACTIONS", "true") + monkeypatch.setenv("GITHUB_SHA", "0" * 40) result = CliRunner().invoke(verify, ["--json"], catch_exceptions=False) assert result.exit_code == 0 payload = json.loads(result.stdout) @@ -55,7 +62,7 @@ def test_json_output_keeps_stdout_parseable(mock_files): def test_unknown_base_ref_is_a_clean_usage_error(mock_run): result = CliRunner().invoke(verify, ["--base-ref", "no-such-branch"]) assert result.exit_code == 1 - rendered = " ".join(result.output.replace("│", " ").split()) + rendered = " ".join(ANSI.sub("", result.output).replace("│", " ").split()) assert "git merge-base failed for base ref 'no-such-branch': fatal: Not a valid object name" in rendered From 851250fde3ce341a48e993787683823d15b1d745 Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 19:30:18 +0800 Subject: [PATCH 03/18] Let a breeze command opt out of the CI reproduction banner instead of moving it to stderr Moving the banner to stderr broke the build-info job: `breeze ci get-workflow-info` and `breeze ci selective-check` write their GitHub Actions outputs on stderr (`2>> $GITHUB_OUTPUT`), so the banner ended up in the outputs file. Neither stream is safe for every command. Keep the banner on stdout and let a command whose stdout is a machine contract, here `breeze verify --json`, switch it off through the click context. --- .../src/airflow_breeze/commands/verify_commands.py | 6 +++++- .../src/airflow_breeze/utils/reproduce_ci.py | 14 ++++++++++---- dev/breeze/tests/test_reproduce_ci.py | 14 +++++++++++++- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/dev/breeze/src/airflow_breeze/commands/verify_commands.py b/dev/breeze/src/airflow_breeze/commands/verify_commands.py index 5af1d69769c25..e79a1c13fc320 100644 --- a/dev/breeze/src/airflow_breeze/commands/verify_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/verify_commands.py @@ -31,6 +31,7 @@ from airflow_breeze.global_constants import GithubEvents from airflow_breeze.utils.console import get_console from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH +from airflow_breeze.utils.reproduce_ci import SKIP_LOCAL_REPRODUCTION from airflow_breeze.utils.run_utils import run_command from airflow_breeze.utils.shared_options import get_verbose from airflow_breeze.utils.verification_plan import build_local_verification_plan @@ -64,9 +65,12 @@ def _git(*args: str) -> list[str]: ) @click.option("--json", "as_json", is_flag=True, help="Print machine-readable JSON instead of a table.") @option_verbose -def verify(base_ref: str, as_json: bool): +@click.pass_context +def verify(ctx: click.Context, base_ref: str, as_json: bool): from airflow_breeze.utils.selective_checks import SelectiveChecks + if as_json: + ctx.meta[SKIP_LOCAL_REPRODUCTION] = True # SelectiveChecks narrates its decisions on stdout; keep stdout for the result only. with contextlib.redirect_stdout(sys.stderr if get_verbose() else io.StringIO()): changed_files = get_changed_files_against(base_ref) diff --git a/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py b/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py index 21603b5f3827f..87dd17f484ba2 100644 --- a/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py +++ b/dev/breeze/src/airflow_breeze/utils/reproduce_ci.py @@ -27,7 +27,7 @@ from rich.markup import escape from airflow_breeze.global_constants import APACHE_AIRFLOW_GITHUB_REPOSITORY -from airflow_breeze.utils.console import get_stderr_console +from airflow_breeze.utils.console import get_console from airflow_breeze.utils.run_utils import commit_sha # Options that are side-effect-only or not meaningful for reproduction (safety net; @@ -211,16 +211,22 @@ def print_local_reproduction(commands: list[ReproductionCommand]) -> None: lines.append(shlex.join(command.argv)) rendered = "\n".join(lines) ruler = "─" * 80 - console = get_stderr_console() + console = get_console() console.print(f"\n[warning]{ruler}[/]") console.print("[warning]HOW TO REPRODUCE LOCALLY[/]\n") console.print(f"[info]{escape(rendered)}[/]\n", soft_wrap=True) console.print(f"[warning]{ruler}[/]\n") +SKIP_LOCAL_REPRODUCTION = "skip_local_reproduction" + + def maybe_print_reproduction(ctx: click.Context) -> None: - """Called by BreezeCommand.invoke() — prints reproduction instructions in CI.""" - if not should_print_local_reproduction(): + """Called by BreezeCommand.invoke() — prints reproduction instructions in CI. + + Commands whose stdout is a machine-readable contract set ``ctx.meta[SKIP_LOCAL_REPRODUCTION]``. + """ + if not should_print_local_reproduction() or ctx.meta.get(SKIP_LOCAL_REPRODUCTION): return github_repository = ctx.params.get("github_repository", APACHE_AIRFLOW_GITHUB_REPOSITORY) diff --git a/dev/breeze/tests/test_reproduce_ci.py b/dev/breeze/tests/test_reproduce_ci.py index b762db6c8a755..5b85b41d3608a 100644 --- a/dev/breeze/tests/test_reproduce_ci.py +++ b/dev/breeze/tests/test_reproduce_ci.py @@ -24,10 +24,12 @@ from airflow_breeze.utils.click_utils import BreezeGroup from airflow_breeze.utils.reproduce_ci import ( + SKIP_LOCAL_REPRODUCTION, ReproductionCommand, build_checkout_reproduction_commands, build_ci_image_reproduction_command, build_reproduction_command_from_context, + maybe_print_reproduction, print_local_reproduction, should_print_local_reproduction, ) @@ -114,7 +116,7 @@ def test_build_checkout_reproduction_commands_plain_sha(monkeypatch): assert commands[0].argv == ["git", "checkout", "def456"] -@mock.patch("airflow_breeze.utils.reproduce_ci.get_stderr_console", autospec=True) +@mock.patch("airflow_breeze.utils.reproduce_ci.get_console", autospec=True) def test_print_local_reproduction_renders_copyable_commands(mock_get_console, monkeypatch): monkeypatch.setenv("CI", "true") monkeypatch.setenv("GITHUB_ACTIONS", "true") @@ -430,3 +432,13 @@ def cmd(**kwargs): ctx = _invoke_and_get_context(cmd, ["-b", "postgres"]) result = build_reproduction_command_from_context(ctx) assert result.argv == ["my-cmd", "--backend", "postgres"] + + +@mock.patch("airflow_breeze.utils.reproduce_ci.print_local_reproduction", autospec=True) +def test_commands_can_opt_out_of_reproduction_banner(mock_print, monkeypatch): + monkeypatch.setenv("CI", "true") + monkeypatch.setenv("GITHUB_ACTIONS", "true") + ctx = click.Context(click.Command("verify")) + ctx.meta[SKIP_LOCAL_REPRODUCTION] = True + maybe_print_reproduction(ctx) + mock_print.assert_not_called() From ec85d6b3a868a6c207843851c5e9809217b6a5bf Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 19:37:44 +0800 Subject: [PATCH 04/18] List renamed files the way CI does when computing breeze verify changed files CI feeds selective checks the output of `git diff-tree`, which has no rename detection and reports a rename as the old path deleted plus the new path added. `git diff --name-only` detects renames by default and lists only the new path, so a rule matching the old path would not fire locally although it fires in CI. --- .../src/airflow_breeze/commands/verify_commands.py | 5 ++++- dev/breeze/tests/test_verify_commands.py | 13 ++++++++++++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/dev/breeze/src/airflow_breeze/commands/verify_commands.py b/dev/breeze/src/airflow_breeze/commands/verify_commands.py index e79a1c13fc320..f7fd98c1186d2 100644 --- a/dev/breeze/src/airflow_breeze/commands/verify_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/verify_commands.py @@ -49,7 +49,10 @@ def _git(*args: str) -> list[str]: return result.stdout.splitlines() merge_base = _git("merge-base", base_ref, "HEAD")[0] - changed = _git("diff", "--name-only", merge_base) + _git("ls-files", "--others", "--exclude-standard") + # CI lists changed files with diff-tree, which reports a rename as a delete plus an add. + changed = _git("diff", "--name-only", "--no-renames", merge_base) + _git( + "ls-files", "--others", "--exclude-standard" + ) return tuple(sorted(set(changed))) diff --git a/dev/breeze/tests/test_verify_commands.py b/dev/breeze/tests/test_verify_commands.py index 6856093a8bc84..884ee180261ef 100644 --- a/dev/breeze/tests/test_verify_commands.py +++ b/dev/breeze/tests/test_verify_commands.py @@ -24,7 +24,7 @@ import pytest from click.testing import CliRunner -from airflow_breeze.commands.verify_commands import verify +from airflow_breeze.commands.verify_commands import get_changed_files_against, verify ANSI = re.compile(r"\x1b\[[0-9;]*m") @@ -89,3 +89,14 @@ def test_full_suite_expansion_is_explained(files: tuple[str, ...], expanded: boo result = CliRunner().invoke(verify, [], catch_exceptions=False) assert result.exit_code == 0 assert ("CI runs the full suite for this change" in " ".join(result.output.split())) is expanded + + +@patch("airflow_breeze.commands.verify_commands.run_command") +def test_changed_files_list_renames_like_ci_diff_tree(mock_run): + mock_run.side_effect = [ + CompletedProcess(args=[], returncode=0, stdout="abc123\n", stderr=""), + CompletedProcess(args=[], returncode=0, stdout="new.py\nold.py\n", stderr=""), + CompletedProcess(args=[], returncode=0, stdout="untracked.py\n", stderr=""), + ] + assert get_changed_files_against("main") == ("new.py", "old.py", "untracked.py") + assert mock_run.call_args_list[1].args[0] == ["git", "diff", "--name-only", "--no-renames", "abc123"] From 372b6cef8867bb44a75a93955135bc68be352c8f Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 19:58:43 +0800 Subject: [PATCH 05/18] Only list jobs triggered by the change in breeze verify Breeze's own unit tests, the shared distribution tests and the translation check run on every PR regardless of what changed, so listing them on every run adds rows nobody acts on and pushes the relevant ones down. Now every row comes from a SelectiveChecks output. The `required` column goes with them: with the translation check gone it was always true. --- dev/breeze/doc/03_developer_tasks.rst | 5 +++-- .../src/airflow_breeze/commands/verify_commands.py | 6 +++--- .../src/airflow_breeze/utils/verification_plan.py | 13 +------------ dev/breeze/tests/test_verification_plan.py | 10 ---------- dev/breeze/tests/test_verify_commands.py | 2 +- 5 files changed, 8 insertions(+), 28 deletions(-) diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst index a7b43e1a8d1d4..de15f3846257c 100644 --- a/dev/breeze/doc/03_developer_tasks.rst +++ b/dev/breeze/doc/03_developer_tasks.rst @@ -332,8 +332,9 @@ usage error such as a base ref git cannot resolve. breeze verify --base-ref upstream/main Each row says what kind of check it is, whether it runs on the host or needs Docker and the -CI image (``breeze``), and the exact command. Use ``--json`` for machine-readable output with -the same fields. +CI image (``breeze``), and the exact command. Jobs CI runs on every PR regardless of the change +(breeze's own unit tests, the shared distributions, the translation check) are not listed. +Use ``--json`` for machine-readable output with the same fields. When a change touches CI tooling or dependency files, selective checks make CI run the full suite and the table grows accordingly. ``breeze verify`` says so under the table. Run what covers your change diff --git a/dev/breeze/src/airflow_breeze/commands/verify_commands.py b/dev/breeze/src/airflow_breeze/commands/verify_commands.py index f7fd98c1186d2..2b8b68f793cfb 100644 --- a/dev/breeze/src/airflow_breeze/commands/verify_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/verify_commands.py @@ -93,11 +93,11 @@ def verify(ctx: click.Context, base_ref: str, as_json: bool): f"[info]{len(changed_files)} changed file(s) against {base_ref}, " f"CI default Python {result['default_python_version']}[/]\n" ) - table = Table(title="Run locally to match CI", show_lines=False) - for column in ("kind", "runs_in", "required", "command"): + table = Table(title="What CI runs for this change", show_lines=False) + for column in ("kind", "runs_in", "command"): table.add_column(column) for item in result["items"]: - table.add_row(item["kind"], item["runs_in"], str(item["required"]).lower(), escape(item["command"])) + table.add_row(item["kind"], item["runs_in"], escape(item["command"])) console.print(table) if result["full_tests_needed"]: console.print( diff --git a/dev/breeze/src/airflow_breeze/utils/verification_plan.py b/dev/breeze/src/airflow_breeze/utils/verification_plan.py index ccc8f58ed9058..a9c42557b6458 100644 --- a/dev/breeze/src/airflow_breeze/utils/verification_plan.py +++ b/dev/breeze/src/airflow_breeze/utils/verification_plan.py @@ -30,7 +30,6 @@ class VerificationItem: kind: str command: str runs_in: str - required: bool = True # SelectiveChecks flag -> (kind, command, runs_in) of the CI job(s) that flag gates. @@ -58,6 +57,7 @@ class VerificationItem: ), "run_ts_sdk_docs": (("docs", "breeze build-docs --sdk-docs-only --sdk=typescript", "breeze"),), "run_breeze_integration_tests": ( + ("unit", "cd dev/breeze && uv run --locked pytest", "host"), ("unit", "cd dev/breeze && uv run --locked pytest -m integration_tests", "host"), ), } @@ -126,17 +126,6 @@ def build_local_verification_plan( items.append( VerificationItem("docs", f"breeze build-docs {sc.docs_list_as_string}".rstrip(), "breeze") ) - items.append(VerificationItem("unit", "cd dev/breeze && uv run --locked pytest", "host")) - if dists := " ".join(sorted(json.loads(sc.shared_distributions_as_json))): - items.append( - VerificationItem( - "unit", - f"for d in {dists}; do (cd shared/$d && uv run --group dev pytest) || exit 1; done", - "host", - ) - ) - # CI runs this with `|| true`, so it never fails a PR. - items.append(VerificationItem("ui", "breeze ui check-translation-completeness", "host", required=False)) return { "base_ref": base_ref, "default_python_version": sc.default_python_version, diff --git a/dev/breeze/tests/test_verification_plan.py b/dev/breeze/tests/test_verification_plan.py index 018d2d6da26dd..a2dd36024e408 100644 --- a/dev/breeze/tests/test_verification_plan.py +++ b/dev/breeze/tests/test_verification_plan.py @@ -61,7 +61,6 @@ def _mock_selective_checks(**flags: object) -> Mock: setattr(sc, flag, False) sc.skip_providers_tests = True sc.skip_prek_hooks = "identity" - sc.shared_distributions_as_json = "[]" sc.default_python_version = DEFAULT_PYTHON_MAJOR_MINOR_VERSION sc.full_tests_needed = False for flag, value in flags.items(): @@ -76,11 +75,7 @@ def test_each_flag_maps_to_its_commands(flag: str): assert commands == [ "SKIP=identity prek run --all-files", *(command for _, command, _ in FLAG_COMMANDS[flag]), - "cd dev/breeze && uv run --locked pytest", - "breeze ui check-translation-completeness", ] - assert [item["required"] for item in result["items"]][-1] is False - assert all(item["required"] for item in result["items"][:-1]) @pytest.mark.parametrize( @@ -112,11 +107,6 @@ def test_docs_only_change_mirrors_the_ci_cell(): assert [item["command"] for item in result["items"]] == [ f"SKIP={sc.skip_prek_hooks} prek run --all-files", "breeze build-docs apache-airflow", - "cd dev/breeze && uv run --locked pytest", - "for d in " - + " ".join(sorted(json.loads(sc.shared_distributions_as_json))) - + "; do (cd shared/$d && uv run --group dev pytest) || exit 1; done", - "breeze ui check-translation-completeness", ] diff --git a/dev/breeze/tests/test_verify_commands.py b/dev/breeze/tests/test_verify_commands.py index 884ee180261ef..322ec106254c9 100644 --- a/dev/breeze/tests/test_verify_commands.py +++ b/dev/breeze/tests/test_verify_commands.py @@ -49,7 +49,7 @@ def test_json_output_keeps_stdout_parseable(mock_files, monkeypatch): "items", } assert payload["changed_files"] == ["airflow-core/docs/index.rst"] - assert set(payload["items"][0]) == {"kind", "command", "runs_in", "required"} + assert set(payload["items"][0]) == {"kind", "command", "runs_in"} assert payload["items"][0]["kind"] == "prek" From cafb5e04543103ebce80c9b157b0dd25f1b69733 Mon Sep 17 00:00:00 2001 From: Andrew Chang Date: Wed, 16 Sep 2026 20:05:58 +0800 Subject: [PATCH 06/18] Make breeze verify list what to run by default and keep the CI expansion behind --full When a change touches CI tooling or dependency files, selective checks make CI run the whole suite. That is the right call for CI, but an agent that executes the list literally would run Go, Java and every provider for a breeze change. The default now applies the same selective-checks rules without that expansion, notes that CI will run more, and --full lists the whole default matrix cell. --- dev/breeze/doc/03_developer_tasks.rst | 33 +++++------ dev/breeze/doc/images/output_verify.svg | 26 ++++++--- dev/breeze/doc/images/output_verify.txt | 2 +- .../commands/developer_commands_config.py | 1 + .../commands/verify_commands.py | 26 ++++++--- .../airflow_breeze/utils/verification_plan.py | 16 ++++-- dev/breeze/tests/test_verification_plan.py | 56 ++++++++++++++++--- dev/breeze/tests/test_verify_commands.py | 6 +- 8 files changed, 117 insertions(+), 49 deletions(-) diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst index de15f3846257c..f58bb779ddc86 100644 --- a/dev/breeze/doc/03_developer_tasks.rst +++ b/dev/breeze/doc/03_developer_tasks.rst @@ -321,14 +321,14 @@ Finding out what CI will run for your change ``breeze verify`` reads the files changed against the target branch (the merge-base of ``--base-ref`` and ``HEAD``, plus untracked files), runs the same selective-checks logic CI uses -and prints the commands you would have to run locally to cover the default CI matrix cell -(default Python, sqlite backend). It is a dry-run: nothing is executed. The only non-zero exit is a -usage error such as a base ref git cannot resolve. +and lists the commands to run locally for the change. Nothing is executed. The only non-zero exit +is a usage error such as a base ref git cannot resolve. .. code-block:: bash breeze verify breeze verify --json + breeze verify --full breeze verify --base-ref upstream/main Each row says what kind of check it is, whether it runs on the host or needs Docker and the @@ -336,19 +336,20 @@ CI image (``breeze``), and the exact command. Jobs CI runs on every PR regardles (breeze's own unit tests, the shared distributions, the translation check) are not listed. Use ``--json`` for machine-readable output with the same fields. -When a change touches CI tooling or dependency files, selective checks make CI run the full suite -and the table grows accordingly. ``breeze verify`` says so under the table. Run what covers your change -locally and leave the rest to CI. - -Local verification is never the full CI matrix. The table covers one matrix cell: the default -Python version on sqlite. Other Python versions, Postgres and MySQL, lowest-dependency runs, -Kubernetes, Helm, e2e suites, the provider compatibility matrix and ARM runners only run in CI. -Three more caveats. Selective checks compare ``pyproject.toml`` contents between ``HEAD`` and ``HEAD^`` -only, so dependency changes that are uncommitted or in earlier commits of your branch are not detected -as such (test selection is unaffected, only the dependency-bump checks are). Untracked files count for -test selection but ``prek`` only sees tracked files, so ``git add`` new files before running the prek -row. Packaging steps CI runs around some tests (building and twine-checking the Task SDK and -airflow-ctl wheels, regenerating the Python API client from its own repository) are not listed. +When a change touches CI tooling or dependency files, selective checks make CI run the full suite. +The default list leaves that expansion out and only shows what the changed files match themselves, +with a note that CI will run more. ``--full`` lists everything CI runs for the default matrix cell +(default Python, sqlite). + +Even ``--full`` is not the full CI matrix. Other Python versions, Postgres and MySQL, +lowest-dependency runs, Kubernetes, Helm, e2e suites, the provider compatibility matrix and ARM +runners only run in CI. Three more caveats. Selective checks compare ``pyproject.toml`` contents +between ``HEAD`` and ``HEAD^`` only, so dependency changes that are uncommitted or in earlier +commits of your branch are not detected as such (test selection is unaffected, only the +dependency-bump checks are). Untracked files count for test selection but ``prek`` only sees +tracked files, so ``git add`` new files before running the prek row. Packaging steps CI runs +around some tests (building and twine-checking the Task SDK and airflow-ctl wheels, regenerating +the Python API client from its own repository) are not listed. These are all available flags of ``verify`` command: diff --git a/dev/breeze/doc/images/output_verify.svg b/dev/breeze/doc/images/output_verify.svg index 2a89a4b6733fc..e930dcba47049 100644 --- a/dev/breeze/doc/images/output_verify.svg +++ b/dev/breeze/doc/images/output_verify.svg @@ -1,4 +1,4 @@ - +