Skip to content

Commit 8e11680

Browse files
jason810496claude
andcommitted
Describe the Lang-SDK e2e modes for contributors
The contributor guide for the e2e tests said nothing about the go_sdk, ts_sdk and java_sdk modes. Explain the wait for the Dag processor before the first test, where a Dag file that must fail goes and why it should fail through its stub tasks, what a new mode records for the wait and the selective checks, and how to run the real probe tests that the same CI job runs. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 9c42b08 commit 8e11680

1 file changed

Lines changed: 43 additions & 0 deletions

File tree

‎contributing-docs/testing/airflow_e2e_tests.rst‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,49 @@ This will trigger Dag execution and validate that the Dag runs successfully.
5656
2. To add new test cases, create a new test file or add it to any existing files in ``./airflow-e2e-tests/tests/*``.
5757

5858

59+
Language SDK modes
60+
-------------------
61+
62+
The ``go_sdk``, ``ts_sdk`` and ``java_sdk`` modes run Dags whose stub tasks run task handlers written in a Language
63+
SDK. Run one with ``breeze testing airflow-e2e-tests --e2e-test-mode go_sdk``. Each builds its bundles in a
64+
toolchain container, or with the host toolchain when ``LANG_SDK_NATIVE_TOOLCHAIN=true`` is set, as CI does. The Dag
65+
processor needs the same artifacts and runtimes as the worker to check the Dag files' stub tasks, which the compose
66+
overrides (``docker/go.yml``, ``java.yml``, ``ts.yml``) give it alongside the worker's own copies.
67+
68+
Before the first test, the tests wait until every Language SDK Dag file of the mode has been parsed, each has
69+
probed a task handler artifact, and the files that fail to import are exactly the ones that are meant to. Import
70+
errors in other files of the Dags folder, such as a stock example Dag, are left alone. If that does not happen in
71+
time, the run fails with the files still missing a probe record and the text of any import error nobody expected.
72+
73+
A Dag file that must fail goes in a test bundle, not in a user-facing example, which the tests pin. The Go test
74+
bundle takes it in ``./airflow-e2e-tests/go-test-bundle/dags``, and the ``go_sdk`` mode copies every Dag file there
75+
wholesale. The Java test bundle takes it in ``./airflow-e2e-tests/java-test-bundle/src/resources/dags``, and
76+
``_setup_java_sdk_integration`` in ``conftest.py`` must also copy it by name, next to ``java_test_dags.py``. An
77+
import error takes its whole Dag file, so keep one Dag file for each import error you expect, and list the file in
78+
both the mode's ``expected_import_errors`` and its ``lang_sdk_dag_files`` in ``conftest.py``.
79+
80+
Let the failing Dag file fail through its stub tasks, such as a task that no artifact registers. Do not make it
81+
fail through an artifact the Dag processor cannot probe: a probe that fails is only a warning, so a file whose
82+
artifact cannot be probed gets no import error at all, and the wait times out instead.
83+
84+
A new Language SDK mode joins ``LANG_SDK_E2E_MODES`` in ``constants.py``, records its ``lang_sdk_dag_files`` and
85+
``expected_import_errors`` in its setup in ``conftest.py``, and puts ``*LANG_SDK_E2E_SHARED_FILES`` in its file
86+
group in ``dev/breeze/src/airflow_breeze/utils/selective_checks.py``. Its setup also writes an
87+
``.airflowignore`` holding ``*`` into each artifact directory it deploys (``_ignore_dag_files_in``), so the
88+
Dag processor does not also try to import the artifacts as Dag files; the stub-task check is unaffected,
89+
because the coordinator scans the filesystem directly.
90+
91+
The same CI job also runs the unit tests that pack and probe the real example of its language, after the e2e tests.
92+
To run one yourself, install the toolchain (Go, Node.js 22 with pnpm, or a JDK) and run:
93+
94+
.. code-block:: bash
95+
96+
AIRFLOW_LANG_SDK_REAL_PROBE_TESTS=1 uv run --project airflow-core --with-editable shared/secrets_masker \
97+
pytest airflow-core/tests/unit/dag_processing/test_task_handler_processor_go.py
98+
99+
Without the toolchain, the test is skipped. In CI, where the ``CI`` variable is set, it fails instead.
100+
101+
59102
Airflow E2E tests in CI
60103
-----------------------
61104

0 commit comments

Comments
 (0)