Skip to content

Commit cbe8096

Browse files
committed
Fix docs omitting asset from the Dag discovery safe-mode heuristic
Signed-off-by: PoAn Yang <payang@apache.org>
1 parent 98747f8 commit cbe8096

4 files changed

Lines changed: 30 additions & 9 deletions

File tree

‎airflow-core/docs/core-concepts/dags.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -164,14 +164,14 @@ While both Dag constructors get called when the file is accessed, only ``dag_1``
164164

165165
.. note::
166166

167-
When searching for Dags inside the Dag bundle, Airflow only considers Python files that contain the strings ``airflow`` and ``dag`` (case-insensitively) as an optimization.
167+
When searching for Dags inside the Dag bundle, Airflow only considers Python files that contain ``airflow`` and at least one of ``dag`` or ``asset`` (all matched case-insensitively) as an optimization.
168168

169169
To consider all Python files instead, set the ``[core] dag_discovery_safe_mode`` configuration flag to ``False``. This is the setting to use when your Dags are defined through a wrapper or abstraction whose source does not contain those strings. The flag is read by the Dag file processor, so set it on that component (and on anything that runs ``airflow dags reserialize``) and restart it for the change to take effect -- otherwise Dags may appear after a manual reserialize and then disappear again on the next processor scan.
170170

171171
You can also provide an ``.airflowignore`` file inside your Dag bundle, or any of its subfolders, which describes patterns of files for the loader to ignore. It covers the directory it's in plus all subfolders underneath it. See :ref:`.airflowignore <concepts:airflowignore>` below for details of the file syntax.
172172

173173
In the case where the ``.airflowignore`` does not meet your needs and you want a more flexible way to control if a python file needs to be parsed by airflow, you can plug your callable by setting ``might_contain_dag_callable`` in the config file.
174-
Note, this callable will replace the default Airflow heuristic, i.e. checking if the strings ``airflow`` and ``dag`` (case-insensitively) are present in the python file.
174+
Note, this callable will replace the default Airflow heuristic, i.e. checking if the python file contains ``airflow`` and at least one of ``dag`` or ``asset`` (all matched case-insensitively).
175175

176176
.. code-block::
177177

‎airflow-core/docs/faq.rst‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,10 @@ There are very many reasons why your task might not be getting scheduled. Here a
3636
may want to confirm that this works both where the scheduler runs as well
3737
as where the worker runs.
3838

39-
- Does the file containing your Dag contain the string ``airflow`` and ``DAG`` somewhere
40-
in the contents? When searching the Dag directory, Airflow ignores files not containing
41-
``airflow`` and ``DAG`` in order to prevent the DagBag parsing from importing all python
42-
files collocated with user's Dags.
39+
- Does the file containing your Dag contain ``airflow`` and at least one of ``dag`` or
40+
``asset`` somewhere in the contents (all matched case-insensitively)? When searching the
41+
Dag directory, Airflow skips files that do not match, in order to prevent the DagBag
42+
parsing from importing all python files collocated with user's Dags.
4343

4444
- Is your ``start_date`` set properly? For time-based Dags, the task won't be triggered until the
4545
the first schedule interval following the start date has passed.

‎airflow-core/src/airflow/config_templates/config.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -320,9 +320,10 @@ core:
320320
default: "True"
321321
dag_discovery_safe_mode:
322322
description: |
323-
If enabled, Airflow only scans files containing both ``DAG`` and ``airflow`` (case-insensitive)
324-
when looking for Dags. Set this to ``False`` to scan every Python file instead -- required when
325-
your Dags are defined through a wrapper or abstraction whose source does not contain those strings.
323+
If enabled, Airflow only scans files containing ``airflow`` and at least one of ``dag`` or
324+
``asset`` (all matched case-insensitively) when looking for Dags. Set this to ``False`` to scan
325+
every Python file instead -- required when your Dags are defined through a wrapper or abstraction
326+
whose source does not contain those strings.
326327
327328
This setting is read by the Dag file processor, so it must be configured for that component. In a
328329
separate-process deployment (for example the Helm chart) set it on the ``dag-processor`` as well as

‎airflow-core/tests/unit/utils/test_file.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
from airflow.utils.file import (
3030
correct_maybe_zipped,
3131
list_py_file_paths,
32+
might_contain_dag_via_default_heuristic,
3233
open_maybe_zipped,
3334
)
3435

@@ -229,3 +230,22 @@ def test_get_unique_dag_module_name(edge_filename, expected_modification):
229230
mocked_sha1.return_value.hexdigest.return_value = "mocked_path_hash_sha1"
230231
modify_module_name = file_utils.get_unique_dag_module_name(edge_filename)
231232
assert modify_module_name == expected_modification
233+
234+
235+
@pytest.mark.parametrize(
236+
("content", "expected"),
237+
[
238+
pytest.param(b"from airflow.sdk import DAG", True, id="airflow-and-dag"),
239+
pytest.param(b"from airflow.sdk import asset", True, id="airflow-and-asset"),
240+
pytest.param(b"from airflow.sdk import Asset", True, id="mixed-case-asset"),
241+
pytest.param(
242+
b"DAG_DEFAULTS = {'asset_bucket': 's3://data-lake'}", False, id="keywords-without-airflow"
243+
),
244+
pytest.param(b"from airflow.configuration import conf", False, id="airflow-without-dag-or-asset"),
245+
],
246+
)
247+
def test_might_contain_dag_via_default_heuristic(tmp_path, content, expected):
248+
candidate = tmp_path / "candidate.py"
249+
candidate.write_bytes(content)
250+
251+
assert might_contain_dag_via_default_heuristic(file_path=str(candidate)) is expected

0 commit comments

Comments
 (0)