Skip to content

Check stub tasks against their task handlers - #74067

Draft
jason810496 wants to merge 2 commits into
jason/core-taskhandler-refactor/07-ts-sdk-task-handler-parsefrom
jason/core-taskhandler-refactor/13-stub-task-checks
Draft

jason810496 wants to merge 2 commits into
jason/core-taskhandler-refactor/07-ts-sdk-task-handler-parsefrom
jason/core-taskhandler-refactor/13-stub-task-checks

Conversation

@jason810496

@jason810496 jason810496 commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Stack (bottom to top), on native #74037 (stack #74170): #73973, #73974, #73975, #74317, #73976, #74067, #74135, #74140, #74141, #73971, #74030, #74031, #74032, #74136, #74137, #74138, #74139

Adds the checks that compare a Dag's stub tasks with the task handlers a probe reports, so a missing handler or an argument a handler cannot take fails the Dag file's import instead of the task on a worker. They are pure functions, and nothing calls them yet: #74135 does.

stub_tasks = collect_stub_tasks(bag.dags.values(), serialized_dags)
etl_go = [stub for stub in stub_tasks if stub.dag_id == "etl" and stub.queue == "go"]
answer = TaskHandlerAnswer(
    bundle_name="go-task-handlers",
    rel_path="bin/etl",
    task_handlers=result.task_handlers,  # the TaskHandlerParsingResult of the probe
)
match = match_task_handlers(etl_go, answer)  # one Dag's stub tasks that route to one coordinator
match.warnings  # name mismatches under named binding, for the parse log
format_import_errors(match.problems)
# {"dags/etl.py": "Stub tasks in dags/etl.py do not match their task handlers:\n"
#                 "- Dag 'etl', task 'load': 'bin/etl' in Dag bundle 'go-task-handlers' registers no task handler for it\n"
#                 "- Dag 'etl', task 'transform' ('bin/etl' in Dag bundle 'go-task-handlers'): "
#                 "argument 'count' is integer or null, the task handler takes integer"}
  • A stub task with no handler is a problem, but a handler with no stub task is not, because one artifact registers handlers for many Dag files. ADR-0011 used to require the ids to match exactly.
  • Arguments are checked the way the runtimes bind them:
Declaration Import error Parse-log warning
positional An argument count that matches the params neither with nor without the defaulted arguments, or a value type the param at that position does not accept none
named A value type the matched param does not accept An argument no param takes, or a param no argument fills, unless a lone argument may be the whole value
params=None, or a mapped stub task none, only the handler's presence none
  • Under named, a param takes the argument of its exact name, else, unless exact_name is set, the one whose name folds to its own (lower case, _ removed). A folded name two arguments share matches neither.
  • Types are compared only where both sides have a schema, as the Go runtime compares them: an argument that may be null needs a param that accepts null, and at least one of its other types must be accepted, with integer accepted by number. So int | str binds to an integer param and int | None does not. $ref, format, ranges and nested items are not compared.
  • Arguments are read from the serialized Dag as JSON, so they are what the worker gets: a tuple becomes a list, and a dict key a string.
  • ADR-0011 (Step 5, Consequences and Appendix B) states these rules, and ADR-0012 and ADR-0013 point to it.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines

@jason810496
jason810496 added this pull request to stack #73978 October 2, 2026 05:49
@jason810496 jason810496 changed the title jason/core taskhandler refactor/13 stub task checks Check stub tasks against their task handlers Oct 2, 2026
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/13-stub-task-checks branch from f8d8146 to 6fb1dd2 Compare October 2, 2026 11:36
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/13-stub-task-checks branch from 6fb1dd2 to cd7fae0 Compare October 2, 2026 14:57
@jason810496 jason810496 self-assigned this Oct 2, 2026
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/13-stub-task-checks branch from cd7fae0 to a857663 Compare October 3, 2026 07:50
The Dag processor will check each stub task against the task handlers of one artifact, the one its coordinator would run for the stub task's Dag, so that a missing handler or an argument the handler cannot take fails the Dag file's import instead of the task on a worker. The argument rules mirror how the Go and Java runtimes bind arguments, defaulted ones included, so the check fails only what would fail at run time and only warns where the runtime still runs the task. The checks are pure functions of the parsed Dags and one probe answer, so every rule is tested with a table and the wiring into the parse can follow on its own.
ADR-0011 still required a Dag's stub task ids to match its handler ids exactly, compared parameter names in order, and unioned the declarations across coordinators. One artifact registers handlers for many Dag files, so a handler with no stub task is expected, and a stub task only runs on the coordinator its queue routes to. ADR-0011 now checks each stub task against the one answer for its coordinator and Dag, and states the binding, defaulted-argument, argless, mapped and schema rules; ADR-0012 points to it.
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/13-stub-task-checks branch from a857663 to 893fffa Compare October 5, 2026 18:12
@jason810496
jason810496 removed this pull request from stack #73978 October 6, 2026 02:27
@jason810496
jason810496 changed the base branch from jason/core-taskhandler-refactor/12-zip-dag-discovery to jason/core-taskhandler-refactor/07-ts-sdk-task-handler-parse October 6, 2026 02:30
@jason810496
jason810496 added this pull request to stack #74318 October 6, 2026 02:35
@jason810496
jason810496 force-pushed the jason/core-taskhandler-refactor/13-stub-task-checks branch from 893fffa to c72868e Compare October 6, 2026 02:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant