Repository navigation
Databricks: support Genie consultations and resumable standalone tasks #74194
Description
Activity
I'd like to take the first bounded slice: the Databricks Genie hook plus common-ai managed-agent toolset adapter, with connection auth, space/conversation/message identity, structured results/errors, optional common-ai dependency handling, docs, and mocked unit coverage. I will keep standalone resumable tasks out of this first PR.
Thanks @garvit-arora, that slice is yours. Hook + managed-agent toolset adapter first, resumable standalone tasks later, exactly as scoped.
A few things that will make review quick:
- Build the adapter on the public
BaseManagedAgentToolsetcontract from Add a vendor-neutral managed-agent hook contract to Common AI #73532 (common/ai/toolsets/managed_agent.py) and don't override private common.ai internals. Keep the adapter thin and separate from the hook, so the hook stands on its own and common.ai stays an optional extra. - Starting a conversation or posting a message is not idempotent. On an ambiguous failure (timeout, 5xx after send) surface an error rather than retrying, and only retry reads (get message, fetch result).
- Bound result size and keep tokens out of anything serialized. Return space/conversation/message IDs plus a truncated, structured result.
- Map 401/403/404/429 to clear errors. Note that Databricks sometimes answers a missing resource with 403.
- Docs: say when the managed Genie MCP server via Unity Gateway is enough, and what the native API adds.
- Please include one real-workspace run (PAT is fine) recorded separately from the mocked tests.
My review on #74198 covers several of the same pitfalls if you want a checklist. Happy to review once it's up.
- Build the adapter on the public
Description
Track a Databricks Genie integration delivered in two bounded PRs: a hook plus managed-agent toolset adapter, then a standalone operator with deferrable waiting where supported.
Use case/motivation
A worker-owned agent may need a bounded consultation with a configured Genie space. A standalone DAG task may instead wait for a longer conversation/message operation. These have different resource and recovery requirements but should share authentication and result handling.
Delivery and acceptance
Related issues
This is a provider-local proposal, independent of #74191. Reuse existing resumable-job facilities only for Airflow versions that support them; document the supported matrix. Initial deliverable does not include Genie administration APIs or an Airflow HITL bridge.