feat(ingest): add Google Drive ingestion source#18107
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Connector Tests ResultsAll connector tests passed for commit To skip connector tests, add the Autogenerated by the connector-tests CI pipeline. |
|
✅ Meticulous spotted 0 visual differences across 1432 screens tested: view results. Meticulous evaluated ~10 hours of user flows against your PR. Expected differences? Click here. Last updated for commit |
Bundle ReportBundle size has no change ✅ |
i18n locale parityStatus:
deespt-BR |
Cherry-pick of acryldata/datahub-fork#10555 onto OSS master. Adds a new `google-drive` ingestion source that ingests Google Docs (and optionally Slides/Sheets) from Google Drive as DataHub Document entities with full-text markdown content and optional embeddings for semantic search. Supports service-account and Application Default Credentials auth, folder scoping, and content extraction via native markdown export with html2text fallback. Follows the Confluence/Notion source pattern. Cherry-pick adaptations to fix breaking CI: - Add class-level `report: GoogleDriveSourceReport` annotation and make GoogleDriveSourceReport extend StaleEntityRemovalSourceReport so the stateful base class types check (fixes mypy / testQuick lint step). - Regenerate pyproject.toml, uv.lock and constraints.txt for the new google-drive dependency set (fixes setup / Python Wheels lock check). - Regenerate autogenerated connector registry (datahub.json) to include the new connector (fixes the "autogenerated JSON up-to-date" check). Original-PR: acryldata/datahub-fork#10555 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ction The source declared DELETION_DETECTION capability and used stateful config but never instantiated StaleEntityRemovalHandler or overrode get_workunit_processors(), so deletion detection had no effect. - Add imports: StaleEntityRemovalHandler, auto_stale_entity_removal, GenericCheckpointState, MetadataWorkUnitProcessor, partial - In __init__: create self.stale_entity_removal_handler - Override get_workunit_processors() to append the stale-removal processor Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the standard four-file documentation set for the google-drive ingestion source under metadata-ingestion/docs/sources/google-drive/: README.md, google-drive_pre.md, google-drive_post.md, and google-drive_recipe.yml. Documentation accurately reflects Document (not Dataset) entity output and covers all real config fields: credentials.service_account_key_file, credentials.service_account_key_json, folder_ids, include_docs, include_slides, include_sheets, ingest_folders, recursive, document_import_mode, filtering, chunking, embedding, max_documents, advanced, requests_per_minute, platform_instance, parent_document_urn, and stateful_ingestion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ocs, tests) - FIX 1: add mimeType to _DRIVE_FOLDER_FIELDS so _collect_folder_ancestors correctly walks the parent chain (previously the mimeType gate was always False, silently producing no folder hierarchy) - FIX 2: remove dead docs_skipped_unchanged counter and report_doc_skipped_unchanged method (confirmed unused across the entire repo); clarify in docs that every document is re-emitted each run and only the embedding step skips unchanged content based on content_hash - FIX 3: rewrite TestDedupAndTrashedFiles to exercise get_workunits_internal directly rather than a self-referential local helper - FIX 4: remove low-value TestGoogleDriveSourceConfigDefaults (pydantic field defaults); remove brittle error-message substring check in test_both_key_file_and_key_json_raises - FIX 5: reconcile contradictory rate-limit quota numbers in google-drive_post.md (connector default is 60 req/min; Google's actual limit presented separately) - Add TestCollectFolderAncestors unit test that fails without the mimeType fix and passes with it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds 38 new unit tests covering authentication (service account key file, key JSON, ADC fallback), content export (markdown, HTML fallback, plain text), file listing with pagination, folder chain ingestion, document building (NATIVE/EXTERNAL modes), chunking error handling, continue_on_failure, get_report, close, and test_connection. Coverage rises from 53% → 94% on google_drive_source.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ow installed google libs)
Remove module-level sys.modules injection that replaced the real installed
google.auth and google.oauth2 packages with MagicMocks, which broke other
connectors' tests under random ordering (bigquery/gcs/dataplex/vertexai all
depend on google-auth).
- Delete the module-level for-loop that unconditionally stubbed google.auth,
google.oauth2, and related packages at import/collection time.
- Introduce an autouse pytest fixture (missing_google_stubs) that injects
MagicMock stubs only for genuinely missing packages (googleapiclient,
html2text) via patch.dict(sys.modules, ...), which fully restores sys.modules
after each test even on failure.
- Replace the two test methods that mutated sys.modules["google.oauth2"] in-place
with restoring patches: patch("google.oauth2.service_account.Credentials.from_service_account_file", ...)
and patch("google.auth.default", ...) — no more permanent namespace corruption.
- Remove now-unnecessary noqa: E402 comments and unused `types` import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ort-path check test_packages::test_check_import_paths uses naive substring matching: a line containing both "import" and "src." is rejected. The local variable `src` combined with `.document_import_mode` tripped it. Renaming the variable to `source` resolves the false positive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l dedup, surface inaccessible folders Found by running the connector live (unit tests mocked DocumentChunkingSourceConfig): - Omit embedding=None / max_documents=None when building DocumentChunkingSourceConfig (both non-optional with defaults; None overrides and fails validation -> startup crash). - Remove manual StaleEntityRemovalHandler + get_workunit_processors override; the base class auto-adds AutoStaleEntityRemovalProcessor because the report is a StaleEntityRemovalSourceReport, so manual wiring double-registered the job_id (AssertionError). Deletion detection still works. - Surface inaccessible folders (404/403) instead of silently ingesting zero docs; the failure links to the setup docs and notes service accounts are not org members (must share the folder with the SA directly). - Type the html2text converter as Optional so it type-checks with or without html2text installed. - Add regression tests (real DocumentChunkingSourceConfig; _verify_folder_access) and document the SA sharing requirement + troubleshooting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…user_email) Add a `delegated_user_email` config option so the service account can impersonate a Workspace user (domain-wide delegation) and read that user's Drive / org-shared content without each folder being shared with the service account directly. Applies only to service-account auth; ignored (with a warning) for Application Default Credentials. Documents the admin-console setup and adds unit tests for the delegation path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sAllDrives) The files().list / files().get calls omitted includeItemsFromAllDrives / supportsAllDrives, so files in (or shared via) Shared Drives were invisible — discovery returned zero files even when the account could access them. Add the flags to the folder listing, all-Drive listing, folder-metadata fetch, and test_connection calls. Verified live: ingestion now discovers Shared-Drive content, extracts Google Doc markdown, and builds the folder hierarchy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g guards, sheets CSV) - report_doc_failed / report_doc_skipped_too_short: use a stable title/message as the structured-log aggregation key and put per-document detail in context (was passing the unique file_id as the message, defeating error grouping). - Remove an internal system name from the module docstring (public repo). - Add gt=0 to requests_per_minute (avoid ZeroDivisionError) and to max_documents; guard the folder_ids validator against non-list input; hoist `import re` to top. - Export Google Sheets as CSV (preserving tabular structure) instead of plain text, matching the documented behavior and using the previously-dead EXPORT_CSV. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…older-metadata Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…undled logo Add the google-drive platform to data-platforms.yaml (bootstrap MCP) so it has a proper displayName and logo by default, add a bundled googledrivelogo.png asset, and point the runtime DataPlatformInfo logoUrl at the bundled asset instead of an external hot-linked gstatic URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… helpers Consolidate _export_doc_as_markdown/_export_doc_as_html/_export_as_plain_text into a single _export_media helper, and factor the repeated NATIVE/EXTERNAL dispatch in _build_document/_build_folder_document into _create_document. Preserves the markdown-specific 403/415 debug logging, _last_extraction_error tracking, and the sheets-CSV/slides-text export formats. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ive flags; tighten assertions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… report Add capped LossyList samples (documents_created, folders_created) to the source report and populate them at emit time, so the structured report shows examples of the URNs produced. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Notion-style)
Derive the document URN from the globally-unique Drive file ID alone
(urn:li:document:google-drive-{file_id}), with platform_instance as an optional
explicit prefix — instead of hashing the credential path / key JSON / "adc".
This mirrors how the Notion source keys off the page ID and how Confluence/GitHub
key off a stable natural identity, and fixes a latent gotcha where the same
document got different URNs across key-file paths, key rotation, ADC, or a change
of service account. The dataPlatformInstance aspect now carries an instance only
when platform_instance is set. Documents the URN scheme + stability in the config
field and connector docs; updates tests accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1680bfe to
f8037d1
Compare
Summary
Adds a new
google-driveingestion source that ingests Google Docs (and optionally Google Sheets / Slides) from Google Drive as DataHub Document entities, with full-text markdown content and optional embeddings for semantic search. It follows the existing doc-source pattern (Confluence / Notion / GitHub Documents).Highlights:
folder_ids(recursively) or all accessible files; toggles for Docs / Sheets / Slides; optional folder-hierarchy materialisation.html2textfallback.StaleEntityRemovalHandler.Changes
metadata-ingestion/src/datahub/ingestion/source/google_drive/(google_drive_config.py,google_drive_report.py,google_drive_source.py).setup.pyplugin + entry point (google-drive) and dependency set; regeneratedpyproject.toml,uv.lock,constraints.txt, and the autogenerated connector registry.metadata-ingestion/tests/unit/ingestion/source/google_drive/test_google_drive_source.py(mocked Drive API).metadata-ingestion/docs/sources/google-drive/.Test plan
pytest tests/unit/ingestion/source/google_drive/— 37 passing (config validation, folder-URL normalisation, dedup/trashed filtering, folder-ancestor walk, document/property building, content-length handling).ruff check/ruff format --checkclean;mypy0 errors.:metadata-ingestion:lint,:checkLockFile,:connectorRegistry,:lineageGenall pass.Checklist
metadata-ingestion/docs/sources/google-drive/)updating-datahub.mdentry needed)🤖 Generated with Claude Code