Skip to content

feat(ingest): add Google Drive ingestion source#18107

Open
pedro93 wants to merge 20 commits into
masterfrom
feat/google-drive-ingestion
Open

feat(ingest): add Google Drive ingestion source#18107
pedro93 wants to merge 20 commits into
masterfrom
feat/google-drive-ingestion

Conversation

@pedro93

@pedro93 pedro93 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a new google-drive ingestion source that ingests Google Docs (and optionally Google Sheets / Slides) from Google Drive as DataHub Document entities, with full-text markdown content and optional embeddings for semantic search. It follows the existing doc-source pattern (Confluence / Notion / GitHub Documents).

Highlights:

  • Auth: service-account key (file or inline JSON) or Application Default Credentials.
  • Scope: crawl specific folder_ids (recursively) or all accessible files; toggles for Docs / Sheets / Slides; optional folder-hierarchy materialisation.
  • Content: native markdown export with an html2text fallback.
  • Stateful ingestion with stale-entity-removal (deletion detection) wired via StaleEntityRemovalHandler.
  • Semantic search: optional chunking + embeddings.

Changes

  • New source: metadata-ingestion/src/datahub/ingestion/source/google_drive/ (google_drive_config.py, google_drive_report.py, google_drive_source.py).
  • setup.py plugin + entry point (google-drive) and dependency set; regenerated pyproject.toml, uv.lock, constraints.txt, and the autogenerated connector registry.
  • Unit tests: metadata-ingestion/tests/unit/ingestion/source/google_drive/test_google_drive_source.py (mocked Drive API).
  • Connector docs: metadata-ingestion/docs/sources/google-drive/.

Test plan

  • pytest tests/unit/ingestion/source/google_drive/ — 37 passing (config validation, folder-URL normalisation, dedup/trashed filtering, folder-ancestor walk, document/property building, content-length handling).
  • ruff check / ruff format --check clean; mypy 0 errors.
  • :metadata-ingestion:lint, :checkLockFile, :connectorRegistry, :lineageGen all pass.

Checklist

  • PR conforms to the Contributing Guideline (PR Title Format)
  • Tests added
  • Docs added (connector docs under metadata-ingestion/docs/sources/google-drive/)
  • No breaking changes (new optional source; no updating-datahub.md entry needed)

🤖 Generated with Claude Code

@github-actions github-actions Bot added the ingestion PR or Issue related to the ingestion of metadata label Jun 30, 2026
@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.09856% with 19 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...gestion/source/google_drive/google_drive_source.py 95.53% 18 Missing ⚠️
...gestion/source/google_drive/google_drive_config.py 97.87% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@datahub-connector-tests

datahub-connector-tests Bot commented Jun 30, 2026

Copy link
Copy Markdown

Connector Tests Results

All connector tests passed for commit f8037d1

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

@alwaysmeticulous

alwaysmeticulous Bot commented Jun 30, 2026

Copy link
Copy Markdown

✅ Meticulous spotted 0 visual differences across 1432 screens tested: view results.

Meticulous evaluated ~10 hours of user flows against your PR.

Expected differences? Click here. Last updated for commit f8037d1 fix(ingest/google-drive): make document URNs credential-independent (Not.... This comment will update as new commits are pushed.

@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Bundle Report

Bundle size has no change ✅

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

i18n locale parity

Status: ⚠️ drift detected (warn-only) — 6 issue(s) across 3 locale(s)

Locale Missing Stale Placeholder
de 2 0 0
es 2 0 0
pt-BR 2 0 0
de
❌ governance.domain.json — 5 missing key(s):
      - navigator.emptyFiltered
      - navigator.overview
      - navigator.section.allDomains
      - navigator.ownerFilter.label
      - navigator.ownerFilter.placeholder
❌ governance.structured-properties.json — 4 missing key(s):
      - allowedPlatforms.anyPlaceholder
      - allowedPlatforms.anyTooltip
      - allowedPlatforms.title
      - allowedPlatforms.tooltip
es
❌ governance.domain.json — 5 missing key(s):
      - navigator.emptyFiltered
      - navigator.overview
      - navigator.section.allDomains
      - navigator.ownerFilter.label
      - navigator.ownerFilter.placeholder
❌ governance.structured-properties.json — 4 missing key(s):
      - allowedPlatforms.anyPlaceholder
      - allowedPlatforms.anyTooltip
      - allowedPlatforms.title
      - allowedPlatforms.tooltip
pt-BR
❌ governance.domain.json — 5 missing key(s):
      - navigator.emptyFiltered
      - navigator.overview
      - navigator.section.allDomains
      - navigator.ownerFilter.label
      - navigator.ownerFilter.placeholder
❌ governance.structured-properties.json — 4 missing key(s):
      - allowedPlatforms.anyPlaceholder
      - allowedPlatforms.anyTooltip
      - allowedPlatforms.title
      - allowedPlatforms.tooltip

pedro93 and others added 2 commits July 1, 2026 11:58
Cherry-pick of acryldata/datahub-fork#10555 onto OSS master.

Adds a new `google-drive` ingestion source that ingests Google Docs
(and optionally Slides/Sheets) from Google Drive as DataHub Document
entities with full-text markdown content and optional embeddings for
semantic search. Supports service-account and Application Default
Credentials auth, folder scoping, and content extraction via native
markdown export with html2text fallback. Follows the Confluence/Notion
source pattern.

Cherry-pick adaptations to fix breaking CI:
- Add class-level `report: GoogleDriveSourceReport` annotation and make
  GoogleDriveSourceReport extend StaleEntityRemovalSourceReport so the
  stateful base class types check (fixes mypy / testQuick lint step).
- Regenerate pyproject.toml, uv.lock and constraints.txt for the new
  google-drive dependency set (fixes setup / Python Wheels lock check).
- Regenerate autogenerated connector registry (datahub.json) to include
  the new connector (fixes the "autogenerated JSON up-to-date" check).

Original-PR: acryldata/datahub-fork#10555

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ction

The source declared DELETION_DETECTION capability and used stateful config
but never instantiated StaleEntityRemovalHandler or overrode
get_workunit_processors(), so deletion detection had no effect.

- Add imports: StaleEntityRemovalHandler, auto_stale_entity_removal,
  GenericCheckpointState, MetadataWorkUnitProcessor, partial
- In __init__: create self.stale_entity_removal_handler
- Override get_workunit_processors() to append the stale-removal processor

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pedro93 and others added 18 commits July 1, 2026 11:58
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the standard four-file documentation set for the google-drive
ingestion source under metadata-ingestion/docs/sources/google-drive/:
README.md, google-drive_pre.md, google-drive_post.md, and
google-drive_recipe.yml.

Documentation accurately reflects Document (not Dataset) entity output
and covers all real config fields: credentials.service_account_key_file,
credentials.service_account_key_json, folder_ids, include_docs,
include_slides, include_sheets, ingest_folders, recursive,
document_import_mode, filtering, chunking, embedding, max_documents,
advanced, requests_per_minute, platform_instance, parent_document_urn,
and stateful_ingestion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ocs, tests)

- FIX 1: add mimeType to _DRIVE_FOLDER_FIELDS so _collect_folder_ancestors
  correctly walks the parent chain (previously the mimeType gate was always
  False, silently producing no folder hierarchy)
- FIX 2: remove dead docs_skipped_unchanged counter and report_doc_skipped_unchanged
  method (confirmed unused across the entire repo); clarify in docs that every
  document is re-emitted each run and only the embedding step skips unchanged
  content based on content_hash
- FIX 3: rewrite TestDedupAndTrashedFiles to exercise get_workunits_internal
  directly rather than a self-referential local helper
- FIX 4: remove low-value TestGoogleDriveSourceConfigDefaults (pydantic field
  defaults); remove brittle error-message substring check in
  test_both_key_file_and_key_json_raises
- FIX 5: reconcile contradictory rate-limit quota numbers in google-drive_post.md
  (connector default is 60 req/min; Google's actual limit presented separately)
- Add TestCollectFolderAncestors unit test that fails without the mimeType fix
  and passes with it

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds 38 new unit tests covering authentication (service account key
file, key JSON, ADC fallback), content export (markdown, HTML fallback,
plain text), file listing with pagination, folder chain ingestion,
document building (NATIVE/EXTERNAL modes), chunking error handling,
continue_on_failure, get_report, close, and test_connection.

Coverage rises from 53% → 94% on google_drive_source.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ow installed google libs)

Remove module-level sys.modules injection that replaced the real installed
google.auth and google.oauth2 packages with MagicMocks, which broke other
connectors' tests under random ordering (bigquery/gcs/dataplex/vertexai all
depend on google-auth).

- Delete the module-level for-loop that unconditionally stubbed google.auth,
  google.oauth2, and related packages at import/collection time.
- Introduce an autouse pytest fixture (missing_google_stubs) that injects
  MagicMock stubs only for genuinely missing packages (googleapiclient,
  html2text) via patch.dict(sys.modules, ...), which fully restores sys.modules
  after each test even on failure.
- Replace the two test methods that mutated sys.modules["google.oauth2"] in-place
  with restoring patches: patch("google.oauth2.service_account.Credentials.from_service_account_file", ...)
  and patch("google.auth.default", ...) — no more permanent namespace corruption.
- Remove now-unnecessary noqa: E402 comments and unused `types` import.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ort-path check

test_packages::test_check_import_paths uses naive substring matching: a line
containing both "import" and "src." is rejected. The local variable `src`
combined with `.document_import_mode` tripped it. Renaming the variable to
`source` resolves the false positive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l dedup, surface inaccessible folders

Found by running the connector live (unit tests mocked DocumentChunkingSourceConfig):
- Omit embedding=None / max_documents=None when building DocumentChunkingSourceConfig (both non-optional with defaults; None overrides and fails validation -> startup crash).
- Remove manual StaleEntityRemovalHandler + get_workunit_processors override; the base class auto-adds AutoStaleEntityRemovalProcessor because the report is a StaleEntityRemovalSourceReport, so manual wiring double-registered the job_id (AssertionError). Deletion detection still works.
- Surface inaccessible folders (404/403) instead of silently ingesting zero docs; the failure links to the setup docs and notes service accounts are not org members (must share the folder with the SA directly).
- Type the html2text converter as Optional so it type-checks with or without html2text installed.
- Add regression tests (real DocumentChunkingSourceConfig; _verify_folder_access) and document the SA sharing requirement + troubleshooting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…user_email)

Add a `delegated_user_email` config option so the service account can
impersonate a Workspace user (domain-wide delegation) and read that user's
Drive / org-shared content without each folder being shared with the service
account directly. Applies only to service-account auth; ignored (with a
warning) for Application Default Credentials. Documents the admin-console
setup and adds unit tests for the delegation path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sAllDrives)

The files().list / files().get calls omitted includeItemsFromAllDrives /
supportsAllDrives, so files in (or shared via) Shared Drives were invisible —
discovery returned zero files even when the account could access them. Add the
flags to the folder listing, all-Drive listing, folder-metadata fetch, and
test_connection calls. Verified live: ingestion now discovers Shared-Drive
content, extracts Google Doc markdown, and builds the folder hierarchy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g guards, sheets CSV)

- report_doc_failed / report_doc_skipped_too_short: use a stable title/message
  as the structured-log aggregation key and put per-document detail in context
  (was passing the unique file_id as the message, defeating error grouping).
- Remove an internal system name from the module docstring (public repo).
- Add gt=0 to requests_per_minute (avoid ZeroDivisionError) and to max_documents;
  guard the folder_ids validator against non-list input; hoist `import re` to top.
- Export Google Sheets as CSV (preserving tabular structure) instead of plain
  text, matching the documented behavior and using the previously-dead EXPORT_CSV.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…older-metadata

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…undled logo

Add the google-drive platform to data-platforms.yaml (bootstrap MCP) so it has a
proper displayName and logo by default, add a bundled googledrivelogo.png asset,
and point the runtime DataPlatformInfo logoUrl at the bundled asset instead of an
external hot-linked gstatic URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… helpers

Consolidate _export_doc_as_markdown/_export_doc_as_html/_export_as_plain_text
into a single _export_media helper, and factor the repeated NATIVE/EXTERNAL
dispatch in _build_document/_build_folder_document into _create_document.
Preserves the markdown-specific 403/415 debug logging, _last_extraction_error
tracking, and the sheets-CSV/slides-text export formats.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ive flags; tighten assertions

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… report

Add capped LossyList samples (documents_created, folders_created) to the source
report and populate them at emit time, so the structured report shows examples
of the URNs produced.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Notion-style)

Derive the document URN from the globally-unique Drive file ID alone
(urn:li:document:google-drive-{file_id}), with platform_instance as an optional
explicit prefix — instead of hashing the credential path / key JSON / "adc".
This mirrors how the Notion source keys off the page ID and how Confluence/GitHub
key off a stable natural identity, and fixes a latent gotcha where the same
document got different URNs across key-file paths, key rotation, ADC, or a change
of service account. The dataPlatformInstance aspect now carries an instance only
when platform_instance is set. Documents the URN scheme + stability in the config
field and connector docs; updates tests accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ingestion PR or Issue related to the ingestion of metadata needs-review Label for PRs that need review from a maintainer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants