Skip to content

ENG-2635: Add namespace support for Google Cloud SQL Postgres#7507

Open
JadeCara wants to merge 6 commits intomainfrom
ENG-2635-gcp-postgres-namespace-support
Open

ENG-2635: Add namespace support for Google Cloud SQL Postgres#7507
JadeCara wants to merge 6 commits intomainfrom
ENG-2635-gcp-postgres-namespace-support

Conversation

@JadeCara
Copy link
Contributor

@JadeCara JadeCara commented Feb 26, 2026

Ticket ENG-2635

Description Of Changes

Add namespace metadata support to the Google Cloud SQL for Postgres connector, enabling schema-qualified and optionally database-qualified table names in generated SQL queries. This is PR 2 of the ENG-2635 work — PR 1 (#7500) added the same capability for standard Postgres.

When a dataset has fides_meta.namespace configured with a schema (and optionally database_name), queries are generated with qualified table names like "billing"."customer" or "prod_db"."billing"."customer" instead of just "customer".

The set_schema connection secret (db_schema) path remains as the legacy approach. The two mechanisms are mutually exclusive in practice — users configure one or the other.

This is PR 2 of 3
This PR also includes the cross-type namespace validation fixes from PR 1 (needed on this branch since it diverges from main before PR 1 merges). #7500

Code Changes

  • src/fides/api/schemas/namespace_meta/google_cloud_sql_postgres_namespace_meta.py - New namespace meta schema with required schema field, optional database_name, and empty fallback secret fields (GCS Postgres defaults to public schema)
  • src/fides/api/service/connectors/query_configs/google_cloud_postgres_query_config.py - Added namespace_meta_schema, generate_table_name(), get_formatted_query_string(), and get_update_stmt() to generate schema-qualified SQL
  • src/fides/api/service/connectors/google_cloud_postgres_connector.py - Added query_config() override to fetch namespace meta from DB, get_qualified_table_name() for table existence checks, clarifying docstring on set_schema
  • src/fides/api/service/connectors/sql_connector.py - Added None guard in get_namespace_meta() for cases where db is None
  • src/fides/service/dataset/validation_steps/namespace_meta.py - Added cross-type namespace skip logic (explicit connection_type check + field overlap check), None secrets guard
  • clients/admin-ui/src/features/integrations/integration-type-info/googleCloudSQLPostgresInfo.tsx - Added "DSR Automation" tag
  • tests/ops/service/connectors/test_google_cloud_postgres_query_config.py - New integration tests: 5 parametrized query generation tests, 1 invalid meta test, 3 update statement tests
  • tests/service/dataset_service/test_namespace_meta_validation.py - Added 5 GCS Postgres validation tests + cross-type skip test, changed unsupported type test to use mariadb

Steps to Confirm

1. Run the namespace meta validation unit tests (no Docker required):

nox -s "pytest(misc-unit)" -- tests/service/dataset_service/test_namespace_meta_validation.py -v

Expected: All 19 tests pass, including the new GCS Postgres tests:

  • test_validate_gcs_postgres_without_namespace_or_schema — PASSED
  • test_validate_gcs_postgres_with_valid_namespace — PASSED
  • test_validate_gcs_postgres_with_connection_defaults — PASSED
  • test_validate_gcs_postgres_with_invalid_namespace — PASSED
  • test_validate_mismatched_namespace_skipped — PASSED

2. Run the query config integration tests (requires Postgres via Docker):

nox -s "pytest(ops-integration)" -- tests/ops/service/connectors/test_google_cloud_postgres_query_config.py -v

Expected: All 10 tests pass:

  • test_generate_query_with_namespace_meta (5 parametrized cases) — queries produce:
    • schema="billing"SELECT ... FROM "billing"."customer" WHERE (email = :email)
    • database_name="prod_db", schema="billing"SELECT ... FROM "prod_db"."billing"."customer" WHERE (email = :email)
    • dict {"schema": "billing"} → same as first case
    • dict with connection_type → same as first case
    • NoneSELECT ... FROM "customer" WHERE (email = :email)
  • test_generate_query_with_invalid_namespace_meta — raises ValidationError
  • test_generate_update_stmtUPDATE "address" SET ...
  • test_generate_namespaced_update_stmtUPDATE "billing"."address" SET ...
  • test_generate_namespaced_update_stmt_with_databaseUPDATE "prod_db"."billing"."address" SET ...

3. Quick local verification (no Docker, no nox):

uv run python -c "
from fides.api.schemas.namespace_meta.namespace_meta import NamespaceMeta
from fides.api.schemas.namespace_meta.google_cloud_sql_postgres_namespace_meta import GoogleCloudSQLPostgresNamespaceMeta
from pydantic import ValidationError

# Auto-registration works
impl = NamespaceMeta.get_implementation('google_cloud_sql_postgres')
assert impl is GoogleCloudSQLPostgresNamespaceMeta
print('PASS: auto-registration')

# Schema-only meta
meta = GoogleCloudSQLPostgresNamespaceMeta(schema='billing')
assert meta.schema == 'billing' and meta.database_name is None
print('PASS: schema-only meta')

# Database + schema meta
meta2 = GoogleCloudSQLPostgresNamespaceMeta(database_name='prod_db', schema='billing')
assert meta2.database_name == 'prod_db' and meta2.schema == 'billing'
print('PASS: database + schema meta')

# Missing schema raises ValidationError
try:
    GoogleCloudSQLPostgresNamespaceMeta(database_name='some_db')
    assert False
except ValidationError:
    print('PASS: missing schema raises ValidationError')

# Fallback fields empty (backward compat — defaults to public)
assert GoogleCloudSQLPostgresNamespaceMeta.get_fallback_secret_fields() == set()
print('PASS: empty fallback fields (backward compat)')
"

Expected output:

PASS: auto-registration
PASS: schema-only meta
PASS: database + schema meta
PASS: missing schema raises ValidationError
PASS: empty fallback fields (backward compat)

Pre-Merge Checklist

  • Issue requirements met
  • All CI pipelines succeeded
  • CHANGELOG.md updated
    • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
    • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
    • Updates unreleased work already in Changelog, no new entry necessary
  • UX feedback:
    • All UX related changes have been reviewed by a designer
    • No UX review needed
  • Followup issues:
    • Followup issues created
    • No followup issues
  • Database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!
    • No migrations
  • Documentation:
    • Documentation complete, PR opened in fidesdocs
    • Documentation issue created in fidesdocs
    • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
    • No documentation updates required

@vercel
Copy link
Contributor

vercel bot commented Feb 26, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
fides-plus-nightly Ready Ready Preview, Comment Feb 26, 2026 8:45pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
fides-privacy-center Ignored Ignored Feb 26, 2026 8:45pm

Request Review

JadeCara pushed a commit that referenced this pull request Feb 26, 2026
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Jade Wibbels and others added 5 commits February 26, 2026 13:19
Add GoogleCloudSQLPostgresNamespaceMeta with database_name (optional)
and schema (required), following the same pattern as Snowflake/BigQuery.

Update GoogleCloudSQLPostgresQueryConfig with generate_table_name()
for schema-qualified SQL, and GoogleCloudSQLPostgresConnector to fetch
namespace_meta from DB and pass it to the query config.

Also includes shared fixes from PR #7500:
- Guard against None db session in get_namespace_meta
- Guard against None secrets in namespace validation
- Skip namespace validation for mismatched connection types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@JadeCara JadeCara force-pushed the ENG-2635-gcp-postgres-namespace-support branch from e0bef11 to b71e285 Compare February 26, 2026 20:20
@JadeCara JadeCara marked this pull request as ready for review February 26, 2026 20:40
@JadeCara JadeCara requested review from a team as code owners February 26, 2026 20:40
@JadeCara JadeCara requested review from lucanovera and thabofletcher and removed request for a team February 26, 2026 20:41
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 26, 2026

Greptile Summary

This PR adds namespace metadata support to the Google Cloud SQL Postgres connector, enabling schema-qualified and optionally database-qualified table names in generated SQL queries. The implementation closely follows the existing Snowflake connector pattern.

Key Changes:

  • New GoogleCloudSQLPostgresNamespaceMeta schema with required schema field and optional database_name
  • Query generation updated to prepend schema/database to table names (e.g., "billing"."customer" or "prod_db"."billing"."customer")
  • Connector overrides query_config() to fetch namespace meta from DB and get_qualified_table_name() for table existence checks
  • Validation enhanced with cross-type namespace skip logic to handle mixed dataset configurations
  • Backward compatibility maintained via empty fallback fields (defaults to public schema)

Test Coverage:

  • 10 integration tests covering query generation, update statements, and validation errors
  • 5 validation tests covering backward compatibility, valid/invalid namespaces, and cross-type scenarios

Observations:

  • The identifier quoting pattern (wrapping in double quotes without escaping internal quotes) follows the existing Snowflake implementation. While identifiers come from admin-controlled configuration rather than user input, consider adding validation to reject identifiers containing double quotes across all namespace meta implementations as a defense-in-depth measure.

Confidence Score: 4/5

  • This PR is safe to merge with minimal risk
  • Score reflects well-tested implementation following established patterns, comprehensive test coverage (15 tests), backward compatibility maintenance, and good defensive coding practices. Minor observation about identifier escaping pattern is pre-existing and not critical since values come from admin configuration rather than user input.
  • No files require special attention

Important Files Changed

Filename Overview
src/fides/api/schemas/namespace_meta/google_cloud_sql_postgres_namespace_meta.py New namespace meta schema defining required schema field and optional database_name for GCS Postgres. Empty fallback fields maintain backward compatibility with public schema default.
src/fides/api/service/connectors/query_configs/google_cloud_postgres_query_config.py Implements namespace-aware query generation with generate_table_name() prepending schema/database to table names. Pattern consistent with existing Snowflake implementation.
src/fides/api/service/connectors/google_cloud_postgres_connector.py Adds query_config() override to fetch namespace meta from DB and get_qualified_table_name() for table existence checks. Clarifying docstring added to set_schema() explaining legacy vs namespace_meta paths.
src/fides/service/dataset/validation_steps/namespace_meta.py Adds cross-type namespace skip logic (checks connection_type field and field overlap) to handle mixed datasets. Also adds None guard for secrets dict.
tests/ops/service/connectors/test_google_cloud_postgres_query_config.py Comprehensive test coverage: 5 parametrized query generation tests, invalid meta validation test, and 3 update statement tests covering non-namespaced, schema-qualified, and database.schema-qualified cases.

Last reviewed commit: 03e297e

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant