Skip to content

gcs_to_bq: Support schema in different bucket  #12520

Closed

Description

@maroshmka

Description

Schema path (schema_object) for table in GCSToBigQueryOperator is downloading schema from source bucket.

New feature should support support also pulling schemas from different bucket.

Use case / motivation

I guess there are lot of use-cases. To mention one concrete:

  • we want to have schemas at 1 place, for us its terraform, where we define all resources and ACLs
  • those schemas are exported to GCS by CI
  • we want to use them in loads to BQ tables

In case there is a schema change, we make 1 change (in terraform) and it works. Otherwise we would need to do terraform change, apply, then either update schema_fields in operator and release it or to allow adding fields on table load (which we don't necessarily want all the time).

Related Issues

Not directly, but at least - #8280 - polish system tests to operator while at it

Activity

  1. maroshmka commented on Nov 20, 2020

    @maroshmka
    ContributorAuthor

    Implemented draft, if its ok to have this feature, I can finish the system tests :)

  2. locked and limited conversation to collaborators on Sep 7, 2022
  3. converted this issue into a discussion #26220 on Sep 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions