You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gcs_to_bq: Support schema in different bucket #12520
Schema path (schema_object) for table in GCSToBigQueryOperator is downloading schema from source bucket.
New feature should support support also pulling schemas from different bucket.
Use case / motivation
I guess there are lot of use-cases. To mention one concrete:
we want to have schemas at 1 place, for us its terraform, where we define all resources and ACLs
those schemas are exported to GCS by CI
we want to use them in loads to BQ tables
In case there is a schema change, we make 1 change (in terraform) and it works. Otherwise we would need to do terraform change, apply, then either update schema_fields in operator and release it or to allow adding fields on table load (which we don't necessarily want all the time).
Related Issues
Not directly, but at least - #8280 - polish system tests to operator while at it
Activity
maroshmka commentedon Nov 20, 2020
Implemented draft, if its ok to have this feature, I can finish the system tests :)