fix(ingest): allow full URNs and global tags for tags_to_ignore_sampling#18099
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Connector Tests ResultsAll connector tests passed for commit To skip connector tests, add the Autogenerated by the connector-tests CI pipeline. |
There was a problem hiding this comment.
Nice fix — reading both schemaMetadata (source-ingested tags) and editableSchemaMetadata closes a real gap, and the set-based collection also removes the old double-count when a column carried two matching tags. A few suggestions inline; none of them block the correctness of the fix.
|
@treff7es I forgot that I needed to change the SQL alchemy profiler as well. After the latest changes, this worked in the actual instance with tags ingested from Snowflake. |
There are two issues with
tags_to_ignore_samplingThe first is that if your tag is urn:li:tag:my_tag then you have to put my_tag in the recipe. I changed this so using the URN prefix will work as well. The documentation is also updated to make this clear.
The larger fix is that
tags_to_ignore_samplingonly used editableSchemaMetadata which does not account for tags that were ingested via a source. Tags for those columns are stored in the schemaMetadata.