Repository navigation
Wire PL_TOOLING_POST_ASQL_TO_FMD to library variables, add GUID guard - #283
Closed
MitchellBitmetric wants to merge 29 commits into
Closed
MitchellBitmetric wants to merge 29 commits into
MitchellBitmetric wants to merge 29 commits into
Conversation
Ongoing CI/CD for src/ (notebooks, pipelines, variable libraries, environment) into INTEGRATION CODE (D)/(P), separate from setup/NB_SETUP_FMD.ipynb which only handles one-time initial provisioning. - .github/workflows/deploy-integration.yml: push to main deploys to development; workflow_dispatch lets you pick development or production. - scripts/deploy_fabric_cicd.py: publishes src/ via fabric-cicd, excluding business_domain/ (belongs to FINANCE/SALES/HR CODE workspaces) and SQLDatabase (belongs to FMD_FRAMEWORK_CONFIGURATION). Auth: existing "Fabric MetadataDriven Framework (FMD)" service principal, now granted Contributor on both INTEGRATION CODE workspaces, with a new client secret added (existing secrets untouched) and stored as repo secrets AZURE_TENANT_ID / AZURE_CLIENT_ID / AZURE_CLIENT_SECRET. Not yet included: unpublish_all_orphan_items (skipped until the exclude regex is proven safe over a few runs) and parameter.yml find/replace (no D/P-specific GUIDs identified yet inside src/ item content). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ration-pilot Add fabric-cicd deployment pipeline for INTEGRATION code workspace
…NTEGRATION pilot - setup/NB_SETUP_FMD.ipynb, setup/NB_SETUP_BUSINESS_DOMAINS.ipynb: add a 'test' entry to the environments/business_domain_deployment config, so a re-run of the bootstrap notebooks also provisions (T) workspaces (same capacity as (D), per the 14 workspaces already created by hand for this pilot: INTEGRATION CODE/DATA, FINANCE/SALES/HR CODE/DATA/REPORTING/SEMANTIC). - VAR_CONFIG_FMD, VAR_FMD, VAR_GOLD_CODE_FMD, VAR_GOLD_SHORTCUTS_FMD VariableLibraries: renamed value sets from Test/Acceptance/Production to Development/Test/Production, matching the D/T/P workspace suffixes. (VAR_GOLD_CODE_FMD's orphaned Prod.json, unreferenced in valueSetsOrder, left untouched.) - deploy-integration.yml / deploy_fabric_cicd.py: added test as a third workflow_dispatch/WORKSPACE_IDS option (INTEGRATION CODE (T)). Not yet done: setting each new (T) workspace's active value set (needs content deployed there first) and rolling the same pattern out to FINANCE/SALES/HR's fabric-cicd pipelines. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add TST environment (workspaces, config, variable libraries)
First real run failed with two issues, now fixed: 1. Auth: the client secret GitHub secret had picked up a trailing newline from how it was set via a PowerShell pipe - re-set with --body, no pipe. 2. folder_path_exclude_regex (used to keep business_domain/ out of this deploy) needs enable_experimental_features + enable_exclude_folder feature flags in fabric-cicd 1.3.0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- VAR_FMD.VariableLibrary/variables.json: lakehouse_schema_enabled had type Boolean but value "" (empty string), which Fabric rejects (InvalidValueOrTypeMismatch). Pre-existing, unrelated to the TST work - first surfaced now because this is the first time VAR_FMD has gone through an automated deploy. Set to "True", matching the framework's documented default. - deploy_fabric_cicd.py: fabric-cicd matches the environment= argument case-sensitively against VariableLibrary value set names. We pass lowercase (development/test/production) but the value sets are capitalized (Development/Test/Production), so it was silently falling back to 'Default value set' instead of activating the right one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Compared against the live item definition in INTEGRATION CODE (D): the
VariableLibrary schema wants a real JSON boolean (true) for type
"Boolean", not a string ("True") like DataPipeline parameters use. My
previous fix used the wrong convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…pool limit Deploy failed: SparkSettingsComputeExceedsPoolLimit - claimed 24 cores/168g (driver 8/56g + max 2 executors 8/56g each), pool limit is 16 cores/112g. Reduced to driver 4/28g + executors 4/28g (worst case 12 cores/84g). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
24 pipelines failed to publish: "User does not have access to the connection used in the Pipeline." Turned out the connection GUIDs committed in pipeline-content.json are stale - CON_FMD_FABRIC_SQL, CON_FMD_FABRIC_PIPELINES and CON_FMD_FABRIC_NOTEBOOKS were recreated in this tenant at some point and nothing updated the git-tracked pipeline files to match. Granted the CI/CD service principal User access on the 3 current connections, and added find_replace entries mapping old ID -> current ID (same value for all three environments - these are tenant-wide shared connections, not per-environment). CON_FMD_ADF_PIPELINES has no live replacement (ADF integration was never set up here) - left unmapped, so ADF-specific pipelines stay broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
22/24 pipelines now publish cleanly. The remaining two fail for distinct, unrelated reasons (see comments in deploy_fabric_cicd.py) - excluding them so a real future regression isn't lost in permanent red-run noise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
INTEGRATION CODE now uses native Fabric Git integration instead (bitmetric-fabric/FMD_INTEGRATION_CODE, dev/test/main branches, its own Deployment Pipeline) - same pattern as every other workspace we've set up. Reasons: fabric-cicd only pushed src/ from this framework fork, leaving customer-specific extensions (e.g. NB_RDW_VOERTUIGEN, invoked same-workspace by NB_FMD_PROCESSING_LANDINGZONE_MAIN's notebookutils.notebook.run) completely untracked in any repo. Native git integration tracks the whole workspace - framework items and custom extensions together - fixing that gap without a second deploy path. This repo (the FMD_FRAMEWORK fork, with its upstream remote to edkreuk/FMD_FRAMEWORK) keeps its role as the place to track and pull in upstream framework updates - applying them to FMD_INTEGRATION_CODE is now a deliberate manual step rather than an automatic redeploy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fabric's own schedule trigger cannot carry pipeline parameters (the
job-scheduler API's `parameters` field is explicitly unsupported for
Data Pipelines), so one pipeline can't have two schedules that load two
different sets of sources. LoadGroup moves that decision into metadata
instead: a `LoadGroup` column on `integration.DataSource`, threaded
through the views, sp_GetBronzelayerEntity/sp_GetSilverlayerEntity, and
every pipeline in the load chain (PL_FMD_LOAD_ALL down to every
PL_FMD_LDZ_COMMAND_*/PL_FMD_LDZ_COPY_FROM_* leaf).
Default is '' ("no filter") everywhere, so untagged sources keep loading
on every run and existing deployments need no metadata migration. A
non-empty LoadGroup is an exact match, letting a thin per-schedule
wrapper pipeline (documented in wiki/LoadGroup_Scheduling.md) invoke
PL_FMD_LOAD_ALL with its own LoadGroup literal and its own Fabric
schedule — e.g. one wrapper nightly, another every two hours.
Note: SQL_FMD_FRAMEWORK.dacpac still needs rebuilding from these
sources (dacpac-guard will fail until then) — no .NET SDK was available
in this environment to do it here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PL_FMD_SCHEDULE_NIGHTLY and PL_FMD_SCHEDULE_INTRADAY are the concrete
example from wiki/LoadGroup_Scheduling.md: a single InvokePipeline
activity calling PL_FMD_LOAD_ALL with LoadGroup hard-coded to a literal
("NIGHTLY"/"INTRADAY"), each schedulable independently in the Fabric
portal. Registered in config/item_deployment.json right after
PL_FMD_LOAD_ALL's own entry, so the setup notebook's mapping_table
already knows PL_FMD_LOAD_ALL's real object ID by the time it rewrites
these wrappers' placeholder pipelineId reference (deploy_item /
replace_ids_and_mark_inactive in NB_UTILITIES_SETUP_FMD) — the same
mechanism every other pipeline in this framework relies on, and the
reason a new pipeline never needs a real, live-tenant object ID up
front.
Also rebuilt SQL_FMD_FRAMEWORK.dacpac from src/Config_Database (a .NET
8 SDK install was needed in this environment first) so dacpac-guard
passes on this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous rebuild was done from a working tree checked out with core.autocrlf=true, which rewrites the committed LF line endings to CRLF on disk. dacpac model.xml embeds each object's literal source header text (including line endings) in a SysCommentsObjectAnnotation, so a Windows build with autocrlf-converted sources produces a byte-different (but semantically identical) dacpac from the Linux CI runner's LF checkout — which is exactly what made dacpac-guard fail on this branch. Rebuilt from a clone forced to core.autocrlf=false so the sources on disk match what CI actually checks out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DacFx embeds the platform's own line-ending convention (Environment.NewLine) into the model's SysCommentsObjectAnnotation header text, independent of the source .sql files' own line endings. That makes a locally-built (Windows) dacpac byte-different from dacpac-guard's Linux build no matter how the sources are checked out, so building this locally was never going to reproduce a matching artifact. Grabbed the actual dacpac dacpac-guard built on this branch (via a temporary upload-artifact step, now reverted) and committed that instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d templates, MLV refresh Found by inventorying live Fabric config (INTEGRATION + FINANCE) against this fork's src/ and config/: several things bitmetric built directly in the Fabric UI never made it back into version control. This formalizes the generic, reusable parts so a new customer deployment gets them out of the box, instead of being clicked together by hand. - NB_UTILITIES_SETUP_FMD: scripts native Fabric Deployment Pipelines (Dev/Test/Prod stages assigned to the matching workspaces) per workspace-group. Wired into NB_SETUP_FMD (INTEGRATION CODE/DATA) and NB_SETUP_BUSINESS_DOMAINS (CODE/DATA/ REPORTING/SEMANTIC per domain). Previously a manual, per-customer click-through. - PL_FMD_ORCHESTRATION_TEMPLATE: generic top-of-day orchestrator (PL_FMD_LOAD_ALL, then one InvokePipeline slot per business domain Gold pipeline). Generalizes the live-only PL_MASTER_ORCHESTRATION pattern. - business_domain/PL_FMD_LOAD_GOLD_TEMPLATE: generic Gold-layer load pipeline (NB_LOAD_GOLD -> NB_MLV_EXAMPLE). Generalizes the live-only PL_FINANCE_LOAD_GOLD, stripped of customer-specific notebook/semantic-model references. - business_domain/NB_MLV_EXAMPLE: backported the RefreshMaterializedLakeViews API call that was only in the live, customer-specific NB_MLV_ADVENTUREWORKS notebook. Not upstreamed (customer/env-specific, stays out of the fork): AdventureWorks/RDW demo content, this org's Spark capacity tuning, the retired fabric-cicd pilot. Untested against a live tenant run — smoke-test before merging. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…kreuk NB_SETUP_FMD and NB_SETUP_BUSINESS_DOMAINS pull framework source straight from GitHub at run time. They were still pointed at edkreuk/FMD_FRAMEWORK@main, so a setup run pulled vanilla upstream and none of bitmetric's own additions (LoadGroup, Purview, and everything in this PR). Point both at bitmetric-fabric/FMD_FRAMEWORK@main. Also fixes a pre-existing bug in NB_SETUP_BUSINESS_DOMAINS: `branch = "main" is default` is not valid Python (NameError: name 'default' is not defined) — this notebook could never have been run successfully as committed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…to feature/upstream-fork-candidates # Conflicts: # config/item_deployment.json
NB_SETUP_BUSINESS_DOMAINS previously left the orchestrator with only its PL_FMD_LOAD_ALL activity — the per-domain InvokePipeline slot from a16a076 had to be added by hand for every business domain and stage. Adds wire_domain_into_orchestration() to NB_UTILITIES_SETUP_FMD: reads the orchestrator's pipeline-content.json via the Fabric REST API, appends an InvokePipeline activity targeting <domain> CODE (<stage)'s PL_FMD_LOAD_GOLD_TEMPLATE if one isn't already present, and writes it back. Idempotent by activity name, so reruns are safe. Wired into NB_SETUP_BUSINESS_DOMAINS right after deployment pipeline creation, looping business_domain_names x business_domain_deployment (D/T/P). Found and validated on the FMD trial capacity while wiring FMD_TRIAL_FINANCE into PL_FMD_ORCHESTRATION_TEMPLATE by hand first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…utility - Add set_variable_library_values() to NB_UTILITIES_SETUP_FMD: patches Variable Library default values via REST API getDefinition/updateDefinition - Add auto-fill cell in NB_SETUP_BUSINESS_DOMAINS: resolves Gold/Silver lakehouse IDs per business domain x stage and writes them into VAR_GOLD_SHORTCUTS_FMD, eliminating manual ID entry - Remove redundant variable_parameters.update cell (superseded by per-domain auto-fill via set_variable_library_values) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The setup notebooks configured themselves from values typed into cells: repo_owner buried at line 274 of an 1900-line notebook, capacity names, domain names, group IDs, Spark sizing and the stage list. A customer implementation had to find and edit all of them, in two notebooks, and the generic layer carried our fork's values (triage V1-V4, ADR-003). Everything that is fixed at delivery now lives in manifest.yaml in the root of the customer's fork. The notebooks fetch it and fail with a readable message naming every missing key at once, rather than falling back to a default. The bootstrap problem - something has to say where the fork is before anything can be downloaded - is solved with NB_BOOTSTRAP_FMD: the one file placed by hand. It fetches both setup notebooks, stamps the three repository values into their bootstrap cell, and creates them in the workspace. The setup notebooks verify those values against repository: in the manifest and stop on a mismatch, so src/ and config/ can never come from a different repo than the manifest. Applied at deploy time, because they differ per environment while src/ holds one copy of each item: - Sparkcompute.yml is rewritten from spark.default with spark.overrides.<env> on top. Line-based, so keys the manifest does not mention (enable_native_execution_engine, dynamic_executor_allocation), comments and CRLF line endings survive untouched. - Variable Libraries ship four value sets (Development/Test/Acceptance/ Production) and are narrowed to the ones the manifest declares active, so a DTP customer does not get an empty Acceptance set and a DTAP customer needs no framework change. Role lists are built from security.*_group_id and skip IDs that are still all zeros. assign_workspace_roles never filtered those; it called fab acl set with the zero GUID and swallowed the failure via silently_continue, so the manifest's promise that unset IDs are skipped did not exist yet. Also removed a stray Prod.json value set that was in neither valueSetsOrder nor any environment, and replaced the ' CONFIG' string surgery that derived the integration prefix - it returned 'INTEGRATION FMD' as soon as framework_post_fix was set. scripts/test_manifest_wiring.py executes the manifest and configuration cells of both notebooks against manifest.example.yaml and asserts the result matches the configuration from before this change, then exercises both deploy-time hooks on copies of the real files. 19 comparisons, no network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ENV_FMD.Environment is deployed once, into the CONFIG workspace, outside any environment loop (NB_SETUP_FMD cell 50) — one item and one Sparkcompute.yml shared by Development, Test and Production. A sizing per environment cannot be expressed against that. The previous commit shipped spark.overrides.<environment> anyway. The call site passed environment['environment_name'], a binding left over from an earlier loop, so it resolved to whichever environment came last in the manifest. Setting spark.overrides.Production applied those values to the item Dev and Test use as well, and spark.overrides.Development never applied at all. It did not fail; it quietly did something other than what the manifest said. So: spark.overrides is out of the manifest and out of apply_spark_settings, and the deploy call now passes 'config' explicitly, matching how the CONFIG workspace itself is deployed. Sizing per environment needs an Environment item per environment first. That is a separate change, and it also touches an existing mismatch: every CODE workspace gets sparkSettings.environment.name = ENV_FMD while the item only exists in CONFIG. Recorded in docs/triage-besluiten.md, which decided V2 the other way before this was known. config_workspace.capacity stays. The CONFIG workspace is deployed once with environment_name='config' and no D/T/P suffix, carries the shared metadata database and this Environment item, and upstream already kept its capacity as a separate value. It has no per-environment counterpart to inherit from. The check now asserts that every line the manifest does not name survives apply_spark_settings byte for byte, instead of exercising override precedence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D3. The setup notebooks had capacity_name_business_domain_dvlm/_prod alongside capacity_name_dvlm/_prod. Taking the example manifest literally collapsed those into one capacity per environment, which removed an existing capability and wrote it off in a doc note instead of asking. environments[].capacity_business_domain brings it back, optional: left out, business domains share the integration layer's capacity exactly as they did before this branch. D4. branch was dev in the manifest and in both bootstrap cells, but origin has no dev branch, so the very first manifest fetch would 404. Everything points at main now. D5. viewer_group_id stays. It is wired to nothing today — the framework has no viewer role — so it is marked as reserved in the manifest, to keep anyone from filling it in and expecting access. The check covers both directions of the fallback: absent capacity_business_domain yields the integration capacity, present yields its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
D6. config/item_deployment_code_business_domain.json declares six variables for VAR_GOLD_SHORTCUTS_FMD, and update_variable_library looks up every declared source in variable_parameters. The cell that supplied those six was removed when the auto-fill cell was added, so deploying that library raised KeyError. deploy_item catches it and prints a cross, so the library was never created and the auto-fill afterwards found nothing to write into. Gold shortcuts have been silently absent since. The earlier plan - drop the four ID fields from the deployment config - would not have worked. update_variable_library assigns content["variables"] wholesale, so undeclared variables are stripped at deploy time, and set_variable_library_values only updates variables that already exist. The four IDs would have disappeared and the auto-fill would have gone on writing nothing. All six stay declared. The two schema names are naming decisions fixed at delivery, so they come from manifest.shortcuts. The four IDs are empty at deploy time by design: they are only knowable once the workspaces and lakehouses exist, and the auto-fill cell resolves and writes them afterwards. That leaves the shortcut settings cell with the two values that are actually an input; the four it used to carry were dead. update_variable_library now names the missing source and says where it should have been set, instead of raising a bare KeyError with one word in it. It also rejects an unknown variable type rather than silently reusing the previous iteration's value, which would have written one variable's value into another. The check covers both directions: without the restored cell the deploy still fails and the message names SourceWorkspaceId, with it the library keeps all six variables, the schemas come from the manifest and the IDs stay empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Places scripts/guid_guard.py, .github/workflows/guid-guard.yml and config/guid_allowlist.txt as delivered, unmodified. Allowlisted: 00000000-0000-0000-0000-000000000001, the sentinel row in integration.Connection meaning "no external connection", written by sp_UpsertConnection from NB_SETUP_FMD. It names no Fabric item, so there is nothing to map it to. The guard is red: 140 hits remain, in four groups that each need a decision rather than a fix. See the PR description. Nothing was resolved by loosening the scan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
142 hits down to 5. A. Jupyter cell ids (131 hits, all in the two setup notebooks). nbformat gives every cell a GUID-shaped id to tell cells apart. It is not a reference and deploy_item() does not replace it — registering one in the mapping table would make the deployment rewrite it and corrupt the notebook. Allowlisting them is no better: a new one appears whenever someone adds a cell. So the scan now parses an .ipynb, collects the ids its own cells declare, and skips exactly those GUIDs in exactly that file. Everything else in the notebook, cell contents included, is still scanned. B. Four items whose config id did not match their .platform logicalId. For 45 of the 50 items they are equal, so the convention is real and these were wrong: NB_LOAD_GOLD c884e094 -> 8b6fabb1 NB_CREATE_DIMDATE 22909f1c -> c884e094 ENV_FMD 4c613d65 -> 773793cd NB_FMD_LOAD_DEMO_DATA fa1efb9b -> 3f6ac1d2 The first two were tangled: c884e094 is NB_CREATE_DIMDATE's logicalId but was registered as NB_LOAD_GOLD's id, so the same GUID meant a different item depending on which file you read. PL_FMD_LOAD_GOLD_TEMPLATE used that value as the placeholder for its NB_LOAD_GOLD activity, so it moves with it. Nothing else referenced any of the four old values, so the replacement table stays consistent; deployed items are matched by name, not by old id, so existing deployments are unaffected. Allowlisted: b9ad046b, NB_FMD_CUSTOM_DQ_CLEANSING's logicalId. That notebook is in src/ but in no item_deployment*.json, so it is never deployed and there is nothing to map it to. Its sibling NB_FMD_CUSTOM_NOTEBOOK_TEMPLATE is registered, so this may be an omission — but registering it would roll a new item out to every customer, which is a decision rather than a repair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mitchell's uncommitted work, parked on its own branch so it is not lost and does not ride along into main. Two separate things, neither finished: 1. PL_TOOLING_POST_ASQL_TO_FMD reads its workspace and lakehouse from VAR_CONFIG_FMD instead of hardcoded GUIDs, and variables.json gains fmd_data_workspace_guid and fmd_landingzone_lakehouse_guid. Not functional yet: config/item_deployment.json declares only four variables for VAR_CONFIG_FMD, and update_variable_library assigns content["variables"] wholesale from that declaration. The two new variables are therefore stripped at deploy time, and the pipeline then references library variables that do not exist. Finishing it needs the two declared there plus a source that resolves them during setup, the way fmd_config_workspace_guid is set from deploy_item. 2. The ADF copy activity points at d18305e6 instead of 02e107b8, with a description saying the Bitmetric tenant has no Azure Data Factory. Worth a second look: 02e107b8 is CON_FMD_ADF_PIPELINES in config/item_config.yaml, so it was a placeholder that deployment remapped to the connection the setup creates. d18305e6 is in no config file, so it is left as-is and stays pointing at whatever it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Declares fmd_data_workspace_guid and fmd_landingzone_lakehouse_guid for VAR_CONFIG_FMD in item_deployment.json, so update_variable_library() no longer strips them at deploy time. NB_SETUP_FMD.ipynb now resolves both per environment, right before that environment's VariableLibrary deploy, since the data workspace and its lakehouse differ per environment while VAR_CONFIG_FMD is deployed once per environment's code workspace. Also notes, with a ponytail comment, that create_or_get_fmd_connection's AzureDataFactory branch silently swallows a failed connection lookup and can corrupt a mapping-table entry -- left as a known gap since no tenant here has ADF to exercise it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
Opened against the wrong repo by mistake (should target bitmetric-fabric/FMD_FRAMEWORK, not upstream). Closing and reopening correctly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/,setup/orwiki/isn't registered fordeploy_item()to remap during deployment..platformlogicalId, and teaches the guard to skip Jupyter notebook cell ids.PL_TOOLING_POST_ASQL_TO_FMDto read its data-workspace and landingzone-lakehouse GUIDs fromVAR_CONFIG_FMDlibrary variables instead of hardcoded/parameter defaults: declaresfmd_data_workspace_guidandfmd_landingzone_lakehouse_guidinconfig/item_deployment.json, and resolves them per environment insetup/NB_SETUP_FMD.ipynbbefore that environment's Variable Library deploy (they differ per environment, unlike the config-workspace variables which are set once).ponytail:comment) increate_or_get_fmd_connection'sAzureDataFactorybranch: a missing manual connection silently propagates a bad value into the mapping table. Left as-is since no tenant here has ADF to exercise it; the existing placeholder GUID onPL_FMD_LDZ_COPY_FROM_ADFavoids the resulting deploy-import failure and is documented in its own description.Test plan
python scripts/guid_guard.py— 5 pre-existing hits, none introduced by this branch (verified againstmainbaseline)get_workspace_id_by_name/get_item_idfor the data workspace and landingzone lakehouse against a live trial tenant via a Livy session — both matched the real IDsupdate_variable_library()against the liveVAR_CONFIG_FMDvariable library (read-modify-write via the Fabric REST API): existing four variables preserved, two new ones added with correct values, read back to confirm