diff --git a/.github/workflows/guid-guard.yml b/.github/workflows/guid-guard.yml new file mode 100644 index 00000000..534cb2fb --- /dev/null +++ b/.github/workflows/guid-guard.yml @@ -0,0 +1,27 @@ +name: guid-guard + +on: + pull_request: + paths: + - 'src/**' + - 'setup/**' + - 'wiki/**' + - 'config/**' + - 'scripts/guid_guard.py' + - '.github/workflows/guid-guard.yml' + push: + branches: [main, dev] + workflow_dispatch: + +jobs: + guid-guard: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Controleer op niet-gemapte GUID's + run: python scripts/guid_guard.py diff --git a/.gitignore b/.gitignore index 00c22a1a..1ce582ca 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,11 @@ Thumbs.db # OMC (oh-my-claudecode) .omc/ + +# FMD klant-manifest +# Het ingevulde manifest hoort niet in dit template-repo: het bevat de namen, +# capaciteiten en Entra-groep-ID's van een specifieke klant. +# LET OP: in een KLANT-fork moet manifest.yaml juist wel gecommit worden - de +# setup-notebooks halen hem daar via raw.githubusercontent.com op. Haal deze +# regel daar dus weg. manifest.example.yaml blijft altijd in git. +manifest.yaml diff --git a/FMD_BUSINESS_DOMAIN_DEPLOYMENT.md b/FMD_BUSINESS_DOMAIN_DEPLOYMENT.md index 9cf5e276..8a50b28d 100644 --- a/FMD_BUSINESS_DOMAIN_DEPLOYMENT.md +++ b/FMD_BUSINESS_DOMAIN_DEPLOYMENT.md @@ -69,16 +69,19 @@ Make sure the following settings are enabled: ### 1. Download deployment assets -Download the deployment notebook from the setup folder to your local machine: +If you already deployed the integration framework, `NB_SETUP_BUSINESS_DOMAINS` +is already in your configuration workspace: the bootstrap notebook places both +setup notebooks at once. In that case skip to step 3. -- `NB_SETUP_BUSINESS_DOMAINS.ipynb` – Automates artifact creation for FMD_FRAMEWORK in Fabric, based on your configuration. +Otherwise download a single file from the `setup` folder: +- `NB_BOOTSTRAP_FMD.ipynb` – fetches the setup notebooks from your fork and places them in the workspace. ### 2. Create a configuration workspace - Create a new workspace (for example, `FMD_FRAMEWORK_CONFIGURATION`). -- Import the deployment notebook into the workspace (ensure you are in the Fabric Experience): - - `NB_SETUP_BUSINESS_DOMAINS.ipynb` +- Import the bootstrap notebook into the workspace (ensure you are in the Fabric Experience): + - `NB_BOOTSTRAP_FMD.ipynb` > [!NOTE] > Make sure you set Spark session timeout to at least 1 hour in the workspace settings/Data Engineering/Jobs. @@ -87,138 +90,27 @@ Download the deployment notebook from the setup folder to your local machine: ### 3. Configure deployment settings -Open `NB_SETUP_BUSINESS_DOMAINS.ipynb` and navigate to the configuration cell. Update the following parameters as needed. +All per-customer configuration lives in **`manifest.yaml`** in the root of your +fork — the same file the integration framework uses. Nothing is configured by +editing notebook cells any more. -#### Key configuration parameters +The keys this notebook needs on top of the integration ones: -**Framework settings** +| Key | What it holds | +|---|---| +| `naming.business_domains` | The business domains to create, for example `[FINANCE, SALES]`. Each gets a CODE, DATA, REPORTING and SEMANTIC workspace per environment. | +| `naming.domain_name` | Used to find the integration domain's CONFIG workspace and metadata database, so they are not configured separately here. | +| `environments[].capacity_business_domain` | Optional. The capacity the business-domain workspaces land on, per environment. Left out, they share `environments[].capacity` with the integration layer. | +| `security` | Object IDs of the Entra groups that get access to the business-domain workspaces. | -> [!NOTE] -> Fabric Administrator Role is required to create a domain. Otherwise, disable domain creation in the next step. - - -**settings** - -```python -assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons - -lakehouse_schema_enabled = True # Set to True if you want to use the lakehouse schema, otherwise set to False - -driver = '{ODBC Driver 18 for SQL Server}'# Change this if you use a different driver -overwrite_variable_library = True # By default the Library is overwritten, change this to "False" if you have custom changes -``` - -**Variable settings** - -```python -SourceWorkspaceId='' -SourceLakehouseId='' # Your Gold lakehouse -SourceSchema='' -Shortcut_TargetSchema='' -Shortcut_TargetWorkspaceId='' -Shortcut_TargetLakehouseId='' # Your Silver Lakehouse -``` - -**Capacity settings** - -Specify the unique name for the capacity: - -```python -capacity_name_dvlm = 'Name of your capacity' -reassign_capacity= True # Set to False if you don't want to reassign the capacity to an existing workspace in case you set the capacity manually -``` - -**Domain settings** - -Define the name for the Main Domain, and you can add 1 or more business domains. - -```python -framework_post_fix= '' # postfix to be added at the end of workspace for example INTEGRATION CODE(D) FMD -if framework_post_fix != '': - framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility - -##Domains -create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed -business_domain_names= ['FINANCE','SALES'] # Define business domains -domain_contributor_role = {"type": "Contributors","principals": [{"id": "00000000-0000-0000-0000-000000000000","type": "Group"} ]} # Which group(Object ID) can add or remove workspaces to this domain -configuration_database_workspace= 'Name of the workspace of your SQL Server database which holds the configuration database' -configuration_database_name= 'Name of the configuration database' - -``` - -You need to create workspace roles for the different workspaces: +See `manifest.example.yaml` for the full structure, and +[FMD_FRAMEWORK_DEPLOYMENT.md](FMD_FRAMEWORK_DEPLOYMENT.md#4-configure-deployment-settings) +for how to fill in and commit the manifest. > [!NOTE] -> The id of the User, Group or Service Principal is the Object ID in Microsoft Entra ID. For a Service Principal, you can find the Object ID in the Azure Portal under 'Enterprise applications'. Don't use the Object ID of the App Registration.' - -workspace_roles_reporting -workspace_roles_gold - -Check the examples below - -```python - -# Replace placeholder IDs with real Object IDs from Microsoft Entra ID. -# Use "Group", "User" or "ServicePrincipal" for "type" as appropriate. - -workspace_roles_code = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Member" - }, - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "ServicePrincipal"}, - "role": "Contributor" - } -] - -workspace_roles_data = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Member" - }, - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Admin" - } -] - -workspace_roles_configuration = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Contributor" - } -] - - - -# Optional: business-domain specific role lists -workspace_roles_data_business_domain = [ - {"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"} -] - -workspace_roles_code_business_domain = [ - {"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"} -] - -workspace_roles_reporting_business_domain = [ - {"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"} -] -``` - -**Repo Configuration** - -Location of the FMD Framework repository. Unless you have a forked version, do not change these settings. If you want to use another branch, you can change the branch name to your own branch. - -```python -#FMD Framework code -##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE #### -repo_owner = "edkreuk" # Owner of the repository -repo_name = "FMD_FRAMEWORK" # Name of the repository -branch = "main" #"main" is default -folder_prefix = "" -################################################### -``` +> Business-domain workspaces share `environments[].capacity` with the integration +> workspaces unless you set `capacity_business_domain` on that environment. Either +> way it is a manifest change, not a notebook change. ### 4. Run the deployment diff --git a/FMD_FRAMEWORK_DEPLOYMENT.md b/FMD_FRAMEWORK_DEPLOYMENT.md index ffe5ec59..8cb4044e 100644 --- a/FMD_FRAMEWORK_DEPLOYMENT.md +++ b/FMD_FRAMEWORK_DEPLOYMENT.md @@ -49,9 +49,13 @@ Select the switch for the type of admin APIs you want to enable: ### 1. Download deployment assets -Download the deployment notebook from the setup folder to your local machine: +Download a single file from the `setup` folder to your local machine: -- `NB_SETUP_FMD.ipynb` – Automates artifact creation for FMD_FRAMEWORK in Fabric, based on your configuration. +- `NB_BOOTSTRAP_FMD.ipynb` – fetches the setup notebooks from your fork and places them in the workspace. + +This is the only file you import by hand. `NB_SETUP_FMD.ipynb` and +`NB_SETUP_BUSINESS_DOMAINS.ipynb` are placed by the bootstrap notebook, with the +location of your fork already filled in. ### 2. Create required connections @@ -81,8 +85,8 @@ If you use Azure Data Factory Pipelines, create this additional connection: ### 3. Create a configuration workspace - Create a new workspace (for example, `FMD_FRAMEWORK_CONFIGURATION`). -- Import the deployment notebook into the workspace (ensure you are in the Fabric Experience): - - `NB_SETUP_FMD.ipynb` +- Import the bootstrap notebook into the workspace (ensure you are in the Fabric Experience): + - `NB_BOOTSTRAP_FMD.ipynb` > [!NOTE] > Make sure you set Set Spark session timeout to at least 1 hour in the workspace settings/Data Engineering/Jobs . @@ -90,172 +94,64 @@ If you use Azure Data Factory Pipelines, create this additional connection: ### 4. Configure deployment settings -Open `NB_SETUP_FMD.ipynb` and navigate to the configuration cell. Update the following parameters as needed. - -#### Key configuration parameters - -**Framework settings** - -> [!NOTE] -> Fabric Administrator Role is required to create a domain. Otherwise, disable domain creation in the next step. - - -**Framework settings** - -```python -assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons -load_demo_data = True # Set to True if you want to load the demo data, otherwise set to False -lakehouse_schema_enabled = True # Set to True if you want to use the lakehouse schema, otherwise set to False - -driver = '{ODBC Driver 18 for SQL Server}'# Change this if you use a different driver -overwrite_variable_library = True # By default the Library is overwritten, change this to "False" if you have custom changes -``` -**Keyvault settings** -For future use. -```python -key_vault_uri_name='val_key_vault_uri_name' -key_vault_tenant_id='val_key_vault_tenant_id' -key_vault_client_id='val_key_vault_client_id' -key_vault_client_secret='val_key_vault_client_secret' -``` +All per-customer configuration lives in **`manifest.yaml`** in the root of your +fork. Nothing is configured by editing notebook cells any more: the setup +notebooks read the manifest and fail with a readable message when a required key +is missing. -**Capacity settings** - Specify the unique name for the capacity: +#### 4a. Fill in the manifest - ```python - capacity_name_dvlm = 'Name of your capacity' - reassign_capacity= True # Set to False if you don't want to reassign the capacity to an existing workspace in case you set the capacity manually - - ``` +Copy `manifest.example.yaml` to `manifest.yaml`, fill it in, and commit it to +your fork. The setup notebooks fetch it over HTTPS from +`raw.githubusercontent.com`, so it must be committed and the repository must be +readable. -**Domain settings** - -Define the name for the Main Domain, and you can add 1 or more business domains. - - -```python -framework_post_fix= '' # post fix to be added at the end of workspace for example INTEGRATION CODE(D) FMD -if framework_post_fix != '': - framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility - -##Domains -create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed -domain_name='INTEGRATION' # Main Domain for Integration for example INTEGRATION CODE(D) - -domain_contributor_role = {"type": "Contributors","principals": [{"id": "00000000-0000-0000-0000-000000000000","type": "Group"} ]} # Which group(Object ID) can add or remove workspaces to this domain - -##Connections -connection_fabric_datapipelines_name='CON_FMD_FABRIC_PIPELINES' -connection_fabric_notebooks_name='CON_FMD_FABRIC_NOTEBOOKS' -connection_fabric_database_name='CON_FMD_FABRIC_SQL' -connection_fabric_adf_name='CON_FMD_ADF_PIPELINES' -connection_role = {"role": "owner","principals": [{"id": "00000000-0000-0000-0000-000000000000","type": "Group"} ]} # Which group(Object ID) can add or remove workspaces to this domain -``` -You need to create workspace roles for the different workspaces: +> [!IMPORTANT] +> This template repository lists `manifest.yaml` in `.gitignore`, so the template +> never carries a specific customer's values. In **your** fork, remove that line — +> the deployment depends on the file being in git. `manifest.example.yaml` stays +> in git in both. + +| Section | What it holds | +|---|---| +| `repository` | Owner, name and branch of your fork. Must match the three values in the bootstrap notebook. | +| `naming` | `domain_name` (the shared ingestion domain), `framework_post_fix`, and the list of business domains. | +| `environments` | One entry per environment: `name`, `short` (D/T/A/P), `capacity`, and optionally `capacity_business_domain` for the business-domain workspaces. The order is the stage order of the deployment pipelines, and it decides which value sets are deployed. | +| `config_workspace` | Capacity for the CONFIG workspace, which exists once per customer rather than once per environment. | +| `spark` | Runtime version and Spark sizing, written into the Environment item at deploy time. One `ENV_FMD.Environment` is deployed into the CONFIG workspace and shared by every environment, so this sizing applies everywhere. | +| `framework` | `lakehouse_schema_enabled`. | +| `security` | Object IDs of the Entra groups, and optionally a service principal. | +| `key_vault` | Key Vault name, and the **names** of the secrets holding the service principal credentials. | > [!NOTE] -> The id of the User, Group or Service Principal is the Object ID in Microsoft Entra ID. For a Service Principal, you can find the Object ID in the Azure Portal under 'Enterprise applications'. Don't use the Object ID of the App Registration.' +> The id of a User, Group or Service Principal is the Object ID in Microsoft Entra ID. +> For a Service Principal, find the Object ID in the Azure Portal under 'Enterprise +> applications' — not the Object ID of the App Registration. > [!NOTE] -> If you are deploying on your own and have no groups or service principals to add, set them to an empty list, as the comment in the cell already suggests. The deployer already owns the workspaces it creates, so nothing is lost: -> ```python -> workspace_roles_code = [] -> workspace_roles_data = [] -> ``` -> Leaving an id in place that does not resolve in your tenant makes workspace creation fail. +> If you are deploying on your own and have no groups or service principal to add, +> leave the IDs in `security` on all zeros. Those entries are skipped, and the +> deployer already owns the workspaces it creates, so nothing is lost. An ID that +> does not resolve in your tenant makes the role assignment fail silently. -> [!IMPORTANT] -> **`workspace_roles_configuration` does not exist.** `NB_SETUP_FMD.ipynb` defines only `workspace_roles_code` and `workspace_roles_data`, and the cell that creates the configuration workspace assigns `workspace_roles_data` to it. Setting `workspace_roles_configuration`, as the example below still shows, has no effect on anything. +#### 4b. Point the bootstrap at your fork -workspace_roles_code -workspace_roles_data -workspace_roles_configuration +Open `NB_BOOTSTRAP_FMD.ipynb` and set: -Check the examples below ```python - -# Replace placeholder IDs with real Object IDs from Microsoft Entra ID. -# Use "Group", "User" or "ServicePrincipal" for "type" as appropriate. - -workspace_roles_code = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Member" - }, - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "ServicePrincipal"}, - "role": "Contributor" - } -] - -workspace_roles_data = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Member" - }, - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Admin" - } -] - -workspace_roles_configuration = [ - { - "principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, - "role": "Contributor" - } -] +repo_owner = "your-org" # GitHub organisation or user +repo_name = "FMD_FRAMEWORK" # Repository name +branch = "main" # Branch to deploy from +folder_prefix = "" # Only if src/ and config/ live in a subfolder ``` -**Workspace configuration** -```python -##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE, FE ADDING NEW ENVIRONMENTS #### -# Define settings for each environment (add more environments as needed) -environments = [ - { - 'environment_name': 'development', # Name of target environment - 'workspaces': { - 'data': { - 'name': domain_name + ' DATA (D)' + framework_post_fix, # Name of target data workspace for development - 'roles': workspace_roles_data, # Roles to assign to the workspace - 'capacity_name': capacity_name_dvlm # Name of target data workspace capacity for development - }, - 'code': { - 'name': domain_name + ' CODE (D)' + framework_post_fix, # Name of target code workspace for development - 'roles': workspace_roles_code, # Roles to assign to the workspace - 'capacity_name': capacity_name_dvlm # Name of target code workspace capacity for development - }, - } - }, - { - 'environment_name': 'production', # Name of target environment - 'workspaces': { - 'data': { - 'name': domain_name + ' DATA (P)' + framework_post_fix, # Name of target data workspace for production - 'roles': workspace_roles_data, # Roles to assign to the workspace - 'capacity_name': capacity_name_prod # Name of target data workspace capacity for production - }, - 'code': { - 'name': domain_name + ' CODE (P)' + framework_post_fix, # Name of target code workspace for production - 'roles': workspace_roles_code, # Roles to assign to the workspace - 'capacity_name': capacity_name_prod # Name of target code workspace capacity for production - }, - } - } -] -``` -**Repo Configuration** - -Location of the FMD Framework repository. Unless you have a forked version, do not change these settings. If you want to use another branch, you can change the branch name to your own branch. - ```python -#FMD Framework code -##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE #### -repo_owner = "edkreuk" # Owner of the repository -repo_name = "FMD_FRAMEWORK" # Name of the repository -branch = "main" #"main" is default -folder_prefix = "" -################################################### -``` +These must match `repository:` in the manifest. The setup notebooks verify this +and stop with an error when the two disagree, because otherwise `src/` and +`config/` would come from a different repository than the manifest describes. + +Run the bootstrap notebook. It creates or updates `NB_SETUP_FMD` and +`NB_SETUP_BUSINESS_DOMAINS` in the workspace, with these values stamped into +their bootstrap cell. ### 5. Run the deployment diff --git a/config/guid_allowlist.txt b/config/guid_allowlist.txt new file mode 100644 index 00000000..bdeb54a5 --- /dev/null +++ b/config/guid_allowlist.txt @@ -0,0 +1,15 @@ +# GUID's die bewust in src/, setup/ of wiki/ staan zonder in de vervang-tabel +# (config/item_config.yaml, config/item_deployment*.json) voor te komen. +# +# Formaat: — +# Een regel zonder reden laat de guard falen. Dat is opzet: afwijken moet een +# zichtbare, beargumenteerde handeling zijn. +# +# Zie docs/adr/009-guid-guard.md. +# +# Voorbeeld: +# 11111111-2222-3333-4444-555555555555 — voorbeeld-GUID in wiki/LoadGroup_Scheduling.md, geen echte verwijzing + +00000000-0000-0000-0000-000000000001 — sentinel-connectie: de rij in integration.Connection die 'geen externe connectie' betekent, gezet door sp_UpsertConnection in NB_SETUP_FMD. Geen verwijzing naar een Fabric-item, dus niet te mappen. + +b9ad046b-6fcc-97ef-4b92-e4f1229b91e5 - logicalId van src/NB_FMD_CUSTOM_DQ_CLEANSING.Notebook. Dat notebook staat in src/ maar in geen enkele item_deployment*.json, dus het wordt nooit gedeployed en er is niets om het aan te mappen. Zijn tegenhanger NB_FMD_CUSTOM_NOTEBOOK_TEMPLATE is wel geregistreerd; mogelijk is dit een omissie. Registreren zou een nieuw item bij elke klant uitrollen, dus dat is een besluit, geen reparatie. diff --git a/config/item_deployment.json b/config/item_deployment.json index a62cdb04..567877ae 100644 --- a/config/item_deployment.json +++ b/config/item_deployment.json @@ -41,7 +41,7 @@ }, { "name": "NB_FMD_LOAD_DEMO_DATA.Notebook", - "id": "fa1efb9b-28ed-92ed-4b0e-5a1e22fd8f9e", + "id": "3f6ac1d2-9b4e-4a17-8c05-1d7e2b9a6f34", "type": "Notebook" }, { @@ -169,9 +169,24 @@ "id": "1b6d7d0d-d04b-4bc2-a6da-7a876216561d", "type": "DataPipeline" }, + { + "name": "PL_FMD_SCHEDULE_NIGHTLY.DataPipeline", + "id": "03ba9db4-d5b4-49b5-a37d-c389a3e0a5dd", + "type": "DataPipeline" + }, + { + "name": "PL_FMD_SCHEDULE_INTRADAY.DataPipeline", + "id": "6beb329c-f284-41b9-9d1b-bd292059e075", + "type": "DataPipeline" + }, + { + "name": "PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline", + "id": "14bb3cb5-26dc-4840-bccc-41556d9a22ec", + "type": "DataPipeline" + }, { "name": "ENV_FMD.Environment", - "id": "4c613d65-f566-4bf3-83be-7d8c773793cd", + "id": "773793cd-7d8c-83be-4bf3-f5664c613d65", "type": "Environment" }, { @@ -230,6 +245,18 @@ "type": "variable", "datatype": "String", "source": "fmd_config_workspace_guid" + }, + { + "name": "fmd_data_workspace_guid", + "type": "variable", + "datatype": "String", + "source": "fmd_data_workspace_guid" + }, + { + "name": "fmd_landingzone_lakehouse_guid", + "type": "variable", + "datatype": "String", + "source": "fmd_landingzone_lakehouse_guid" } ] } diff --git a/config/item_deployment_code_business_domain.json b/config/item_deployment_code_business_domain.json index bed9a356..74c218b9 100644 --- a/config/item_deployment_code_business_domain.json +++ b/config/item_deployment_code_business_domain.json @@ -1,7 +1,7 @@ [ { "name": "NB_LOAD_GOLD.Notebook", - "id": "c884e094-2716-83ff-4fb6-bc61f94617c5", + "id": "8b6fabb1-f9a2-af87-4260-23061c74f547", "type": "Notebook" }, { @@ -16,7 +16,7 @@ }, { "name": "NB_CREATE_DIMDATE.Notebook", - "id": "22909f1c-fc6b-ad56-4199-09f452192ddf", + "id": "c884e094-2716-83ff-4fb6-bc61f94617c5", "type": "Notebook" }, { @@ -24,6 +24,11 @@ "id": "54288605-abe0-ad7a-4bb9-249481b85052", "type": "Notebook" }, + { + "name": "PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline", + "id": "838bc064-4221-48e3-ae54-0fdb0ce5a703", + "type": "DataPipeline" + }, { "name": "VAR_GOLD_CODE_FMD.VariableLibrary", "id": "622086c9-2b9b-b2ca-4963-13b0a688d8bb", diff --git a/docs/triage-besluiten.md b/docs/triage-besluiten.md new file mode 100644 index 00000000..9fa31ecc --- /dev/null +++ b/docs/triage-besluiten.md @@ -0,0 +1,77 @@ +# Besluiten n.a.v. de triage van 2026-09-15 + +Antwoorden op de vragen uit `docs/triage.md`. Kleinere keuzes dan een ADR waard is; +de grote zijn doorgezet naar ADR-006 en ADR-009. + +| # | Onderwerp | Besluit | Waar | +|---|---|---|---| +| V1 | `repo_owner` | Naar manifest | PR 3 | +| V2 | Sparkcompute | Naar manifest (override per omgeving teruggedraaid, zie hieronder) | PR 3 | +| V3 | `lakehouse_schema_enabled` | Bug fixen, default `true`, naar manifest | PR 3 | +| V4 | Stages | Vier valueSets meeleveren, manifest bepaalt actieve | PR 3 | +| V5 | MLV-refresh lakehouse-ID | Lezen uit Variable Library | los | +| V6 | Celvolgorde auto-fill | Geverifieerd; onderliggende bug gefixt | PR 3 | +| V7 | GUID-guard | Bouwen | ADR-009, PR 2 | +| V8 | Demo-notebooks | Naar `examples/`, PR naar upstream | PR 1 | + +## Toelichting waar die nodig is + +**V2 — Sparkcompute.** De comment in `Sparkcompute.yml` verwees naar de pool-limiet +van onze eigen Dev/Test-capacity. Dat is een omgevingswaarde in een template. De +waarden gaan naar het manifest, en de comment is herschreven naar "conservatieve +default, verhoog bij grotere capacity". + +> **Bijgesteld tijdens PR 3.** De override per omgeving is er niet gekomen. Bij het +> bedraden bleek `ENV_FMD.Environment` één keer gedeployed te worden, in de +> CONFIG-workspace, gedeeld door Dev/Test/Prod — één item, één `Sparkcompute.yml`. +> Een waarde per omgeving is daar niet in uit te drukken; de eerste implementatie +> paste stilzwijgend de laatste omgeving uit het manifest toe. `spark.overrides` is +> daarom uit het manifest gehaald. Wil je sizing per omgeving, dan moet ENV_FMD +> eerst per omgeving gedeployed worden (los ticket, raakt ook de bestaande +> mismatch: elke CODE-workspace krijgt `sparkSettings.environment.name = ENV_FMD` +> terwijl het item alleen in CONFIG bestaat). + +**V3 — `lakehouse_schema_enabled`.** Upstream had `"value": ""` bij type `Boolean`. +Dat is geen geldige waarde en dus een echte bug — los van de vraag wat de default +moet zijn. Fix gaat ook terug naar upstream. + +**V4 — stages.** Vier value sets meeleveren kost niets, en voorkomt dat een klant +met een echte Acceptance-omgeving een aanpassing aan het framework nodig heeft. +`deploy_deployment_pipeline()` neemt de stages al uit een lijst. + +**V5 — MLV-refresh.** `NB_MLV_EXAMPLE` had `lakehouse_id = ""` +als handmatige invulplek, terwijl `set_variable_library_values()` in dezelfde branch +precies dat ID al automatisch invult in `VAR_GOLD_SHORTCUTS_FMD.SourceLakehouseId`. +Het notebook leest die variabele voortaan via `notebookutils.variableLibrary`. De +waarde van een voorbeeld zit in het patroon dat het laat zien; een handmatige +placeholder leert de verkeerde gewoonte aan. + +**V6 — celvolgorde.** Geverifieerd, en de celvolgorde bleek in orde: cel 42 +deployt alleen de Variable Libraries, cel 44 vult ze, cel 45 deployt de rest. +Geen latere cel her-deployt de library. + +Het probleem zat een laag dieper. `VAR_GOLD_SHORTCUTS_FMD` declareert zes +variabelen in `config/item_deployment_code_business_domain.json`, en +`update_variable_library` zoekt elke bron op in `variable_parameters`. De cel die +die zes bronnen vulde was verwijderd, dus het deployen van die library faalde met +een `KeyError` die `deploy_item` opving en als een rood kruisje afdrukte — de +library werd nooit aangemaakt, en de auto-fill vond daarna niets om te vullen. + +Gefixt in PR 3: de twee schemanamen komen nu uit `manifest.shortcuts`, de vier +ID's krijgen een lege waarde en worden ná deployment door de auto-fill-cel +ingevuld. `update_variable_library` meldt voortaan wélke bron ontbreekt in plaats +van een kale `KeyError`. Een echte setup-run op een schone workspace blijft +wenselijk, maar de fout was statisch reproduceerbaar en is dat nu niet meer. + +## Wat teruggaat naar upstream + +Los van onze eigen mapindeling zijn dit generieke fixes en features: + +- LoadGroup-feature (SQL, views, stored procedures, 26 pipelines) +- Orchestratie-template en Gold-template +- `lakehouse_schema_enabled: ""` → geldige Boolean (V3) +- De drie hardcoded tenant-ID's uit `NB_CREATE_DIMDATE`, `NB_MLV_DEMO_GOLD` en + `Taskflow` (V8) + +Hoe kleiner onze fork-delta, hoe beter we bij een volgende klant nog kunnen zien +wat van ons is. diff --git a/docs/triage.md b/docs/triage.md new file mode 100644 index 00000000..3685e3e1 --- /dev/null +++ b/docs/triage.md @@ -0,0 +1,228 @@ +# Triage: afwijkingen van upstream + +Datum: 2026-09-15 · branch `feature/upstream-fork-candidates` · analyse-only, niets gewijzigd. + +## 0. Let op: de juiste baseline + +`git diff upstream/main` geeft **104 bestanden / 24.098 regels** — dat is misleidend. +`upstream/main` staat precies één commit vóór op ons merge-base: `1ff1684 "Docs1 (#282)"` +(Erwin de Kreuk, 8 sep 2026), die de Docusaurus-site `docs/` + `docsw/` weggooit en +`Images/Branding/fmd-cover.png` terugzet. Die ~24k regels zijn dus **upstream die zijn +eigen docs opruimt**, geen werk van jou. + +De echte fork-delta is `git diff ..HEAD`: + +``` +git merge-base upstream/main HEAD # ebe97d4 "Docs (#281)" +git diff ebe97d4..HEAD --stat # 60 files changed, 2319 insertions(+), 68 deletions(-) +``` + +Alles hieronder gaat over die 60 bestanden. + +## 1. Samenvatting + +| Categorie | Bestanden | Toelichting | +|---|---|---| +| **A** — generieke verbetering | 55 | LoadGroup-feature (SQL + 26 pipelines), deployment-pipeline-helpers, orchestratie-/Gold-templates, MLV-refresh, Development-valueSet | +| **B** — omgevingsconfig gelekt | 1 | `repo_owner = "bitmetric-fabric"` in beide setup-notebooks (1 logisch blok, 2 bestanden) | +| **C** — experiment/restant | 0 | de fabric-cicd-experimenten zijn al netjes teruggedraaid (`0ac7e1a`, `68561b3`) — netto nul t.o.v. upstream | +| **?** — twijfel | 4 | Sparkcompute-sizing, `lakehouse_schema_enabled: true`, `Acceptance`→`Development` rename, verwijderde `variable_parameters.update`-cel | + +Nuance bij "B = 1": de generieke laag is **schoner dan verwacht**. Alle GUID's in de nieuwe +pipeline-JSON zijn de placeholder-GUID's uit `config/item_config.yaml` / +`config/item_deployment.json` die `deploy_item()` bij deployment vervangt (zie §3.1) — +geen trial-capacity-lek. De setup-notebooks staan nog volledig op ``, +`business_domain_names = ['FINANCE','SALES']`, `00000000-…` group-ID's. + +## 2. Tabel per wijziging + +### 2.1 LoadGroup — plannen van bronnen op verschillende cadences + +| Bestand | Regel(s) | Cat. | Bestemming | Toelichting | +|---|---|---|---|---| +| [src/Config_Database/integration/Tables/DataSource.sql](../src/Config_Database/integration/Tables/DataSource.sql#L8) | 8 | A | — | `[LoadGroup] VARCHAR(50) DEFAULT('') NOT NULL`. Generiek, backwards compatible (lege default = huidig gedrag). Kolom is metadata-drager; de *waarden* horen in `integration.DataSource` — dat is precies waar ze staan. | +| [src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql](../src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql) | 12, 37, 45, 56 | A | — | `@LoadGroup VARCHAR(50) = ''` doorgezet naar INSERT en UPDATE. | +| [src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql](../src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql#L11) | 11 | A | — | `DS.[LoadGroup]` toegevoegd aan select-list. | +| [src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql](../src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql#L19-L21) | 19-21 | A | — | idem. | +| [src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql](../src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql#L28-L29) | 28-29 | A | — | idem. | +| [src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql](../src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql#L4) | 4, 30-32 | A | — | `@LoadGroup`-parameter + `(@LoadGroup = '' OR LoadGroup = @LoadGroup)`. Lege string = alles, dus bestaande aanroepen ongewijzigd. | +| [src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql](../src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql#L4) | 4, 39-41 | A | — | idem. | +| `src/PL_FMD_LDZ_COMMAND_*.DataPipeline/pipeline-content.json` (9×: ADF, ADLS, ASQL, FTP, NOTEBOOK, ONELAKE, ORACLE, SFTP, SQLMI) | ~245-250, ~287, ~471-477 | A | — | `LoadGroup`-parameter (defaultValue `""`) + doorgifte aan child-pipeline + `and (''='@{…LoadGroup}' or LoadGroup='@{…LoadGroup}')` in de `sqlReaderQuery`. Uniform patroon over alle 9. | +| `src/PL_FMD_LDZ_COPY_FROM_*.DataPipeline/pipeline-content.json` (10×) | parameterblok | A | — | Alleen parameter-doorgifte, geen logica. | +| [src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json](../src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json#L753-L759) | 11-14, 49-52, 549-552, 756-759 | A | — | `LoadGroup` doorgegeven aan Landingzone/Bronze/Silver + eigen parameter. | +| [src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json](../src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json#L642) | 9× doorgifte, 642 (query), 814-817 | A | — | `SELECT distinct [ConnectionType] … and (''='@{…}' or LoadGroup='@{…}')`. | +| [src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json](../src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json#L244-L250) | 244-250, 523-526 | A | — | `LoadGroup` als `String`-parameter naar `sp_GetBronzelayerEntity`. | +| [src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json](../src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json#L17-L23) | 17-23, 527-530 | A | — | idem voor `sp_GetSilverlayerEntity`. | +| [wiki/LoadGroup_Scheduling.md](../wiki/LoadGroup_Scheduling.md) | hele bestand (132) | A | — | Uitleg van de feature. Bevat geen omgevingswaarden (gecheckt op GUID's, workspace-namen, capacity). | +| [src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac](../src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac) | binair | A | — | Herbouwd uit bovenstaande `.sql`-bronnen. `.github/workflows/dacpac-guard.yml` (upstream) faalt als hij niet matcht, dus dit is een verplichte meeverandering, geen los artefact. | + +### 2.2 Schedule-wrappers + +| Bestand | Regel(s) | Cat. | Bestemming | Toelichting | +|---|---|---|---|---| +| [src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/pipeline-content.json](../src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/pipeline-content.json) | 1-40 | A | — | Wrapper die `PL_FMD_LOAD_ALL` aanroept met `"LoadGroup": "NIGHTLY"`. Hang er een Fabric-schedule aan. | +| [src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/pipeline-content.json](../src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/pipeline-content.json) | 1-40 | A | — | Idem met `"INTRADAY"`. | +| ↳ beide: `.platform` | 1-12 | A | — | `logicalId` 03ba9db4-… / 6beb329c-…, geregistreerd in `config/item_deployment.json`. Framework-conventie, geen omgevingswaarde. | +| ↳ beide: `"defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc"` | 36 | **A** (was mijn B-verdenking) | — | Dít is niet jouw trial-workspace: exact dezelfde GUID staat als `workspaces.workspace_data` in upstream's [config/item_config.yaml:2](../config/item_config.yaml#L2) en in upstream's eigen `PL_FMD_LOAD_ALL` / `PL_FMD_LOAD_LANDINGZONE`. `deploy_item()` vervangt hem per environment (zie §3.1). Gekopieerd volgens de conventie — correct. | +| ↳ beide: `"connection": "6d8146c6-a438-47df-94e2-540c552eb6d7"` | 20 | A | — | = `CON_FMD_FABRIC_PIPELINES` in `item_config.yaml:10`, ook upstream-placeholder. | +| [config/item_deployment.json](../config/item_deployment.json#L172-L186) | 172-186 | A | — | Registratie van de 3 nieuwe pipelines. Vereist, anders worden ze niet gedeployed én niet ge-ID-remapt. | + +### 2.3 Orchestratie- en Gold-templates + +| Bestand | Regel(s) | Cat. | Bestemming | Toelichting | +|---|---|---|---|---| +| [src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/pipeline-content.json](../src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/pipeline-content.json) | 1-28 | A | — | Skelet met één `InvokePipeline` naar `PL_FMD_LOAD_ALL`; domeinen worden er per klant bij gegenereerd (§2.4). | +| ↳ `.platform` description | 7 | A | — | Beschrijft expliciet "copy/rename per customer" — goede template-hygiëne. | +| [src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/pipeline-content.json](../src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/pipeline-content.json) | 1-43 | A | — | `NB_LOAD_GOLD` → `NB_MLV_EXAMPLE`. `notebookId` c884e094-… en 7f3419e4-… komen uit `config/item_deployment_code_business_domain.json`, dus placeholders. | +| [config/item_deployment_code_business_domain.json](../config/item_deployment_code_business_domain.json#L27-L31) | 27-31 | A | — | Registratie van `PL_FMD_LOAD_GOLD_TEMPLATE`. | +| [src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql](../src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql#L37-L66) | 37-66 | A | — | Markdown + pyspark-cel die `RefreshMaterializedLakeViews` POST doet. `workspace_id` komt uit `notebookutils.runtime.context`. **Maar** `lakehouse_id = ""` is handmatig invulwerk — zie vraag V5. | + +### 2.4 Setup-automatisering (`NB_UTILITIES_SETUP_FMD` + setup-notebooks) + +| Bestand | Regel(s) | Cat. | Bestemming | Toelichting | +|---|---|---|---|---| +| [src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py](../src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py#L721-L800) | 721-800 | A | — | `get_deployment_pipeline_id_by_name`, `ensure_deployment_pipeline`, `assign_deployment_pipeline_stage`, `deploy_deployment_pipeline`. Native Fabric ALM scripten i.p.v. per klant klikken. Idempotent. | +| ↳ | 801-855 | A | — | `wire_domain_into_orchestration()` — voegt per business-domein een `InvokePipeline`-activity toe aan `PL_FMD_ORCHESTRATION_TEMPLATE` via getDefinition/updateDefinition. Idempotent op activity-naam. | +| ↳ | 856-890 | A | — | `set_variable_library_values()` — schrijft waarden in een gedeployde Variable Library. Dit is precies het mechanisme dat B-lekken *voorkomt*. | +| [setup/NB_SETUP_FMD.ipynb](../setup/NB_SETUP_FMD.ipynb) | cel `d6931af9`/`8f569535` | A | — | Roept `deploy_deployment_pipeline` aan voor `code` en `data`. Namen afgeleid van `domain_name` + `environments` — geen hardcoding. | +| [setup/NB_SETUP_BUSINESS_DOMAINS.ipynb](../setup/NB_SETUP_BUSINESS_DOMAINS.ipynb) | cel 47 | A | — | Idem voor `code/data/reporting/semantic` per business-domein. | +| ↳ | cel 49 | A | — | Wire-loop naar de orchestrator, met `integration_workspace_prefix` afgeleid uit `configuration_database_workspace`. | +| ↳ | cel 44 | A | — | Auto-fill van `VAR_GOLD_SHORTCUTS_FMD`: zoekt Gold/Silver workspace+lakehouse-ID's op per stage en zet ze via `set_variable_library_values`. Lakehouse-namen `LH_GOLD_LAYER`/`LH_SILVER_LAYER` zijn framework-conventie, niet omgevingsspecifiek. | +| [setup/NB_SETUP_FMD.ipynb](../setup/NB_SETUP_FMD.ipynb) | ~393-408 | A | — | TST-environment toegevoegd (`' DATA (T)'`, `' CODE (T)'`), capacity `capacity_name_dvlm`. Namen worden opgebouwd uit `domain_name` + `framework_post_fix`, dus generiek. | +| [setup/NB_SETUP_BUSINESS_DOMAINS.ipynb](../setup/NB_SETUP_BUSINESS_DOMAINS.ipynb) | cel 19 | A | — | Idem: `environment_name: 'test'`, `environment_short: 'T'`. | + +### 2.5 De enige echte B + +| Bestand | Regel(s) | Cat. | Bestemming | Aangetroffen waarde | +|---|---|---|---|---| +| [setup/NB_SETUP_FMD.ipynb](../setup/NB_SETUP_FMD.ipynb#L274) | 274 (cel "Repo Configuration") | **B** | `manifest` | `repo_owner = "bitmetric-fabric" # Owner of the repository (bitmetric's fork, not upstream edkreuk)` | +| [setup/NB_SETUP_BUSINESS_DOMAINS.ipynb](../setup/NB_SETUP_BUSINESS_DOMAINS.ipynb#L288) | 288 (idem) | **B** | `manifest` | idem | + +Toelichting: dit is niet je trial-capacity, maar het is wél een waarde die per +klantimplementatie vaststaat en nooit meer verandert — precies de definitie van `manifest`. +Een klant die dit template forkt naar `klant-x/FMD_FRAMEWORK` moet deze regel aanpassen, +anders trekt de setup-notebook code uit jullie repo. Nu staat het op regel 274 van een +`.ipynb`, waar niemand het vindt. Commit `cb748d8` laat zien dat het bewust is gedaan +(upstream stond op `edkreuk`), dus het is geen ongeluk — maar de plek klopt niet. +Zie vraag V1. + +### 2.6 Twijfelgevallen + +| Bestand | Regel(s) | Cat. | Toelichting | +|---|---|---|---| +| [src/ENV_FMD.Environment/Setting/Sparkcompute.yml](../src/ENV_FMD.Environment/Setting/Sparkcompute.yml#L2-L9) | 2-9 | **?** | `driver_cores: 8 → 4`, `driver_memory: 56g → 28g`, idem executors. De `ponytail:`-comment zegt het zelf: *"sized for the Dev/Test capacity's pool limit (16 cores / 112g)"* en *"If Prod has a bigger capacity, its pool limit may allow raising this back up"*. Dat is letterlijk een omgevingsafhankelijke waarde die als framework-default in de code staat. Maar upstream's 8/56g is óók een willekeurige keuze, en een default moet er zijn. Zie V2. | +| [src/VAR_FMD.VariableLibrary/variables.json](../src/VAR_FMD.VariableLibrary/variables.json#L14) | 14 | **?** | `"value": ""` → `"value": true` voor `lakehouse_schema_enabled` (type `Boolean`). Technisch een bugfix: `""` is geen geldige Boolean. Maar `true` is ook een inhoudelijke keuze; `false` is een legitieme klantkeuze en de setup-notebooks hebben er al een eigen `lakehouse_schema_enabled`-variabele voor. Zie V3. | +| `src/**/VariableLibrary/settings.json` (4×) + `valueSets/Acceptance.json → Development.json` (4×) | settings 3-6 | **?** | `valueSetsOrder` was `[Test, Acceptance, Production]`, is nu `[Development, Test, Production]`. Consistent met de TST-uitbreiding en met de deployment-pipeline-stages (`Development/Test/Production`). Maar het is wél een keuze over jullie DTAP-inrichting die je een klant oplegt. Zie V4. | +| [setup/NB_SETUP_BUSINESS_DOMAINS.ipynb](../setup/NB_SETUP_BUSINESS_DOMAINS.ipynb) | cel 33/34 | **?** | De cel `variable_parameters.update({"SourceWorkspaceId": …, "Shortcut_TargetLakehouseId": …})` is **verwijderd**, vervangen door de auto-fill-cel (cel 44). Ziet er bewust uit, maar de oude cel vulde die ID's vóór deployment in de library-definitie, de nieuwe ná deployment via de API. Als iets de oude route nog gebruikt (bijv. `overwrite_variable_library=True` die de library overschrijft ná de auto-fill) verdwijnen de waarden weer. Niet geverifieerd. Zie V6. | + +Alle overige regels in de 4 variable-library-mappen zijn `\ No newline at end of file`-fixes +— ruis, geen categorie. + +## 3. Sweep: hardcoded omgevingswaarden in *ongewijzigde* bestanden + +Gescand: alle GUID's, `abfss://`-paden, e-mailadressen, connectiestrings, tenant-ID's, +capaciteitsverwijzingen en `*.fabric.microsoft.com`-endpoints in `src/`, `config/`, +`setup/`, `wiki/`, `Taskflow/`, `.github/` en de root-`.md`'s. + +### 3.1 Placeholder-GUID's — geen probleem, maar wel goed om te weten + +`config/item_config.yaml` is upstream's *vervang-tabel*: elke GUID daarin is een "old_id" +die `deploy_item()` → `replace_ids_and_mark_inactive()` tijdens deployment vervangt door de +echte ID in de doelworkspace (`notebook-content.py:707`, `:570-571`). Ze staan met opzet in +de JSON-bronnen. + +| Waarde | Voorkomens | Betekenis | +|---|---|---| +| `372237f9-709a-48f8-8fb2-ce06940c990e` | 161 | `CON_FMD_FABRIC_SQL` | +| `6d8146c6-a438-47df-94e2-540c552eb6d7` | 26 | `CON_FMD_FABRIC_PIPELINES` | +| `40e27fdc-775a-4ee2-84d5-48893c92d7cc` | 9 | `workspaces.workspace_data` | +| `5929775e-aff1-430c-b56e-f855d0bc63b8` | 4 | `CON_FMD_FABRIC_NOTEBOOKS` | +| `00000000-0000-0000-0000-000000000000` | 57 | expliciet gemapt naar de doelworkspace | + +**Wel een risico:** dit is een impliciete conventie zonder enige validatie. Als iemand een +nieuw item toevoegt met een GUID die *niet* in `item_config.yaml` of +`item_deployment*.json` staat, wordt hij niet vervangen en wijst het gedeployde artefact +stilletjes naar Erwins tenant. Zie V7. + +### 3.2 Echte hardcoded waarden die al in upstream zitten + +| Bestand | Regel | Waarde | Probleem | +|---|---|---|---| +| [config/item_config.yaml](../config/item_config.yaml#L19) | 19 | `nl7yhqnbrscude3yv6mas6bxpq-tndhi54vbs4urpccunwiydulzi.datawarehouse.fabric.microsoft.com,1433` | Echte SQL-endpoint-hostname van upstream's config-database. Wordt wel gemapt (`notebook-content.py:696`, old_id = `deployment_item["endpoint"]`), dus functioneel ok — maar het lekt een tenant-specifiek endpoint in een publiek template. | +| [src/business_domain/NB_CREATE_DIMDATE.Notebook/notebook-content.py](../src/business_domain/NB_CREATE_DIMDATE.Notebook/notebook-content.py#L11-L16) | 11, 13, 16 | `default_lakehouse: 63ddad67-a2c0-424b-bae6-d93db2fc592a`, `default_lakehouse_workspace_id: 89c45add-7ca9-4fdf-be0f-8e5665294031` | Deze twee GUID's staan **niet** in `item_config.yaml` en worden dus **niet** vervangen. Notebook opent bij een klant met een lakehouse-binding naar een vreemde tenant. | +| [src/business_domain/NB_MLV_DEMO_GOLD.Notebook/notebook-content.sql](../src/business_domain/NB_MLV_DEMO_GOLD.Notebook/notebook-content.sql#L11-L16) | 11, 13, 16 | `default_lakehouse: b6e09e2a-99d3-4a4a-9d6e-8d978cd44956`, `default_lakehouse_workspace_id: c90c9850-99fe-4050-899c-417cc30d7f70` | `c90c9850-…` ís `workspace_business_domain_data` uit `item_config.yaml:5` en wordt vervangen; `b6e09e2a-…` (de lakehouse) staat nergens in de mapping. Half-gemapt. | +| [Taskflow/FMD_FABRIC_TASKFLOW.json](../Taskflow/FMD_FABRIC_TASKFLOW.json) | meerdere | 9 unieke GUID's, o.a. `168cccc8-dca2-4ab6-a77a-8dece2129896`, `a3e2e795-cd83-4083-8f7d-a5f4cd906116`, `d544f540-9b4e-4b31-a936-aaaafb9d21f6` | Rauwe export van upstream's taskflow; geen enkele staat in de vervang-tabel. Wordt voor zover ik zie ook niet gedeployed — waarschijnlijk documentatie-artefact. | + +Alle andere notebooks (`NB_FMD_LOAD_BRONZE_SILVER`, `NB_FMD_LOAD_LANDING_BRONZE`, +`NB_FMD_CUSTOM_NOTEBOOK_TEMPLATE`, `NB_FMD_LOAD_DEMO_DATA`, `NB_CREATE_SHORTCUTS`, +`NB_FMD_DQ_CLEANSING`, `NB_FMD_PROCESSING_*`) hebben `default_lakehouse: null` en bouwen +hun `abfss://`-paden volledig uit parameters. Schoon. + +### 3.3 Schoon bevonden + +- Geen e-mailadressen, tenant-ID's, connectiestrings of wachtwoorden in `src/`, `config/`, `setup/`, `wiki/`. +- Geen capaciteits-ID's; overal `''`. +- Alle group-/principal-ID's in de setup-notebooks en de root-`.md`'s staan op `00000000-0000-0000-0000-000000000000`. +- `business_domain_names = ['FINANCE','SALES']`, `domain_name = 'INTEGRATION'`, `framework_post_fix = ''` — upstream-defaults, ongewijzigd. +- Alle vier `variables.json` hebben lege `value`-velden (enige uitzondering: `lakehouse_schema_enabled`, zie V3). +- `wiki/LoadGroup_Scheduling.md` bevat geen GUID's, workspace-namen of capaciteitsverwijzingen. + +### 3.4 Losse opmerking + +`docs/fmd-readiness.md` staat untracked in de working tree (`git status`). Niet meegenomen +in deze analyse; laat weten of hij mee moet in het template of weg kan. + +## 4. Vragen aan Mitchell + +**V1 — `repo_owner`.** De setup-notebooks halen de framework-code uit +`github.com/{repo_owner}/{repo_name}`. Nu hardcoded op `bitmetric-fabric` op regel 274/288 +van twee `.ipynb`'s. Wat wil je? +  (a) Laten staan — klanten forken dit template en jij verwacht dat ze de regel zelf aanpassen. +  (b) Naar `manifest` — één `manifest.yml`/`manifest.json` in de repo-root met `repo_owner`, `repo_name`, `branch`, `domain_name`, `framework_post_fix`, capaciteitsnamen; de notebooks lezen die. Eén plek die een klant bij oplevering invult. +  (c) Terug naar `edkreuk` en klanten laten upstream trekken — lijkt me niet, want dan krijgen ze jullie fork-verbeteringen niet. + +**V2 — Sparkcompute.** `driver/executor: 4 cores / 28g` is gekozen voor de pool-limiet van +jouw Dev/Test-capacity (16 cores / 112g). Bij een klant met een F64 is dat onnodig krap; bij +een klant met een F2 nog steeds te groot. Wat is de bedoeling? +  (a) Zo laten als veilige ondergrens die op elke capacity start — dan de `ponytail:`-comment herschrijven naar "conservatieve default, verhoog bij grotere capacity" i.p.v. een verwijzing naar *jouw* Dev/Test. +  (b) Naar `variable-library` / per-stage config — maar `Sparkcompute.yml` is een Environment-item, dat kan geen library-variabelen lezen; dan moet de setup-notebook het bestand per environment schrijven. Meer werk. +  (c) Terug naar upstream's 8/56g en jouw trial-limiet als lokale afwijking behandelen die je niet commit. + +**V3 — `lakehouse_schema_enabled`.** Upstream had `"value": ""` bij type `Boolean` — kapot. +Jij hebt er `true` van gemaakt. Is `true` de framework-default die je aan elke klant wil +meegeven, of moet dit `false` zijn, of moet het bij oplevering gezet worden (dan `manifest`, +want het verandert daarna nooit meer en de setup-notebooks hebben er al een eigen vlag voor)? + +**V4 — `Acceptance` → `Development`.** Je hebt de valueSet hernoemd en de volgorde op +`[Development, Test, Production]` gezet, passend bij de drie deployment-pipeline-stages. Maar +sommige klanten hebben wél een echte Acceptance-omgeving (DTAP i.p.v. DTP). +  (a) Drie stages vastzetten als framework-standaard. +  (b) Vier valueSets meeleveren (`Development`, `Test`, `Acceptance`, `Production`) en de klant laat er één leeg — kost niets, `deploy_deployment_pipeline()` neemt de stages al uit een lijst. +  (c) Stages uit `manifest` laten komen. + +**V5 — MLV-refresh lakehouse-ID.** `NB_MLV_EXAMPLE` regel ~49 heeft +`lakehouse_id = ""` als handmatige invulplek. Je hebt in dezelfde +branch `set_variable_library_values()` gebouwd die precies dit soort ID's automatisch invult +(`VAR_GOLD_SHORTCUTS_FMD.SourceLakehouseId` ís de Gold-lakehouse-ID). Was het de bedoeling +dat het notebook die variabele leest via `notebookutils.variableLibrary`, of is de handmatige +placeholder bewust omdat de klant zijn eigen MLV-notebook schrijft en dit puur een voorbeeld is? + +**V6 — verwijderde `variable_parameters.update`-cel.** In `NB_SETUP_BUSINESS_DOMAINS` is de +cel die `SourceWorkspaceId`/`Shortcut_TargetLakehouseId` etc. in `variable_parameters` zette +verdwenen, vervangen door de auto-fill-cel ná deployment. Bevestig je dat dit bewust is? +En: draait de auto-fill-cel gegarandeerd ná de cel die de VariableLibrary deployt met +`overwrite_variable_library=True`? Zo niet, worden de auto-ingevulde waarden bij een +her-run overschreven met leeg. Ik heb de celvolgorde niet geverifieerd tegen een echte run. + +**V7 — placeholder-GUID-conventie.** Het hele deployment-model leunt erop dat elke GUID in +`src/` in `item_config.yaml` of `item_deployment*.json` staat. Er is geen check. De sweep +vond al drie GUID's die er doorheen glippen (`NB_CREATE_DIMDATE`, `NB_MLV_DEMO_GOLD`, +`Taskflow`). Wil je daar een CI-guard op, in de trant van `dacpac-guard.yml`: "elke GUID in +`src/` staat in de vervang-tabel, anders faalt de build"? Dat vangt precies het soort lek dat +je hier zoekt, automatisch, voor elke toekomstige wijziging. Dit is niet iets dat ik nu +gedaan heb — alleen een voorstel. + +**V8 — `NB_CREATE_DIMDATE` / `NB_MLV_DEMO_GOLD` / `Taskflow`.** Deze drie dragen upstream's +tenant-ID's mee (§3.2). Ze zijn niet van jou en zaten er al in. Moeten ze in het template +blijven (demo-materiaal), of horen ze in een aparte `examples/`-map die je bij oplevering +weggooit? Dat is een upstream-fix die je als PR terug zou kunnen geven. diff --git a/manifest.example.yaml b/manifest.example.yaml new file mode 100644 index 00000000..c527c3cf --- /dev/null +++ b/manifest.example.yaml @@ -0,0 +1,120 @@ +# FMD klant-manifest +# +# Eén bestand per klantimplementatie, ingevuld bij oplevering. Bevat alles wat per +# klant vaststaat en daarna niet meer verandert (ADR-003). +# +# Hoort hier NIET: +# - waarden die per omgeving verschillen binnen één klant → Variable Library +# - waarden die tijdens gebruik veranderen → integration.* tabellen +# +# Kopieer naar manifest.yaml (niet in git bij gevoelige waarden) en vul in. +# De setup-notebooks lezen dit bestand; niets hiervan hoort nog in een notebookcel. + +# --- Herkomst van de framework-code ----------------------------------------- +# Was hardcoded op regel 274/288 van de setup-notebooks (triage V1). +repository: + owner: bitmetric-fabric + name: FMD_FRAMEWORK + branch: main + +# --- Naamgeving -------------------------------------------------------------- +naming: + domain_name: INTEGRATION # gedeelde ingestielaag + framework_post_fix: '' # achtervoegsel op alle workspace-namen, bv. '_ACME' + business_domains: + - FINANCE + - SALES + # - HR + +# --- Omgevingen -------------------------------------------------------------- +# Vier value sets worden altijd meegeleverd; hier bepaal je welke actief zijn +# (triage V4). Een klant met DTAP zet Acceptance erbij, een klant met DTP niet. +# De volgorde bepaalt de stage-volgorde van de Deployment Pipeline. +# +# capacity : capaciteit voor de gedeelde ingestielaag (INTEGRATION). +# capacity_business_domain : optioneel. Capaciteit voor de workspaces van de +# business domains in deze omgeving. Weggelaten betekent: dezelfde als +# `capacity`. Vul hem in als Gold op een eigen capaciteit moet draaien. +environments: + - name: Development + short: D + capacity: '' + # capacity_business_domain: '' + - name: Test + short: T + capacity: '' + # capacity_business_domain: '' + # - name: Acceptance + # short: A + # capacity: '' + # # capacity_business_domain: '' + - name: Production + short: P + capacity: '' + # capacity_business_domain: '' + +# --- Configuratie-workspace -------------------------------------------------- +# TOEGEVOEGD t.o.v. het aangeleverde voorbeeld (triage D2). De CONFIG-workspace +# draagt de metadata-database en bestaat één keer per klant, niet één keer per +# omgeving — hij past daarom niet in environments hierboven. Kwam uit +# capacity_name_config in NB_SETUP_FMD cel 7. +config_workspace: + capacity: '' + +# --- Spark compute ----------------------------------------------------------- +# Wordt door de setup-notebook naar src/ENV_FMD.Environment/Setting/Sparkcompute.yml +# geschreven (triage V2). Conservatieve default die op elke capacity start; verhoog +# bij een grotere capacity. +# +# Bewust geen override per omgeving: ENV_FMD.Environment wordt één keer gedeployed, +# in de CONFIG-workspace, en door Dev/Test/Prod gedeeld. Een waarde per omgeving is +# in die vorm niet uit te drukken. Zolang dat zo is, geldt deze sizing overal. +spark: + # TOEGEVOEGD: stond op twee plekken (spark_version in cel 3 van beide setup- + # notebooks, runtime_version in Sparkcompute.yml). Nu één bron. + runtime_version: '2.0' + default: + driver_cores: 4 + driver_memory: 28g + executor_cores: 4 + executor_memory: 28g + +# --- Shortcuts (Gold -> Silver) ---------------------------------------------- +# NB_CREATE_SHORTCUTS legt shortcuts van de Gold-lakehouse naar de Silver-lakehouse. +# De workspace- en lakehouse-ID's worden na deployment opgezocht en ingevuld; alleen +# de schemanamen zijn een keuze. Leeg laten als je lakehouses zonder schema's gebruikt. +shortcuts: + source_schema: '' # schema in de Gold-lakehouse + target_schema: '' # schema in de Silver-lakehouse + +# --- Framework-instellingen -------------------------------------------------- +framework: + # Schema-enabled lakehouses. Upstream had hier een lege string bij type Boolean — + # dat is geen geldige waarde (triage V3). true is de default; false is een + # legitieme klantkeuze, maar kies bij oplevering en verander daarna niet meer. + lakehouse_schema_enabled: true + +# --- Beveiliging ------------------------------------------------------------- +# Object-ID's van Entra-groepen die toegang krijgen tot de workspaces. +# Laat op nullen staan tot ze bekend zijn; de setup-notebook slaat ze dan over. +security: + admin_group_id: '00000000-0000-0000-0000-000000000000' + contributor_group_id: '00000000-0000-0000-0000-000000000000' + # Gereserveerd: het framework kent nog geen viewer-rol, dus deze waarde wordt + # op dit moment nergens toegepast. Bewust bewaard voor later. + viewer_group_id: '00000000-0000-0000-0000-000000000000' + # TOEGEVOEGD (triage D5). De rollijsten in de setup-notebooks kennen naast + # Groups ook een ServicePrincipal met contributor-rechten; zonder dit veld zou + # die toegang bij het omzetten naar het manifest wegvallen. Laat op nullen + # staan als je geen SP gebruikt. + service_principal_id: '00000000-0000-0000-0000-000000000000' + +# --- Key Vault --------------------------------------------------------------- +# TOEGEVOEGD. Kwam uit NB_SETUP_FMD cel 9. Dit zijn de NAMEN van de secrets, +# niet de secrets zelf; de waarden worden tijdens de run uit Key Vault gehaald. +key_vault: + name: '' + secrets: + sql_sp_tenant_id: tenantid + sql_sp_client_id: sp-fmd-fabric-pipelines-app-id + sql_sp_secret: sp-fmd-fabric-pipelines-secret diff --git a/scripts/guid_guard.py b/scripts/guid_guard.py new file mode 100644 index 00000000..2c6e46b4 --- /dev/null +++ b/scripts/guid_guard.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +""" +GUID-guard — handhaaft ADR-003 en ADR-009. + +Het deployment-model leunt erop dat elke GUID in src/ een placeholder is die tijdens +deployment vervangen wordt. Staat een GUID niet in de vervang-tabel, dan wijst het +gedeployde artefact stilzwijgend naar een vreemde tenant. + +Dit script faalt wanneer een GUID in SCAN_DIRS niet voorkomt in config/ en niet op de +allowlist staat. + +Geen dependencies: draait op een kale python3. +""" + +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent + +# Waar we op lekken scannen. +SCAN_DIRS = ["src", "setup", "wiki"] + +# Waar bekende (te vervangen) GUID's vandaan komen. Elke GUID die hier ergens +# voorkomt geldt als gemapt — dat is precies de bestaande conventie. +CONFIG_DIRS = ["config"] + +ALLOWLIST_FILE = REPO_ROOT / "config" / "guid_allowlist.txt" + +# Bestandstypen die we openen. Binaire artefacten (.dacpac, .png) slaan we over. +TEXT_SUFFIXES = {".json", ".yaml", ".yml", ".py", ".sql", ".ipynb", ".md", ".txt", ".platform"} + +GUID_RE = re.compile( + r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b" +) + +# Expliciet gemapt naar de doelworkspace door het deployment-script. +ALWAYS_ALLOWED = {"00000000-0000-0000-0000-000000000000"} + + +def iter_text_files(root: Path): + for path in sorted(root.rglob("*")): + if not path.is_file(): + continue + if any(part in {".git", "node_modules", "__pycache__"} for part in path.parts): + continue + if path.suffix.lower() in TEXT_SUFFIXES or path.name == ".platform": + yield path + + +def read(path: Path) -> str | None: + try: + return path.read_text(encoding="utf-8") + except (UnicodeDecodeError, OSError): + return None + + +def notebook_cell_ids(path: Path, text: str) -> set[str]: + """ + Cel-id's van een .ipynb. Die zijn GUID-vormig, maar het zijn geen + verwijzingen: nbformat gebruikt ze om cellen uit elkaar te houden, en + deploy_item() vervangt ze niet (dat zou de notebook juist kapotmaken). + + Ze per stuk op de allowlist zetten kan niet: er komen er bij zodra iemand een + cel toevoegt. Daarom worden ze per bestand overgeslagen, en alleen als ze + daadwerkelijk als cel-id in datzelfde bestand voorkomen. + """ + if path.suffix.lower() != ".ipynb": + return set() + try: + notebook = json.loads(text) + except (json.JSONDecodeError, ValueError): + return set() + ids = set() + for cell in notebook.get("cells", []): + cell_id = cell.get("id") if isinstance(cell, dict) else None + if isinstance(cell_id, str) and GUID_RE.fullmatch(cell_id): + ids.add(cell_id.lower()) + return ids + + +def collect_known_guids() -> set[str]: + known: set[str] = set(ALWAYS_ALLOWED) + for directory in CONFIG_DIRS: + base = REPO_ROOT / directory + if not base.exists(): + continue + for path in iter_text_files(base): + # De allowlist staat zelf in config/, maar telt niet als vervang-tabel: + # anders zou een GUID erin ook zonder reden geaccepteerd worden. + if path == ALLOWLIST_FILE: + continue + text = read(path) + if text: + known.update(g.lower() for g in GUID_RE.findall(text)) + return {g.lower() for g in known} + + +def load_allowlist() -> dict[str, str]: + """Regels als: — . Regels zonder reden worden geweigerd.""" + allowed: dict[str, str] = {} + if not ALLOWLIST_FILE.exists(): + return allowed + for lineno, raw in enumerate(ALLOWLIST_FILE.read_text(encoding="utf-8").splitlines(), 1): + line = raw.strip() + if not line or line.startswith("#"): + continue + match = GUID_RE.search(line) + if not match: + print(f"::warning file={ALLOWLIST_FILE},line={lineno}::regel zonder GUID genegeerd") + continue + guid = match.group(0).lower() + reason = line[match.end():].lstrip(" \t-—:").strip() + if not reason: + print( + f"::error file={ALLOWLIST_FILE},line={lineno}::" + f"{guid} staat op de allowlist zonder reden" + ) + sys.exit(2) + allowed[guid] = reason + return allowed + + +def main() -> int: + known = collect_known_guids() + allowlist = load_allowlist() + permitted = known | set(allowlist) + + violations: list[tuple[Path, int, str]] = [] + + for directory in SCAN_DIRS: + base = REPO_ROOT / directory + if not base.exists(): + continue + for path in iter_text_files(base): + text = read(path) + if not text: + continue + ignored = notebook_cell_ids(path, text) + for lineno, line in enumerate(text.splitlines(), 1): + for guid in GUID_RE.findall(line): + if guid.lower() in permitted or guid.lower() in ignored: + continue + violations.append((path.relative_to(REPO_ROOT), lineno, guid)) + + if not violations: + print(f"GUID-guard OK — {len(known)} gemapte GUID's, {len(allowlist)} op de allowlist.") + return 0 + + print(f"\nGUID-guard: {len(violations)} niet-gemapte GUID(s) gevonden.\n") + for path, lineno, guid in violations: + print(f"::error file={path},line={lineno}::niet-gemapte GUID {guid}") + print(f" {path}:{lineno} {guid}") + + print( + "\nElke GUID in src/, setup/ of wiki/ moet in config/ voorkomen, zodat" + "\ndeploy_item() hem tijdens deployment vervangt. Twee manieren om dit op te lossen:" + "\n" + "\n 1. Registreer het item in config/item_config.yaml of config/item_deployment*.json" + "\n — dit is vrijwel altijd de juiste oplossing." + "\n 2. Staat de GUID er bewust (voorbeeld in documentatie, comment), zet hem dan met" + f"\n een reden in {ALLOWLIST_FILE.relative_to(REPO_ROOT)}." + "\n" + "\nZie docs/adr/009-guid-guard.md." + ) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test_manifest_wiring.py b/scripts/test_manifest_wiring.py new file mode 100644 index 00000000..21b2c425 --- /dev/null +++ b/scripts/test_manifest_wiring.py @@ -0,0 +1,375 @@ +"""Check op de manifest-bedrading (PR 3). Geen dependencies, geen netwerk. + + python3 scripts/test_manifest_wiring.py + +Twee dingen: + 1. manifest.example.yaml reproduceert de configuratie van voor PR 3, door de + manifest- en configuratiecellen van beide setup-notebooks echt uit te voeren. + 2. apply_spark_settings en apply_value_sets doen wat ze beloven op kopieen van + de echte bestanden - inclusief het behouden van sleutels, commentaar en + regeleindes die het manifest niet noemt. +""" + +import json +import os +import shutil +import sys +import tempfile +import types +from pathlib import Path + +REPO = Path(sys.argv[1] if len(sys.argv) > 1 else Path(__file__).parent.parent).resolve() +EXAMPLE = (REPO / "manifest.example.yaml").read_text(encoding="utf-8") + +# --- stub voor requests -------------------------------------------------- +class _Response: + status_code = 200 + text = EXAMPLE + + def raise_for_status(self): + pass + + +fake_requests = types.ModuleType("requests") +fake_requests.get = lambda url, *a, **k: _Response() +sys.modules["requests"] = fake_requests + + +def cells_of(path): + notebook = json.loads(path.read_text(encoding="utf-8")) + return {c.get("id"): "".join(c["source"]) for c in notebook["cells"]}, notebook["cells"] + + +def compile_all(cells, name): + bad = 0 + for cell in cells: + if cell["cell_type"] != "code": + continue + source = "".join(cell["source"]) + if source.lstrip().startswith(("%", "!")) or "\n%" in source: + continue # magics: geen geldige Python + try: + compile(source, f"{name}:{cell.get('id')}", "exec") + except SyntaxError as exc: + print(f" SYNTAXFOUT {name} cel {cell.get('id')}: {exc}") + bad += 1 + return bad + + +def run(path, cell_ids): + sources, cells = cells_of(path) + errors = compile_all(cells, path.name) + scope = {} + for cell_id in cell_ids: + exec(compile(sources[cell_id], f"{path.name}:{cell_id}", "exec"), scope) + return scope, errors + + +print("=== compileren + uitvoeren ===") + +fmd, e1 = run( + REPO / "setup" / "NB_SETUP_FMD.ipynb", + [ + "fmd-manifest-bootstrap", + "afbf0bf0-89ce-4eb9-bdc3-8efd8444ea4c", + "c6095d96-c13d-49fd-b388-7347598d2032", + "c5e6f777-a65a-491b-ba14-0ce8b19c04ff", + "99ab87f1-553e-426f-9c42-1f2232cd5e8f", + "3ada47fa-6823-4cdc-bebe-3f573f4f2bd1", + "40ebe458-bd01-495a-81a1-6b7b5aef9ec8", + ], +) + +bd, e2 = run( + REPO / "setup" / "NB_SETUP_BUSINESS_DOMAINS.ipynb", + [ + "bd-manifest-bootstrap", + "afbf0bf0-89ce-4eb9-bdc3-8efd8444ea4c", + "2086f732-b219-4101-9eab-92c95044c075", + "c6095d96-c13d-49fd-b388-7347598d2032", + "c5e6f777-a65a-491b-ba14-0ce8b19c04ff", + "99ab87f1-553e-426f-9c42-1f2232cd5e8f", + "60ded5a6-4998-4f43-a7aa-5a9c26dad6a1", + ], +) + +print(f" syntaxfouten: {e1 + e2}") + +# --- verwachte waarden van VOOR PR 3 ------------------------------------- +CAP = "" +expected = [] + + +def check(label, got, want): + expected.append((label, got == want, got, want)) + + +check("FMD domain_name", fmd["domain_name"], "INTEGRATION") +check("FMD framework_post_fix", fmd["framework_post_fix"], "") +check("FMD lakehouse_schema_enabled", fmd["lakehouse_schema_enabled"], True) +check("FMD spark_version", fmd["spark_version"], "2.0") +check("FMD capacity_name_config", fmd["capacity_name_config"], CAP) +check("FMD key_vault", fmd["key_vault"], "") +check("FMD sp_tenant_id", fmd["fabric_sql_sp_tenant_id"], "tenantid") +check("FMD sp_client_id", fmd["fabric_sql_sp_client_id"], "sp-fmd-fabric-pipelines-app-id") +check("FMD sp_secret", fmd["fabric_sql_sp_secret"], "sp-fmd-fabric-pipelines-secret") + +check( + "FMD environments (namen/capaciteit)", + [ + (e["environment_name"], e["workspaces"]["data"]["name"], e["workspaces"]["code"]["name"], + e["workspaces"]["data"]["capacity_name"]) + for e in fmd["environments"] + ], + [ + ("development", "INTEGRATION DATA (D)", "INTEGRATION CODE (D)", CAP), + ("test", "INTEGRATION DATA (T)", "INTEGRATION CODE (T)", CAP), + ("production", "INTEGRATION DATA (P)", "INTEGRATION CODE (P)", CAP), + ], +) + +check( + "FMD configuration", + (fmd["configuration"]["workspace"]["name"], fmd["configuration"]["DatabaseName"], + fmd["configuration"]["workspace"]["capacity_name"]), + ("INTEGRATION CONFIG", "SQL_INTEGRATION_FRAMEWORK", CAP), +) + +# Nul-ID's in het voorbeeldmanifest -> lege rollijsten (voorheen: lijsten met +# nul-GUID's waar fab acl set stil op faalde). +check("FMD workspace_roles_code", fmd["workspace_roles_code"], []) +check("FMD workspace_roles_data", fmd["workspace_roles_data"], []) +check("FMD domain_contributor_role principals", fmd["domain_contributor_role"]["principals"], []) +check("FMD connection_role principals", fmd["connection_role"]["principals"], []) + +check("BD business_domain_names", bd["business_domain_names"], ["FINANCE", "SALES"]) +check("BD SourceSchema uit manifest", bd["SourceSchema"], "") +check("BD Shortcut_TargetSchema uit manifest", bd["Shortcut_TargetSchema"], "") +check("BD configuration_database_workspace", bd["configuration_database_workspace"], "INTEGRATION CONFIG") +check("BD configuration_database_name", bd["configuration_database_name"], "SQL_INTEGRATION_FRAMEWORK") +check( + "BD business_domain_deployment", + [(e["environment_name"], e["environment_short"], sorted(e["workspaces"]), + e["workspaces"]["data"]["capacity_name"]) for e in bd["business_domain_deployment"]], + [ + ("development", "D", ["code", "data", "reporting", "semantic"], CAP), + ("test", "T", ["code", "data", "reporting", "semantic"], CAP), + ("production", "P", ["code", "data", "reporting", "semantic"], CAP), + ], +) + +# D3: zodra capacity_business_domain wel is ingevuld, moet hij die pakken. +# In place muteren, want require() sluit over de globals van zijn eigen cel. +bd_sources, _ = cells_of(REPO / "setup" / "NB_SETUP_BUSINESS_DOMAINS.ipynb") +bd_environments_cell = bd_sources["60ded5a6-4998-4f43-a7aa-5a9c26dad6a1"] +for env in bd["manifest"]["environments"]: + env["capacity_business_domain"] = f"GOLD-{env['short']}" +exec(compile(bd_environments_cell, "bd-env", "exec"), bd) +check( + "BD capaciteit volgt capacity_business_domain", + [e["workspaces"]["data"]["capacity_name"] for e in bd["business_domain_deployment"]], + ["GOLD-D", "GOLD-T", "GOLD-P"], +) + +print("\n=== vergelijking met de defaults van voor PR 3 ===") +failed = 0 +for label, ok, got, want in expected: + if ok: + print(f" OK {label}") + else: + failed += 1 + print(f" FOUT {label}\n nu: {got!r}\n was: {want!r}") + +# --- rollen met echte ID's ----------------------------------------------- +print("\n=== rollen met ingevulde ID's (simulatie) ===") +role_scope = dict(fmd) +role_scope["manifest"]["security"] = { + "admin_group_id": "11111111-1111-1111-1111-111111111111", + "contributor_group_id": "22222222-2222-2222-2222-222222222222", + "service_principal_id": "33333333-3333-3333-3333-333333333333", +} +sources, _ = cells_of(REPO / "setup" / "NB_SETUP_FMD.ipynb") +exec(compile(sources["99ab87f1-553e-426f-9c42-1f2232cd5e8f"], "roles", "exec"), role_scope) +for name in ("workspace_roles_code", "workspace_roles_data"): + print(f" {name}:") + for role in role_scope[name]: + print(f" {role['principal']['type']:<16} {role['role']:<12} {role['principal']['id']}") + +print(f"\nresultaat: {failed} afwijking(en), {e1 + e2} syntaxfout(en)") +NOTEBOOK_FAILURES = failed + e1 + e2 + + +# --- de twee functies uit het notebook halen ----------------------------- +utilities_source = (REPO / "src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py").read_text(encoding="utf-8") +start = utilities_source.index("def apply_spark_settings(") +end = utilities_source.index("# -------------------------------\n# Item deployment") +scope = {"os": os, "json": json} + +MANIFEST = { + "spark": { + "runtime_version": "2.0", + "default": {"driver_cores": 4, "driver_memory": "28g", + "executor_cores": 4, "executor_memory": "28g"}, + }, + "environments": [{"name": "Development"}, {"name": "Test"}, {"name": "Production"}], +} + + +def _lookup(path): + node = MANIFEST + for part in path.split("."): + if not isinstance(node, dict) or part not in node: + return None, False + node = node[part] + return node, True + + +scope["require"] = lambda p: _lookup(p)[0] +scope["optional"] = lambda p, d=None: _lookup(p)[0] if _lookup(p)[1] else d +exec(compile(utilities_source[start:end], "hooks", "exec"), scope) + +failures = [] + + +def check(label, condition, detail=""): + print((" OK " if condition else " FOUT ") + label + ("" if condition else f" {detail}")) + if not condition: + failures.append(label) + + +# --- Sparkcompute --------------------------------------------------------- +print("=== apply_spark_settings ===") +original = (REPO / "src/ENV_FMD.Environment/Setting/Sparkcompute.yml").read_bytes() + +for environment, expected_driver in (("config", "4"),): + tmp = Path(tempfile.mkdtemp()) + (tmp / "Setting").mkdir() + (tmp / "Setting/Sparkcompute.yml").write_bytes(original) + + scope["apply_spark_settings"](str(tmp)) + result = (tmp / "Setting/Sparkcompute.yml").read_bytes().decode("utf-8") + + check(f"[{environment}] driver_cores = {expected_driver}", + f"driver_cores: {expected_driver}\r\n" in result or f"driver_cores: {expected_driver}\n" in result, + result) + check(f"[{environment}] onbekende sleutels behouden", + "enable_native_execution_engine: true" in result + and "dynamic_executor_allocation:" in result + and "max_executors: 2" in result) + check(f"[{environment}] commentaar behouden", result.count("#") >= 4) + check(f"[{environment}] CRLF behouden", "\r\n" in result) + check(f"[{environment}] geen dubbele sleutels", + all(sum(1 for line in result.splitlines() if line.startswith(key + ":")) == 1 + for key in ("driver_cores", "driver_memory", "executor_cores", + "executor_memory", "runtime_version"))) + shutil.rmtree(tmp, ignore_errors=True) + +# alles wat het manifest niet noemt blijft staan, byte voor byte +tmp = Path(tempfile.mkdtemp()) +(tmp / "Setting").mkdir() +(tmp / "Setting/Sparkcompute.yml").write_bytes(original) +scope["apply_spark_settings"](str(tmp)) +after_bytes = (tmp / "Setting/Sparkcompute.yml").read_bytes() +untouched = [line for line in original.decode("utf-8").splitlines() + if not line.split(":", 1)[0].strip() in + ("driver_cores", "driver_memory", "executor_cores", "executor_memory", "runtime_version")] +check("regels buiten het manifest ongewijzigd", + all(line in after_bytes.decode("utf-8") for line in untouched)) +shutil.rmtree(tmp, ignore_errors=True) + +# ontbrekend bestand mag niet crashen +tmp = Path(tempfile.mkdtemp()) +scope["apply_spark_settings"](str(tmp)) +check("ontbrekende Sparkcompute.yml wordt overgeslagen", True) +shutil.rmtree(tmp, ignore_errors=True) + +# --- valueSets ------------------------------------------------------------ +print("\n=== apply_value_sets ===") +library = REPO / "src/VAR_FMD.VariableLibrary" +tmp = Path(tempfile.mkdtemp()) / "lib" +shutil.copytree(library, tmp) + +before = sorted(p.stem for p in (tmp / "valueSets").glob("*.json")) +scope["apply_value_sets"](str(tmp)) +after = sorted(p.stem for p in (tmp / "valueSets").glob("*.json")) +order = json.loads((tmp / "settings.json").read_text(encoding="utf-8"))["valueSetsOrder"] + +check("vier value sets in src/", before == ["Acceptance", "Development", "Production", "Test"], before) +check("Acceptance verwijderd (niet in manifest)", after == ["Development", "Production", "Test"], after) +check("valueSetsOrder volgt manifest", order == ["Development", "Test", "Production"], order) +shutil.rmtree(tmp.parent, ignore_errors=True) + +# met Acceptance erbij moet hij blijven staan +MANIFEST["environments"].insert(2, {"name": "Acceptance"}) +tmp = Path(tempfile.mkdtemp()) / "lib" +shutil.copytree(library, tmp) +scope["apply_value_sets"](str(tmp)) +after = sorted(p.stem for p in (tmp / "valueSets").glob("*.json")) +check("DTAP behoudt Acceptance", after == ["Acceptance", "Development", "Production", "Test"], after) +shutil.rmtree(tmp.parent, ignore_errors=True) + + +# --- D6: VAR_GOLD_SHORTCUTS_FMD kan weer gedeployed worden ---------------- +print("\n=== update_variable_library / VAR_GOLD_SHORTCUTS_FMD ===") + +declared = next( + item for item in json.loads( + (REPO / "config/item_deployment_code_business_domain.json").read_text(encoding="utf-8")) + if item["name"].startswith("VAR_GOLD_SHORTCUTS") +)["variables"] + +uvl_start = utilities_source.index("def update_variable_library(") +uvl_end = utilities_source.index("def copy_to_tmp(") +uvl_source = utilities_source[uvl_start:uvl_end] + + +def run_update(parameters): + """Draait update_variable_library op een kopie van de echte library.""" + library = Path(tempfile.mkdtemp()) / "lib" + shutil.copytree(REPO / "src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary", library) + uvl_scope = {"json": json, "variable_parameters": parameters} + exec(compile(uvl_source, "uvl", "exec"), uvl_scope) + uvl_scope["update_variable_library"](str(library), declared) + written = json.loads((library / "variables.json").read_text(encoding="utf-8")) + shutil.rmtree(library.parent, ignore_errors=True) + return written + + +# zoals NB_UTILITIES variable_parameters aanmaakt, voordat de herstelcel draait +base_parameters = { + "key_vault_uri_name": "val_key_vault_uri_name", + "lakehouse_schema_enabled": True, + "purview_account_name": "val_purview_account_name", +} + +try: + run_update(dict(base_parameters)) + check("zonder de herstelcel faalt het deployen", False, "er ging niets mis") +except KeyError as exc: + check("zonder de herstelcel faalt het deployen, met bruikbare melding", + "SourceWorkspaceId" in str(exc) and "variable_parameters" in str(exc), str(exc)) + +restored = dict(base_parameters) +restore_scope = dict(bd) +restore_scope["variable_parameters"] = restored +exec(compile(bd_sources["bd-variable-parameters"], "vp", "exec"), restore_scope) + +check("herstelcel vult precies de zes gedeclareerde bronnen", + sorted(k for k in restored if k not in base_parameters), + sorted(v["source"] for v in declared)) + +written = run_update(restored) +check("library houdt zes variabelen", + [v["name"] for v in written["variables"]], + [v["name"] for v in declared]) +check("schema komt uit het manifest", + [v["value"] for v in written["variables"] if v["name"] == "SourceSchema"], + [bd["manifest"]["shortcuts"]["source_schema"]]) +check("ID's blijven leeg tot de auto-fill ze invult", + [v["value"] for v in written["variables"] + if v["name"] in ("SourceWorkspaceId", "SourceLakehouseId", + "Shortcut_TargetWorkspaceId", "Shortcut_TargetLakehouseId")], + ["", "", "", ""]) + +print(f"\nresultaat: {len(failures)} fout(en)") +sys.exit(1 if (failures or NOTEBOOK_FAILURES) else 0) diff --git a/setup/NB_BOOTSTRAP_FMD.ipynb b/setup/NB_BOOTSTRAP_FMD.ipynb new file mode 100644 index 00000000..317b8df9 --- /dev/null +++ b/setup/NB_BOOTSTRAP_FMD.ipynb @@ -0,0 +1,186 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "bootstrap-intro", + "metadata": {}, + "source": [ + "# FMD bootstrap\n", + "\n", + "Het enige bestand dat je met de hand in Fabric hoeft te zetten.\n", + "\n", + "Vul hieronder in waar jouw fork staat, en draai het notebook. Het haalt\n", + "`NB_SETUP_FMD` en `NB_SETUP_BUSINESS_DOMAINS` uit die repo, stempelt dezelfde\n", + "drie waarden in hun bootstrap-cel, en zet ze in deze workspace.\n", + "\n", + "Alle overige configuratie staat in `manifest.yaml` in de root van je fork; de\n", + "setup-notebooks lezen die zelf. Zie `manifest.example.yaml` voor de structuur.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bootstrap-config", + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "outputs": [], + "source": [ + "# =============================================================================\n", + "# Waar staat jouw fork?\n", + "# =============================================================================\n", + "repo_owner = \"bitmetric-fabric\" # GitHub-organisatie of -gebruiker\n", + "repo_name = \"FMD_FRAMEWORK\" # Naam van de repository\n", + "branch = \"main\" # Branch waaruit gedeployed wordt\n", + "folder_prefix = \"\" # Alleen invullen als src/ en config/ in een submap staan\n", + "\n", + "# Deze waarden moeten gelijk zijn aan repository.* in manifest.yaml; de\n", + "# setup-notebooks controleren dat en stoppen bij een verschil.\n" + ] + }, + { + "cell_type": "markdown", + "id": "bootstrap-deploy-md", + "metadata": {}, + "source": [ + "## Setup-notebooks ophalen en plaatsen" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bootstrap-deploy", + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "outputs": [], + "source": [ + "import base64\n", + "import json\n", + "\n", + "import requests\n", + "\n", + "SETUP_NOTEBOOKS = {\n", + " \"NB_SETUP_FMD\": \"fmd-manifest-bootstrap\",\n", + " \"NB_SETUP_BUSINESS_DOMAINS\": \"bd-manifest-bootstrap\",\n", + "}\n", + "\n", + "FABRIC_API = \"https://api.fabric.microsoft.com/v1\"\n", + "\n", + "workspace_id = notebookutils.runtime.context.get(\"currentWorkspaceId\")\n", + "headers = {\n", + " \"Authorization\": f\"Bearer {notebookutils.credentials.getToken('pbi')}\",\n", + " \"Content-Type\": \"application/json\",\n", + "}\n", + "\n", + "\n", + "def fetch(notebook_name):\n", + " \"\"\"Haalt het setup-notebook op uit de opgegeven fork.\"\"\"\n", + " prefix = f\"{folder_prefix}/\" if folder_prefix else \"\"\n", + " url = (\n", + " f\"https://raw.githubusercontent.com/{repo_owner}/{repo_name}/{branch}/\"\n", + " f\"{prefix}setup/{notebook_name}.ipynb\"\n", + " )\n", + " response = requests.get(url)\n", + " if response.status_code == 404:\n", + " raise RuntimeError(f\"Niet gevonden: {url}\")\n", + " response.raise_for_status()\n", + " return json.loads(response.text), url\n", + "\n", + "\n", + "def stamp(notebook, cell_id):\n", + " \"\"\"Zet repo_owner/repo_name/branch/folder_prefix in de bootstrap-cel.\"\"\"\n", + " values = {\n", + " \"repo_owner\": repo_owner,\n", + " \"repo_name\": repo_name,\n", + " \"branch\": branch,\n", + " \"folder_prefix\": folder_prefix,\n", + " }\n", + " for cell in notebook[\"cells\"]:\n", + " if cell.get(\"id\") != cell_id:\n", + " continue\n", + " source = cell[\"source\"]\n", + " source = source if isinstance(source, list) else source.splitlines(keepends=True)\n", + " for index, line in enumerate(source):\n", + " key = line.split(\"=\", 1)[0].strip()\n", + " if key in values:\n", + " ending = \"\\n\" if line.endswith(\"\\n\") else \"\"\n", + " source[index] = f'{key:<13} = \"{values[key]}\"{ending}'\n", + " del values[key]\n", + " cell[\"source\"] = source\n", + " if values:\n", + " raise RuntimeError(f\"Niet gestempeld in {cell_id}: {sorted(values)}\")\n", + " return notebook\n", + " raise RuntimeError(\n", + " f\"Cel '{cell_id}' ontbreekt. Draait deze fork nog op een versie van voor \"\n", + " f\"het manifest (PR 3)?\"\n", + " )\n", + "\n", + "\n", + "def find_item(display_name):\n", + " response = requests.get(f\"{FABRIC_API}/workspaces/{workspace_id}/items?type=Notebook\", headers=headers)\n", + " response.raise_for_status()\n", + " for item in response.json().get(\"value\", []):\n", + " if item[\"displayName\"] == display_name:\n", + " return item[\"id\"]\n", + " return None\n", + "\n", + "\n", + "def publish(display_name, notebook):\n", + " payload = base64.b64encode(json.dumps(notebook).encode(\"utf-8\")).decode(\"utf-8\")\n", + " definition = {\n", + " \"format\": \"ipynb\",\n", + " \"parts\": [\n", + " {\"path\": \"notebook-content.ipynb\", \"payload\": payload, \"payloadType\": \"InlineBase64\"}\n", + " ],\n", + " }\n", + " item_id = find_item(display_name)\n", + " if item_id:\n", + " url = f\"{FABRIC_API}/workspaces/{workspace_id}/items/{item_id}/updateDefinition\"\n", + " body = {\"definition\": definition}\n", + " verb = \"bijgewerkt\"\n", + " else:\n", + " url = f\"{FABRIC_API}/workspaces/{workspace_id}/items\"\n", + " body = {\"displayName\": display_name, \"type\": \"Notebook\", \"definition\": definition}\n", + " verb = \"aangemaakt\"\n", + "\n", + " response = requests.post(url, headers=headers, json=body)\n", + " if response.status_code not in (200, 201, 202):\n", + " raise RuntimeError(f\"{display_name}: HTTP {response.status_code} {response.text}\")\n", + " print(f\"{display_name} {verb}\")\n", + "\n", + "\n", + "for notebook_name, cell_id in SETUP_NOTEBOOKS.items():\n", + " notebook, source_url = fetch(notebook_name)\n", + " publish(notebook_name, stamp(notebook, cell_id))\n", + " print(f\" bron: {source_url}\")\n", + "\n", + "print(\n", + " \"\\nKlaar. Open NB_SETUP_FMD en draai hem. Zorg dat manifest.yaml in de root \"\n", + " f\"van {repo_owner}/{repo_name} (branch '{branch}') staat en ingevuld is.\"\n", + ")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Synapse PySpark", + "name": "synapse_pyspark" + }, + "language_info": { + "name": "python" + }, + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/setup/NB_SETUP_BUSINESS_DOMAINS.ipynb b/setup/NB_SETUP_BUSINESS_DOMAINS.ipynb index ecb07ad3..d18e1bb7 100644 --- a/setup/NB_SETUP_BUSINESS_DOMAINS.ipynb +++ b/setup/NB_SETUP_BUSINESS_DOMAINS.ipynb @@ -1 +1,1576 @@ -{"cells":[{"cell_type":"markdown","source":["![FMD_Overview](https://github.com/edkreuk/FMD_FRAMEWORK/blob/main/Images/FMD_DOMAIN_OVERVIEW.png?raw=true)"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"ce6a663d-0e92-4d07-99e6-365c48b9c726"},{"cell_type":"code","source":["%pip install --upgrade ms-fabric-cli pillow cairosvg --quiet"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"1f028883-3c89-4adf-8da2-9a1d7f5b979d"},{"cell_type":"markdown","source":["# Configuration and Parameters\n","\n","**Fabric Administrator Role is required to create domain**"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"63ed7375-2e5f-4b5a-ae9f-672b337ddd94"},{"cell_type":"code","source":["assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons\n","lakehouse_schema_enabled = True # Set to True if you want to use the lakehouse schema, otherwise set to False\n","\n","driver = '{ODBC Driver 18 for SQL Server}' # Change this if you use a different driver\n","overwrite_variable_library=True # By default the Library is overwritten, change this to \"False\" if you have custom changes\n","spark_version='2.0'"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"afbf0bf0-89ce-4eb9-bdc3-8efd8444ea4c"},{"cell_type":"markdown","source":["## KeyVault settings\n","For future usage"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"4e9e5d8d-7471-4d92-8385-9cc5ae81a47b"},{"cell_type":"code","source":["key_vault_uri_name='val_key_vault_uri_name'\n","purview_account_name='val_purview_account_name'"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"2df9de02-a95b-495f-b3a2-c97ee3abb92c"},{"cell_type":"markdown","source":["## Shortcut settings"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"e2c55186-dabf-4be9-b1b4-028ccb6a6a45"},{"cell_type":"code","source":["##No need to fill in the first time, because the workspaces does not exist yet. After the first run, you can fill in the workspace and lakehouse ids to speed up the deployment process, otherwise the script will look them up by name every time which takes more time.\n","##Used for Shortcuts, can also manualy filled in\n","SourceWorkspaceId='' \n","SourceLakehouseId='' # Your Gold lakehouse \n","SourceSchema=''\n","Shortcut_TargetSchema=''\n","Shortcut_TargetWorkspaceId=''\n","Shortcut_TargetLakehouseId='' # Your Silver Lakehouse"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"2086f732-b219-4101-9eab-92c95044c075"},{"cell_type":"markdown","source":["## Capacity configuration"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"aaa9cdee-13a8-4154-a7ce-6b13aec0d244"},{"cell_type":"code","source":["reassign_capacity= True # If set to False existing assigned capacities to workspaces will no be overwritten\n","\n","capacity_name_business_domain_dvlm = '' # Which capacity will be used for the workspaces in this business domain\n","capacity_name_business_domain_prod = '' # Which capacity will be used for the workspaces in this business domain"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"c6095d96-c13d-49fd-b388-7347598d2032"},{"cell_type":"markdown","source":["## Domain and Framework settings"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"aac4aa59-bbe8-4937-a983-8ddf5859760e"},{"cell_type":"code","source":["framework_post_fix= '' # post fix to be added at the end of workspace for example INTEGRATION CODE(D) FMD\n","if framework_post_fix != '':\n"," framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility\n","\n","##Domains\n","create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed\n","\n","business_domain_names= ['FINANCE','SALES'] # Define business domains\n","domain_contributor_role = {\"type\": \"Contributors\",\"principals\": [{\"id\": \"00000000-0000-0000-0000-000000000000\",\"type\": \"Group\"} ]} # Which group(Object ID) can add or remove workspaces to this domain\n","configuration_database_workspace='INTEGRATION CONFIG' # Name of the workspace of Fabric database which was deployed for the Integration Domain\n","configuration_database_name='SQL_INTEGRATION_FRAMEWORK' # Name of the Fabric database which was deployed for the Integration Domain"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"c5e6f777-a65a-491b-ba14-0ce8b19c04ff"},{"cell_type":"code","source":["configuration = {\n"," 'workspace': {\n"," 'name' : configuration_database_workspace # Name of target workspace\n"," },\n"," 'DatabaseName' : configuration_database_name # Name of target configuration SQL Database\n","}"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"8f8adca7-35a2-4bd6-b5d2-0cabfb9c4049"},{"cell_type":"markdown","source":["# Configuration\n"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"207ce033-5b29-4c7a-9b6f-03854ddf933c"},{"cell_type":"markdown","source":["## Repo Configuration"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"a8e67871-6536-43c0-ac2e-21ff8571d022"},{"cell_type":"code","source":["#FMD Framework code\n","##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE ####\n","repo_owner = \"edkreuk\" # Owner of the repository\n","repo_name = \"FMD_FRAMEWORK\" # Name of the repository\n","branch = \"main\" is default \n","folder_prefix = \"\"\n","###################################################"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"e3945911-68db-4961-99db-5020ac37e1b7"},{"cell_type":"markdown","source":["## Workspace Roles Configuration"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"0443c89a-6aec-46c1-9456-37e8f4ab93aa"},{"cell_type":"code","source":["workspace_roles_data_business_domain = [ # Keep emtpy [] if you only want to assign this to your personal account\n"," {\n"," \"principal\": {\n"," \"id\": '00000000-0000-0000-0000-000000000000',\n"," \"type\": \"Group\"\n"," },\n"," \"role\": \"admin\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n"," }\n"," ]\n","\n","workspace_roles_code_business_domain = [ # Keep emtpy [] if you only want to assign this to your personal account\n"," {\n"," \"principal\": {\n"," \"id\": '00000000-0000-0000-0000-000000000000',\n"," \"type\": \"Group\"\n"," },\n"," \"role\": \"admin\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n"," }\n"," ]\n","\n","workspace_roles_reporting_business_domain = [ # Keep emtpy [] if you only want to assign this to your personal account\n"," {\n"," \"principal\": {\n"," \"id\": '00000000-0000-0000-0000-000000000000',\n"," \"type\": \"Group\"\n"," },\n"," \"role\": \"admin\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n"," }\n"," ]"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"99ab87f1-553e-426f-9c42-1f2232cd5e8f"},{"cell_type":"markdown","source":["## Business Domain Configuration"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"5faac4b5-58fd-4c94-af27-adfd443d43ee"},{"cell_type":"code","source":["##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE, FE ADDING NEW ENVIRONMENTS ####\n","business_domain_deployment = [\n"," {\n"," 'environment_name' : 'development', # Name of target environment\n"," 'environment_short' : 'D', # Short of target environment\n"," 'workspaces': {\n"," \n"," 'data' : {\n"," 'roles' : workspace_roles_data_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_dvlm # Name of target data workspace for development\n"," },\n"," 'code' : {\n"," 'roles' : workspace_roles_code_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_dvlm # Name of target code workspace for development\n"," },\n"," 'reporting' : {\n"," 'roles' : workspace_roles_reporting_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_dvlm # Name of target code workspace for development\n"," },\n"," 'semantic' : {\n"," 'roles' : workspace_roles_reporting_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_dvlm # Name of target code workspace for development\n"," }\n"," }\n"," },\n"," {\n"," 'environment_name' : 'production', # Name of target environment\n"," 'environment_short' : 'P', # Short of target environment\n"," 'workspaces': {\n"," \n"," 'data' : {\n"," 'roles' : workspace_roles_data_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_prod # Name of target data workspace for development\n"," },\n"," 'code' : {\n"," 'roles' : workspace_roles_code_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_prod # Name of target code workspace for development\n"," },\n"," 'reporting' : {\n"," 'roles' : workspace_roles_reporting_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_prod # Name of target code workspace for development\n"," },\n"," 'semantic' : {\n"," 'roles' : workspace_roles_reporting_business_domain, # Roles to assign to the workspace\n"," 'capacity_name' : capacity_name_business_domain_prod # Name of target code workspace for development\n"," }\n"," }\n"," }\n"," ]\n","###################################################"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"60ded5a6-4998-4f43-a7aa-5a9c26dad6a1"},{"cell_type":"markdown","source":["## Download source & config files"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"32269821-d83f-41ac-a3d4-a19f8cb555de"},{"cell_type":"code","source":["import subprocess\n","import os\n","from time import sleep, time\n","import json\n","import shutil\n","import re\n","import requests\n","import zipfile\n","import yaml\n","import struct\n","import pyodbc\n","import cairosvg\n","import base64\n","import xml.etree.ElementTree as ET\n","\n","from PIL import Image, ImageDraw, ImageFont\n","from requests.adapters import HTTPAdapter, Retry\n","from io import BytesIO\n","from zipfile import ZipFile "],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"31e1c04a-6901-4ec9-a4f1-0ee84c56dc77"},{"cell_type":"code","source":["def download_folders_as_zip(repo_owner, repo_name, output_zip, branch=\"main\", folders_to_extract=None, remove_folder_prefix=\"\"):\n"," if folders_to_extract is None:\n"," folders_to_extract = []\n","\n"," # Construct the URL for the GitHub API to download the repository as a zip file\n"," url = f\"https://api.github.com/repos/{repo_owner}/{repo_name}/zipball/{branch}\"\n"," response = requests.get(url)\n"," response.raise_for_status()\n","\n"," # Ensure the directory for the output zip file exists\n"," os.makedirs(os.path.dirname(output_zip), exist_ok=True)\n","\n"," # Create a zip file in memory from GitHub response\n"," with zipfile.ZipFile(BytesIO(response.content)) as zipf:\n"," # Open output zip in append mode\n"," with zipfile.ZipFile(output_zip, 'w') as output_zipf:\n"," \n"," for file_info in zipf.infolist():\n"," for folder in folders_to_extract:\n"," folder_path = f\"/{folder}\" if not folder.startswith(\"/\") else folder\n"," if re.sub(r'^.*?/', '/', file_info.filename).startswith(folder_path):\n"," file_data = zipf.read(file_info.filename)\n"," parts = file_info.filename.split('/')\n"," if remove_folder_prefix:\n"," parts = [p for p in parts if p != remove_folder_prefix]\n"," output_zipf.writestr('/'.join(parts[1:]), file_data)\n","\n","def uncompress_zip_to_folder(zip_path, extract_to):\n"," os.makedirs(extract_to, exist_ok=True)\n"," with zipfile.ZipFile(zip_path, 'r') as zip_ref:\n"," zip_ref.extractall(extract_to)\n"," os.remove(zip_path)\n","\n","def copy_to_tmp(name):\n"," shutil.rmtree(\"./builtin/tmp\", ignore_errors=True)\n"," path2zip = \"./builtin/src/src.zip\"\n","\n"," prefixes = [\n"," f\"src/{name}\"\n"," ]\n","\n"," with ZipFile(path2zip) as archive:\n"," for prefix in prefixes:\n"," matched_files = [file for file in archive.namelist() if file.startswith(prefix)]\n"," if matched_files:\n"," for file in matched_files:\n"," archive.extract(file, \"./builtin/tmp\")\n"," return f\"./builtin/tmp/{prefix}\" # Return only the first matching prefix\n","\n"," return None # Nothing found\n","\n","# ✅ Combine all folders into one zip\n","download_folders_as_zip(repo_owner, repo_name, output_zip = \"./builtin/src/src.zip\", branch = branch, folders_to_extract= [f\"{folder_prefix}/src\"] , remove_folder_prefix = f\"{folder_prefix}\")\n","download_folders_as_zip(repo_owner, repo_name, output_zip = \"./builtin/config/config.zip\", branch = branch, folders_to_extract= [f\"{folder_prefix}/config\"] , remove_folder_prefix = f\"{folder_prefix}\")\n","# ✅ Uncompress everything into ./builtin\n","uncompress_zip_to_folder(zip_path = \"./builtin/config/config.zip\", extract_to= \"./builtin\")\n","\n","mapping_table=[]\n","tasks=[]\n"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"e22cc332-d939-4def-8971-84f762931ce2"},{"cell_type":"markdown","source":["# CLI Login"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"8a5c3526-9a63-430c-90e4-00ece0aebfa5"},{"cell_type":"code","source":["# Set environment parameters for Fabric CLI\n","token = notebookutils.credentials.getToken('pbi')\n","os.environ['FAB_TOKEN'] = token\n","os.environ['FAB_TOKEN_ONELAKE'] = token"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"f00e2fa8-e10d-45a9-82a7-266feaedc4ba"},{"cell_type":"markdown","source":["# Deployment functions"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"34b39ba9-62fb-421b-be07-2607f79fccdb"},{"cell_type":"markdown","source":["## Load configuration\n"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"33eb5fa4-5bf7-408b-9630-1533ba0d031a"},{"cell_type":"code","source":["base_path = './builtin/'\n","config_path = os.path.join(base_path, 'config/item_config.yaml')\n","\n","with open(config_path, 'r') as file:\n"," config = yaml.safe_load(file)\n","\n","deploy_order_path = os.path.join(base_path, 'config/item_initial_setup.json')\n","with open(deploy_order_path, 'r') as file:\n"," item_initial_setup =json.load(file)\n","\n","deploy_order_path = os.path.join(base_path, 'config/data_deployment.json')\n","with open(deploy_order_path, 'r') as file:\n"," data_deployment =json.load(file)\n","\n","deploy_order_path = os.path.join(base_path, 'config/item_deployment_code_business_domain.json')\n","with open(deploy_order_path, 'r') as file:\n"," item_deployment_code_business_domain =json.load(file)\n","\n","deploy_order_path = os.path.join(base_path, 'config/item_deployment_data_business_domain.json')\n","with open(deploy_order_path, 'r') as file:\n"," item_deployment_data_business_domain =json.load(file)\n","\n","deploy_icons_path = os.path.join(base_path, 'config/fabric_icons.xml')\n","\n","# Parse the XML file\n","tree = ET.parse(deploy_icons_path)\n","root = tree.getroot()\n","\n","# Create a dictionary to store icon name and base64\n","fabric_icons_fmd = {}\n","for item in root.findall('icon'):\n"," name = item.find('name').text if item.find('name') is not None else \"No name\"\n"," base64_str = item.find('base64').text if item.find('base64') is not None else \"\"\n"," fabric_icons_fmd[name] = base64_str\n"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"source_hidden":false},"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"937feb4e-a67f-486c-ad43-0ae628b71e34"},{"cell_type":"markdown","source":["# Deployment"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"3143250a-9c75-4f6a-9f44-78dde9852195"},{"cell_type":"markdown","source":["## Load latest version NB_UTILITIES_SETUP_FMD"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"e38a1ab7-998e-4959-86fd-8e82c0894076"},{"cell_type":"code","source":["def run_fab_command(command, capture_output=False, silently_continue=False, raw_output=False):\n"," \"\"\"\n"," Executes a Fabric CLI command with optional output capture and error handling.\n"," \"\"\"\n"," result = subprocess.run([\"fab\", \"-c\", command], capture_output=capture_output, text=True)\n"," if not silently_continue and (result.returncode > 0 or result.stderr):\n"," raise Exception(f\"Error running fab command. exit_code: '{result.returncode}'; stderr: '{result}'\")\n"," if capture_output:\n"," return result if raw_output else result.stdout.strip()\n"," return None"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"1ccfaae6-01d5-4ae2-a1bf-9d3fca3b240b"},{"cell_type":"code","source":["workspace_id = notebookutils.runtime.context.get('currentWorkspaceId')\n","result=run_fab_command(\"api -X get workspaces/\"+workspace_id, capture_output=True, silently_continue=False)\n","workspace_name=json.loads(result)[\"text\"][\"displayName\"]\n","\n","for it in item_initial_setup:\n"," name = it[\"name\"]\n"," type = it[\"type\"]\n"," tmp_path = copy_to_tmp(name)\n","\n"," cli_parameter = ''\n","\n"," if \"Notebook\" in name:\n"," cli_parameter += \" --format .py\"\n"," result = run_fab_command(f\"import {workspace_name}.Workspace/{name} -i {tmp_path} -f {cli_parameter}\",capture_output=False, silently_continue=True)\n"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"1ebe7e00-f8dd-44bb-80ee-0d4e68222b10"},{"cell_type":"code","source":["%run NB_UTILITIES_SETUP_FMD"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"18f2fd5f-00cd-4892-ae2a-588def93869d"},{"cell_type":"code","source":["variable_parameters.update ( {\n"," \"SourceWorkspaceId\": SourceWorkspaceId,\n"," \"SourceLakehouseId\": SourceLakehouseId,\n"," \"SourceSchema\": SourceSchema,\n"," \"Shortcut_TargetSchema\": Shortcut_TargetSchema,\n"," \"Shortcut_TargetWorkspaceId\": Shortcut_TargetWorkspaceId,\n"," \"Shortcut_TargetLakehouseId\": Shortcut_TargetLakehouseId,\n","})"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"bc993f0f-a32b-4fc2-8738-bfd07c9ba8ed"},{"cell_type":"markdown","source":["## Integration Domain"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"8a1d9b38-da94-42a7-a561-99bc7d2d86ac"},{"cell_type":"markdown","source":["### Get Fabric database (Configuration)"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"33263830-e547-48a7-bc11-9ece4c56d6da"},{"cell_type":"code","source":["for deployment_item in data_deployment:\n"," if deployment_item['type'] in ('SQLDatabase'):\n"," name = configuration['DatabaseName']\n"," type = deployment_item[\"type\"]\n"," name=name+'.'+type\n"," server, database_name=deploy_item(workspace_name=configuration['workspace']['name'],name=name,mapping_table=mapping_table, environment_name='config',tasks= tasks, lakehouse_schema_enabled=lakehouse_schema_enabled,it=deployment_item)\n","\n","##!NOTE: in Case of an error check if you're not using a trial capacitiy or the capacity you assigned is paused"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"59c4ddf9-cb4b-4849-9749-053d7aeb7a87"},{"cell_type":"markdown","source":["## Business Domain"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"78cd9671-cdb3-476d-a447-7181f76af32c"},{"cell_type":"markdown","source":["### Create workspaces(Business Domains)"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"97127361-afea-438f-9569-af3a6c0b981f"},{"cell_type":"code","source":["for business_domain_name in business_domain_names:\n"," if create_domains:\n"," create_fabric_domain(business_domain_name)\n","\n"," for business_domain in business_domain_deployment:\n"," print(f\"--------------------------\")\n"," print(f\"Updating Workspace: {business_domain['environment_name']}\")\n"," deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['code'],workspace_name=business_domain_name + ' CODE ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_code\"], mapping_table=mapping_table, tasks=tasks)\n"," deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['data'],workspace_name=business_domain_name + ' DATA ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_data\"], mapping_table=mapping_table, tasks=tasks)\n"," deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['reporting'],workspace_name=business_domain_name + ' REPORTING ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_reporting\"], mapping_table=mapping_table, tasks=tasks)\n"," deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['semantic'],workspace_name=business_domain_name + ' SEMANTIC ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_reporting\"], mapping_table=mapping_table, tasks=tasks)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"330d211a-1b1e-44b2-b989-93e751d5612c"},{"cell_type":"markdown","source":["### Create Lakehouses(Business Domains)"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"532de2ef-cc3a-40a1-8bfd-436c09ec1d4d"},{"cell_type":"code","source":["for workspace_business_domain in business_domain_deployment:\n"," for business_domain in business_domain_names:\n"," for workspace in [workspace_business_domain['workspaces']['data']]:\n"," exclude = []\n"," for it in item_deployment_data_business_domain:\n"," new_id = None \n"," name = it[\"name\"]\n"," type = it[\"type\"]\n"," if name in exclude:\n"," continue\n"," deploy_item(business_domain + ' DATA ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"7c2a7033-c33f-4802-ba74-a93ac10fd798"},{"cell_type":"markdown","source":["### Create Items(Business Domains)"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"b0faae07-cb6e-4ae4-a3ef-37fe3db24dcd"},{"cell_type":"code","source":["for workspace_business_domain in business_domain_deployment:\n"," for business_domain_name in business_domain_names:\n"," for workspace in [workspace_business_domain['workspaces']['code']]:\n"," for it in item_deployment_code_business_domain:\n"," new_id = None\n"," if it['type'] in ('VariableLibrary'):\n"," name = it[\"name\"]\n"," type = it[\"type\"]\n"," deploy_item(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"8d8220fc-6410-4ed7-a8f5-3936da7f5557"},{"cell_type":"code","source":["for workspace_business_domain in business_domain_deployment:\n"," for business_domain_name in business_domain_names:\n"," for workspace in [workspace_business_domain['workspaces']['code']]:\n"," for it in item_deployment_code_business_domain:\n"," new_id = None\n"," if it['type'] not in ('VariableLibrary'):\n"," name = it[\"name\"]\n"," type = it[\"type\"]\n"," deploy_item(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"c321c689-995e-4943-ad05-db9db2d885f8"},{"cell_type":"markdown","source":["## Workspace Icons"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"5d91a5c1-bd07-46b6-a88f-ba7fcb305c4f"},{"cell_type":"code","source":["if assign_icons: \n"," seen = set()\n"," workspaces = []\n"," # Get cluster URL for use in metadata endpoints\n"," cluster_base_url = get_cluster_url()\n","\n"," for item in mapping_table:\n"," if item['ItemType'] == 'Workspace':\n"," key = (item['Description'], item['new_id'])\n"," if key not in seen:\n"," seen.add(key)\n"," workspaces.append({'displayName': item['Description'], 'id': item['new_id']})\n"," fabric_icons = fabric_icons_fmd \n","\n"," for workspace in workspaces:\n"," display_name = workspace['displayName'].lower()\n"," \n"," # Assign icon\n"," for icon_key, icon_value in workspace_icon_def['icons'].items():\n"," if icon_key in display_name:\n"," workspace[\"icon\"] = icon_value\n"," workspace_icon = fabric_icons.get(icon_value)\n"," break\n"," else:\n"," workspace[\"icon\"] = None\n"," workspace_icon = None\n","\n"," workspace[\"icon_base64img\"] = workspace_icon"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"662a019b-f9dc-48f5-9fad-14ca06c30193"},{"cell_type":"code","source":["if assign_icons:\n"," # Dry run - Display pre and post icons based on specified workspace filters and workspace icon definition. Will NOT update any icons!\n"," display_workspace_icons(workspaces)\n"," sleep(2)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"f0bdcf88-cac2-4546-8c7f-cbfdb390461e"},{"cell_type":"markdown","source":["### Deploy Workspace Icons"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"ca909ec2-916a-4400-82bc-acf434bc4550"},{"cell_type":"code","source":["if assign_icons:\n"," for workspace in workspaces:\n"," set_workspace_icon(workspace.get('id'), workspace.get('icon_base64img'))"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"806f42d6-e74c-4909-8ac0-ab14d8f68e0e"},{"cell_type":"markdown","source":["## Create SQL deployment Manifest"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"4cc252ff-6f50-41f9-9c6d-165300837bb6"},{"cell_type":"markdown","source":["### Add Workspaces to Fabric Database"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"8ed267d7-4aca-445c-9189-243307a972e6"},{"cell_type":"code","source":["#add all created workspace to database\n","custom_sql_deployment = {\"queries_stored_procedures\": []}\n","unique_items = {}\n","for item in mapping_table:\n"," if item.get(\"ItemType\") == \"Workspace\":\n"," unique_items[item[\"new_id\"]] = item[\"Description\"]\n","\n","# Convert to list of tuples or dicts\n","workspaces = [{\"Description\": desc, \"new_id\": nid} for nid, desc in unique_items.items()]\n","\n","for workspace in workspaces:\n"," print(f'EXEC [integration].[sp_UpsertWorkspace](@WorkspaceId = \"{workspace[\"new_id\"]}\" ,@Name = \"{workspace[\"Description\"]}\")')\n"," custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertWorkspace] @WorkspaceId = \"{workspace[\"new_id\"]}\", @Name = \"{workspace[\"Description\"]}\"')"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"5ee6f1cd-8aaa-4c94-be39-6721312e6cde"},{"cell_type":"code","source":["for workspace_business_domain in business_domain_deployment:\n"," for business_domain_name in business_domain_names:\n"," for workspace in [workspace_business_domain['workspaces']['code']]:\n"," workspace_id = get_workspace_id_by_name(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix)\n"," result = run_fab_command(f\"api -X get workspaces/{workspace_id}/items\", capture_output=True, silently_continue=True)\n"," existing_items = json.loads(result)['text']\n"," for item in existing_items.get('value', []):\n"," if item['type'] == 'DataPipeline':\n"," print(f'EXEC [integration].[sp_UpsertPipeline] @PipelineId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n"," custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertPipeline] @PipelineId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"5a9c1eb2-b4a4-448b-82a8-fdeef790d1e8"},{"cell_type":"markdown","source":["### Add Lakehouses to Fabric Database"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"4867745a-f0e7-4a9a-b326-5609b1b5bb3a"},{"cell_type":"code","source":["for workspace_business_domain in business_domain_deployment:\n"," for business_domain_name in business_domain_names:\n"," for workspace in [workspace_business_domain['workspaces']['data']]:\n"," workspace_id = get_workspace_id_by_name(business_domain_name + ' DATA ('+ workspace_business_domain['environment_short']+')'+framework_post_fix)\n"," result = run_fab_command(f\"api -X get workspaces/{workspace_id}/items\", capture_output=True, silently_continue=True)\n"," existing_items = json.loads(result)['text']\n"," for item in existing_items.get('value', []):\n"," if item['type'] == 'Lakehouse':\n"," print(f'EXEC [integration].[sp_UpsertLakehouse] @LakehouseId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n"," custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertLakehouse] @LakehouseId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n"," "],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"3144899e-74f6-4eb0-a203-525437d8c127"},{"cell_type":"markdown","source":["## Deploy SQL Code"],"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"},"nteract":{"transient":{"deleting":false}}},"id":"31e3bfba-b56e-47c7-a1e5-c764c2c3e622"},{"cell_type":"code","source":["#Make sure out token is still valid before we continou, sometime it can take a bit longer that's we will check it again\n","token = notebookutils.credentials.getToken('pbi')\n","os.environ['FAB_TOKEN'] = token\n","os.environ['FAB_TOKEN_ONELAKE'] = token"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"jupyter_python"}},"id":"6ac7b58e-1240-4110-a4a1-c7c767bad1ee"},{"cell_type":"code","source":["for deployment_item in data_deployment:\n"," if deployment_item['type'] in ('SQLDatabase'):\n"," name = configuration['DatabaseName']\n"," type = deployment_item[\"type\"]\n"," name=name+'.'+type\n"," if not server: #it was already set, just in case\n"," server=get_item_id(configuration['workspace']['name'], name, 'properties.serverFqdn')\n"," if not database_name: \n"," database_name=get_item_id(configuration['workspace']['name'], name, 'properties.databaseName')\n","\n","try:\n"," i = 0\n","\n"," token = notebookutils.credentials.getToken('pbi').encode('utf-16-le')\n"," token_struct = struct.pack(f' principal-dicts. Nul-ID's worden weggelaten: die\n", + " betekenen 'nog niet bekend', en fab acl set loopt erop stuk.\"\"\"\n", + " out = []\n", + " for principal_id, principal_type in specs:\n", + " if principal_id and str(principal_id).replace(\"-\", \"\").strip(\"0\"):\n", + " out.append({\"id\": str(principal_id), \"type\": principal_type})\n", + " return out\n", + "\n", + "\n", + "def roles(*specs):\n", + " \"\"\"[(id, type, role), ...] -> rollijst voor deploy_workspaces.\"\"\"\n", + " out = []\n", + " for principal_id, principal_type, role in specs:\n", + " for principal in principals((principal_id, principal_type)):\n", + " out.append({\"principal\": principal, \"role\": role})\n", + " return out\n", + "\n", + "\n", + "# De bootstrap-cel en het manifest moeten dezelfde repo aanwijzen; anders komen\n", + "# src/ en config/ straks ergens anders vandaan dan het manifest beschrijft.\n", + "for _key, _bootstrap_value in (\n", + " (\"repository.owner\", repo_owner),\n", + " (\"repository.name\", repo_name),\n", + " (\"repository.branch\", branch),\n", + "):\n", + " if str(require(_key)) != str(_bootstrap_value):\n", + " raise RuntimeError(\n", + " f\"Bootstrap en manifest spreken elkaar tegen: {_key} is \"\n", + " f\"'{require(_key)}' in manifest.yaml maar '{_bootstrap_value}' in de \"\n", + " f\"bootstrap-cel hierboven.\"\n", + " )\n", + "\n", + "print(f\"Manifest geladen van {MANIFEST_URL}\")\n" + ] + }, + { + "cell_type": "code", + "source": [ + "%pip install --upgrade ms-fabric-cli pillow cairosvg --quiet" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "1f028883-3c89-4adf-8da2-9a1d7f5b979d" + }, + { + "cell_type": "markdown", + "source": [ + "# Configuration and Parameters\n", + "\n", + "**Fabric Administrator Role is required to create domain**" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "63ed7375-2e5f-4b5a-ae9f-672b337ddd94" + }, + { + "cell_type": "code", + "source": [ + "assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons\n", + "\n", + "driver = '{ODBC Driver 18 for SQL Server}' # Change this if you use a different driver\n", + "overwrite_variable_library=True # By default the Library is overwritten, change this to \"False\" if you have custom changes\n", + "\n", + "# Uit het manifest - pas deze aan in manifest.yaml, niet hier.\n", + "lakehouse_schema_enabled = require('framework.lakehouse_schema_enabled')\n", + "spark_version = str(require('spark.runtime_version'))\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "afbf0bf0-89ce-4eb9-bdc3-8efd8444ea4c" + }, + { + "cell_type": "markdown", + "source": [ + "## KeyVault settings\n", + "For future usage" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "4e9e5d8d-7471-4d92-8385-9cc5ae81a47b" + }, + { + "cell_type": "code", + "source": [ + "key_vault_uri_name='val_key_vault_uri_name'\n", + "purview_account_name='val_purview_account_name'" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "2df9de02-a95b-495f-b3a2-c97ee3abb92c" + }, + { + "cell_type": "markdown", + "source": [ + "## Shortcut settings" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "e2c55186-dabf-4be9-b1b4-028ccb6a6a45" + }, + { + "cell_type": "code", + "source": [ + "##### Shortcut-schema's komen uit manifest.shortcuts. De workspace- en lakehouse-ID's\n", + "##### stonden hier vroeger ook, maar die worden na deployment opgezocht en ingevuld\n", + "##### door de cel \"Auto-fill VAR_GOLD_SHORTCUTS_FMD\" verderop.\n", + "SourceSchema = require('shortcuts.source_schema') # schema in de Gold-lakehouse\n", + "Shortcut_TargetSchema = require('shortcuts.target_schema') # schema in de Silver-lakehouse\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "2086f732-b219-4101-9eab-92c95044c075" + }, + { + "cell_type": "markdown", + "source": [ + "## Capacity configuration" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "aaa9cdee-13a8-4154-a7ce-6b13aec0d244" + }, + { + "cell_type": "code", + "source": [ + "reassign_capacity= True # If set to False existing assigned capacities to workspaces will no be overwritten\n", + "\n", + "# De capaciteit per omgeving komt uit manifest.environments[].capacity en wordt\n", + "# in de cel \"Business Domain Configuration\" toegepast.\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "c6095d96-c13d-49fd-b388-7347598d2032" + }, + { + "cell_type": "markdown", + "source": [ + "## Domain and Framework settings" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "aac4aa59-bbe8-4937-a983-8ddf5859760e" + }, + { + "cell_type": "code", + "source": [ + "framework_post_fix= require('naming.framework_post_fix') # post fix to be added at the end of workspace for example INTEGRATION CODE(D) FMD\n", + "if framework_post_fix != '':\n", + " framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility\n", + "\n", + "##Domains\n", + "create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed\n", + "\n", + "business_domain_names= require('naming.business_domains') # Define business domains\n", + "\n", + "# Groep die workspaces aan dit domein mag toevoegen of verwijderen.\n", + "domain_contributor_role = {\"type\": \"Contributors\", \"principals\": principals(\n", + " (require('security.contributor_group_id'), \"Group\"),\n", + ")}\n", + "\n", + "# Afgeleid van naming.domain_name, net als in NB_SETUP_FMD - niet apart instellen.\n", + "configuration_database_workspace = require('naming.domain_name') + ' CONFIG' + framework_post_fix\n", + "configuration_database_name = 'SQL_' + require('naming.domain_name') + '_FRAMEWORK'\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "c5e6f777-a65a-491b-ba14-0ce8b19c04ff" + }, + { + "cell_type": "code", + "source": [ + "configuration = {\n", + " 'workspace': {\n", + " 'name' : configuration_database_workspace # Name of target workspace\n", + " },\n", + " 'DatabaseName' : configuration_database_name # Name of target configuration SQL Database\n", + "}" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "8f8adca7-35a2-4bd6-b5d2-0cabfb9c4049" + }, + { + "cell_type": "markdown", + "source": [ + "# Configuration\n" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "207ce033-5b29-4c7a-9b6f-03854ddf933c" + }, + { + "cell_type": "markdown", + "source": [ + "## Workspace Roles Configuration" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "0443c89a-6aec-46c1-9456-37e8f4ab93aa" + }, + { + "cell_type": "code", + "source": [ + "# Rollen komen uit manifest.security. Een ID dat nog op nullen staat betekent\n", + "# \"nog niet bekend\" en wordt overgeslagen; fab acl set loopt daar anders op stuk.\n", + "workspace_roles_data_business_domain = roles(\n", + " (require('security.admin_group_id'), 'Group', 'admin'),\n", + ")\n", + "\n", + "workspace_roles_code_business_domain = roles(\n", + " (require('security.admin_group_id'), 'Group', 'admin'),\n", + ")\n", + "\n", + "workspace_roles_reporting_business_domain = roles(\n", + " (require('security.admin_group_id'), 'Group', 'admin'),\n", + ")\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "99ab87f1-553e-426f-9c42-1f2232cd5e8f" + }, + { + "cell_type": "markdown", + "source": [ + "## Business Domain Configuration" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "5faac4b5-58fd-4c94-af27-adfd443d43ee" + }, + { + "cell_type": "code", + "source": [ + "##### Opgebouwd uit manifest.environments - voeg omgevingen daar toe, niet hier ####\n", + "\n", + "def business_domain_capacity(env):\n", + " \"\"\"Business domains mogen op een eigen capaciteit draaien. Staat\n", + " capacity_business_domain niet in het manifest, dan delen ze die van de\n", + " ingestielaag - zoals het voor het manifest ook ging.\"\"\"\n", + " return env.get('capacity_business_domain') or env['capacity']\n", + "\n", + "\n", + "business_domain_deployment = [\n", + " {\n", + " 'environment_name': env['name'].lower(), # Sleutel in mapping_table; lowercase zoals deploy_item verwacht\n", + " 'environment_short': env['short'], # D / T / A / P\n", + " 'workspaces': {\n", + " 'data': {'roles': workspace_roles_data_business_domain, 'capacity_name': business_domain_capacity(env)},\n", + " 'code': {'roles': workspace_roles_code_business_domain, 'capacity_name': business_domain_capacity(env)},\n", + " 'reporting': {'roles': workspace_roles_reporting_business_domain, 'capacity_name': business_domain_capacity(env)},\n", + " 'semantic': {'roles': workspace_roles_reporting_business_domain, 'capacity_name': business_domain_capacity(env)},\n", + " },\n", + " }\n", + " for env in require('environments')\n", + "]\n", + "###################################################\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "60ded5a6-4998-4f43-a7aa-5a9c26dad6a1" + }, + { + "cell_type": "markdown", + "source": [ + "## Download source & config files" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "32269821-d83f-41ac-a3d4-a19f8cb555de" + }, + { + "cell_type": "code", + "source": [ + "import subprocess\n", + "import os\n", + "from time import sleep, time\n", + "import json\n", + "import shutil\n", + "import re\n", + "import requests\n", + "import zipfile\n", + "import yaml\n", + "import struct\n", + "import pyodbc\n", + "import cairosvg\n", + "import base64\n", + "import xml.etree.ElementTree as ET\n", + "\n", + "from PIL import Image, ImageDraw, ImageFont\n", + "from requests.adapters import HTTPAdapter, Retry\n", + "from io import BytesIO\n", + "from zipfile import ZipFile " + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "31e1c04a-6901-4ec9-a4f1-0ee84c56dc77" + }, + { + "cell_type": "code", + "source": [ + "def download_folders_as_zip(repo_owner, repo_name, output_zip, branch=\"main\", folders_to_extract=None, remove_folder_prefix=\"\"):\n", + " if folders_to_extract is None:\n", + " folders_to_extract = []\n", + "\n", + " # Construct the URL for the GitHub API to download the repository as a zip file\n", + " url = f\"https://api.github.com/repos/{repo_owner}/{repo_name}/zipball/{branch}\"\n", + " response = requests.get(url)\n", + " response.raise_for_status()\n", + "\n", + " # Ensure the directory for the output zip file exists\n", + " os.makedirs(os.path.dirname(output_zip), exist_ok=True)\n", + "\n", + " # Create a zip file in memory from GitHub response\n", + " with zipfile.ZipFile(BytesIO(response.content)) as zipf:\n", + " # Open output zip in append mode\n", + " with zipfile.ZipFile(output_zip, 'w') as output_zipf:\n", + " \n", + " for file_info in zipf.infolist():\n", + " for folder in folders_to_extract:\n", + " folder_path = f\"/{folder}\" if not folder.startswith(\"/\") else folder\n", + " if re.sub(r'^.*?/', '/', file_info.filename).startswith(folder_path):\n", + " file_data = zipf.read(file_info.filename)\n", + " parts = file_info.filename.split('/')\n", + " if remove_folder_prefix:\n", + " parts = [p for p in parts if p != remove_folder_prefix]\n", + " output_zipf.writestr('/'.join(parts[1:]), file_data)\n", + "\n", + "def uncompress_zip_to_folder(zip_path, extract_to):\n", + " os.makedirs(extract_to, exist_ok=True)\n", + " with zipfile.ZipFile(zip_path, 'r') as zip_ref:\n", + " zip_ref.extractall(extract_to)\n", + " os.remove(zip_path)\n", + "\n", + "def copy_to_tmp(name):\n", + " shutil.rmtree(\"./builtin/tmp\", ignore_errors=True)\n", + " path2zip = \"./builtin/src/src.zip\"\n", + "\n", + " prefixes = [\n", + " f\"src/{name}\"\n", + " ]\n", + "\n", + " with ZipFile(path2zip) as archive:\n", + " for prefix in prefixes:\n", + " matched_files = [file for file in archive.namelist() if file.startswith(prefix)]\n", + " if matched_files:\n", + " for file in matched_files:\n", + " archive.extract(file, \"./builtin/tmp\")\n", + " return f\"./builtin/tmp/{prefix}\" # Return only the first matching prefix\n", + "\n", + " return None # Nothing found\n", + "\n", + "# ✅ Combine all folders into one zip\n", + "download_folders_as_zip(repo_owner, repo_name, output_zip = \"./builtin/src/src.zip\", branch = branch, folders_to_extract= [f\"{folder_prefix}/src\"] , remove_folder_prefix = f\"{folder_prefix}\")\n", + "download_folders_as_zip(repo_owner, repo_name, output_zip = \"./builtin/config/config.zip\", branch = branch, folders_to_extract= [f\"{folder_prefix}/config\"] , remove_folder_prefix = f\"{folder_prefix}\")\n", + "# ✅ Uncompress everything into ./builtin\n", + "uncompress_zip_to_folder(zip_path = \"./builtin/config/config.zip\", extract_to= \"./builtin\")\n", + "\n", + "mapping_table=[]\n", + "tasks=[]\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "e22cc332-d939-4def-8971-84f762931ce2" + }, + { + "cell_type": "markdown", + "source": [ + "# CLI Login" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "8a5c3526-9a63-430c-90e4-00ece0aebfa5" + }, + { + "cell_type": "code", + "source": [ + "# Set environment parameters for Fabric CLI\n", + "token = notebookutils.credentials.getToken('pbi')\n", + "os.environ['FAB_TOKEN'] = token\n", + "os.environ['FAB_TOKEN_ONELAKE'] = token" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "f00e2fa8-e10d-45a9-82a7-266feaedc4ba" + }, + { + "cell_type": "markdown", + "source": [ + "# Deployment functions" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "34b39ba9-62fb-421b-be07-2607f79fccdb" + }, + { + "cell_type": "markdown", + "source": [ + "## Load configuration\n" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "33eb5fa4-5bf7-408b-9630-1533ba0d031a" + }, + { + "cell_type": "code", + "source": [ + "base_path = './builtin/'\n", + "config_path = os.path.join(base_path, 'config/item_config.yaml')\n", + "\n", + "with open(config_path, 'r') as file:\n", + " config = yaml.safe_load(file)\n", + "\n", + "deploy_order_path = os.path.join(base_path, 'config/item_initial_setup.json')\n", + "with open(deploy_order_path, 'r') as file:\n", + " item_initial_setup =json.load(file)\n", + "\n", + "deploy_order_path = os.path.join(base_path, 'config/data_deployment.json')\n", + "with open(deploy_order_path, 'r') as file:\n", + " data_deployment =json.load(file)\n", + "\n", + "deploy_order_path = os.path.join(base_path, 'config/item_deployment_code_business_domain.json')\n", + "with open(deploy_order_path, 'r') as file:\n", + " item_deployment_code_business_domain =json.load(file)\n", + "\n", + "deploy_order_path = os.path.join(base_path, 'config/item_deployment_data_business_domain.json')\n", + "with open(deploy_order_path, 'r') as file:\n", + " item_deployment_data_business_domain =json.load(file)\n", + "\n", + "deploy_icons_path = os.path.join(base_path, 'config/fabric_icons.xml')\n", + "\n", + "# Parse the XML file\n", + "tree = ET.parse(deploy_icons_path)\n", + "root = tree.getroot()\n", + "\n", + "# Create a dictionary to store icon name and base64\n", + "fabric_icons_fmd = {}\n", + "for item in root.findall('icon'):\n", + " name = item.find('name').text if item.find('name') is not None else \"No name\"\n", + " base64_str = item.find('base64').text if item.find('base64') is not None else \"\"\n", + " fabric_icons_fmd[name] = base64_str\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "jupyter": { + "source_hidden": false + }, + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "937feb4e-a67f-486c-ad43-0ae628b71e34" + }, + { + "cell_type": "markdown", + "source": [ + "# Deployment" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "3143250a-9c75-4f6a-9f44-78dde9852195" + }, + { + "cell_type": "markdown", + "source": [ + "## Load latest version NB_UTILITIES_SETUP_FMD" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "e38a1ab7-998e-4959-86fd-8e82c0894076" + }, + { + "cell_type": "code", + "source": [ + "def run_fab_command(command, capture_output=False, silently_continue=False, raw_output=False):\n", + " \"\"\"\n", + " Executes a Fabric CLI command with optional output capture and error handling.\n", + " \"\"\"\n", + " result = subprocess.run([\"fab\", \"-c\", command], capture_output=capture_output, text=True)\n", + " if not silently_continue and (result.returncode > 0 or result.stderr):\n", + " raise Exception(f\"Error running fab command. exit_code: '{result.returncode}'; stderr: '{result}'\")\n", + " if capture_output:\n", + " return result if raw_output else result.stdout.strip()\n", + " return None" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "1ccfaae6-01d5-4ae2-a1bf-9d3fca3b240b" + }, + { + "cell_type": "code", + "source": [ + "workspace_id = notebookutils.runtime.context.get('currentWorkspaceId')\n", + "result=run_fab_command(\"api -X get workspaces/\"+workspace_id, capture_output=True, silently_continue=False)\n", + "workspace_name=json.loads(result)[\"text\"][\"displayName\"]\n", + "\n", + "for it in item_initial_setup:\n", + " name = it[\"name\"]\n", + " type = it[\"type\"]\n", + " tmp_path = copy_to_tmp(name)\n", + "\n", + " cli_parameter = ''\n", + "\n", + " if \"Notebook\" in name:\n", + " cli_parameter += \" --format .py\"\n", + " result = run_fab_command(f\"import {workspace_name}.Workspace/{name} -i {tmp_path} -f {cli_parameter}\",capture_output=False, silently_continue=True)\n" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "1ebe7e00-f8dd-44bb-80ee-0d4e68222b10" + }, + { + "cell_type": "code", + "source": [ + "%run NB_UTILITIES_SETUP_FMD" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "18f2fd5f-00cd-4892-ae2a-588def93869d" + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bd-variable-parameters", + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "outputs": [], + "source": [ + "# VAR_GOLD_SHORTCUTS_FMD declareert zes variabelen in\n", + "# config/item_deployment_code_business_domain.json, en update_variable_library zoekt\n", + "# elke bron op in variable_parameters. Zonder deze regels faalt het deployen van die\n", + "# library met een KeyError die door deploy_item wordt opgevangen en alleen als een\n", + "# rood kruisje voorbijkomt - de library wordt dan nooit aangemaakt.\n", + "#\n", + "# De vier ID's krijgen hier een lege waarde: ze zijn pas bekend als de workspaces en\n", + "# lakehouses bestaan. De cel \"Auto-fill VAR_GOLD_SHORTCUTS_FMD\" zoekt ze verderop op\n", + "# en schrijft ze in de gedeployde library.\n", + "variable_parameters.update({\n", + " \"SourceWorkspaceId\": \"\",\n", + " \"SourceLakehouseId\": \"\",\n", + " \"SourceSchema\": SourceSchema,\n", + " \"Shortcut_TargetSchema\": Shortcut_TargetSchema,\n", + " \"Shortcut_TargetWorkspaceId\": \"\",\n", + " \"Shortcut_TargetLakehouseId\": \"\",\n", + "})\n" + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Integration Domain" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "8a1d9b38-da94-42a7-a561-99bc7d2d86ac" + }, + { + "cell_type": "markdown", + "source": [ + "### Get Fabric database (Configuration)" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "33263830-e547-48a7-bc11-9ece4c56d6da" + }, + { + "cell_type": "code", + "source": [ + "for deployment_item in data_deployment:\n", + " if deployment_item['type'] in ('SQLDatabase'):\n", + " name = configuration['DatabaseName']\n", + " type = deployment_item[\"type\"]\n", + " name=name+'.'+type\n", + " server, database_name=deploy_item(workspace_name=configuration['workspace']['name'],name=name,mapping_table=mapping_table, environment_name='config',tasks= tasks, lakehouse_schema_enabled=lakehouse_schema_enabled,it=deployment_item)\n", + "\n", + "##!NOTE: in Case of an error check if you're not using a trial capacitiy or the capacity you assigned is paused" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "59c4ddf9-cb4b-4849-9749-053d7aeb7a87" + }, + { + "cell_type": "markdown", + "source": [ + "## Business Domain" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "78cd9671-cdb3-476d-a447-7181f76af32c" + }, + { + "cell_type": "markdown", + "source": [ + "### Create workspaces(Business Domains)" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "97127361-afea-438f-9569-af3a6c0b981f" + }, + { + "cell_type": "code", + "source": [ + "for business_domain_name in business_domain_names:\n", + " if create_domains:\n", + " create_fabric_domain(business_domain_name)\n", + "\n", + " for business_domain in business_domain_deployment:\n", + " print(f\"--------------------------\")\n", + " print(f\"Updating Workspace: {business_domain['environment_name']}\")\n", + " deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['code'],workspace_name=business_domain_name + ' CODE ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_code\"], mapping_table=mapping_table, tasks=tasks)\n", + " deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['data'],workspace_name=business_domain_name + ' DATA ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_data\"], mapping_table=mapping_table, tasks=tasks)\n", + " deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['reporting'],workspace_name=business_domain_name + ' REPORTING ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_reporting\"], mapping_table=mapping_table, tasks=tasks)\n", + " deploy_workspaces(business_domain_name,workspace=business_domain['workspaces']['semantic'],workspace_name=business_domain_name + ' SEMANTIC ('+business_domain['environment_short']+')'+framework_post_fix, environment_name=business_domain['environment_name'], old_id=config[\"workspaces\"][\"workspace_business_domain_reporting\"], mapping_table=mapping_table, tasks=tasks)" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "330d211a-1b1e-44b2-b989-93e751d5612c" + }, + { + "cell_type": "markdown", + "source": [ + "### Create Lakehouses(Business Domains)" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "532de2ef-cc3a-40a1-8bfd-436c09ec1d4d" + }, + { + "cell_type": "code", + "source": [ + "for workspace_business_domain in business_domain_deployment:\n", + " for business_domain in business_domain_names:\n", + " for workspace in [workspace_business_domain['workspaces']['data']]:\n", + " exclude = []\n", + " for it in item_deployment_data_business_domain:\n", + " new_id = None \n", + " name = it[\"name\"]\n", + " type = it[\"type\"]\n", + " if name in exclude:\n", + " continue\n", + " deploy_item(business_domain + ' DATA ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "7c2a7033-c33f-4802-ba74-a93ac10fd798" + }, + { + "cell_type": "markdown", + "source": [ + "### Create Items(Business Domains)" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "b0faae07-cb6e-4ae4-a3ef-37fe3db24dcd" + }, + { + "cell_type": "code", + "source": [ + "for workspace_business_domain in business_domain_deployment:\n", + " for business_domain_name in business_domain_names:\n", + " for workspace in [workspace_business_domain['workspaces']['code']]:\n", + " for it in item_deployment_code_business_domain:\n", + " new_id = None\n", + " if it['type'] in ('VariableLibrary'):\n", + " name = it[\"name\"]\n", + " type = it[\"type\"]\n", + " deploy_item(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "8d8220fc-6410-4ed7-a8f5-3936da7f5557" + }, + { + "cell_type": "markdown", + "id": "adaae148", + "source": "### Auto-fill VAR_GOLD_SHORTCUTS_FMD (Gold <-> Silver workspace/lakehouse IDs)", + "metadata": {} + }, + { + "cell_type": "code", + "id": "123a5aa6", + "source": [ + "##### Resolves the Gold lakehouse (this business domain) and Silver lakehouse (integration domain) IDs #####\n", + "##### per stage, and writes them into VAR_GOLD_SHORTCUTS_FMD so NB_CREATE_SHORTCUTS doesn't need manual IDs. #####\n", + "##### SourceSchema / Shortcut_TargetSchema still come from the \"Shortcut settings\" cell above (naming convention, not per-domain). #####\n", + "integration_workspace_prefix = require('naming.domain_name')\n", + "gold_lakehouse_name = \"LH_GOLD_LAYER\"\n", + "silver_lakehouse_name = \"LH_SILVER_LAYER\"\n", + "\n", + "for business_domain_name in business_domain_names:\n", + " for bd in business_domain_deployment:\n", + " short = bd['environment_short']\n", + " gold_workspace_name = f\"{business_domain_name} DATA ({short}){framework_post_fix}\"\n", + " silver_workspace_name = f\"{integration_workspace_prefix} DATA ({short}){framework_post_fix}\"\n", + " code_workspace_name = f\"{business_domain_name} CODE ({short}){framework_post_fix}\"\n", + "\n", + " gold_workspace_id = get_workspace_id_by_name(gold_workspace_name)\n", + " gold_lakehouse_id = get_item_id(gold_workspace_name, f\"{gold_lakehouse_name}.Lakehouse\", \"id\")\n", + " silver_workspace_id = get_workspace_id_by_name(silver_workspace_name)\n", + " silver_lakehouse_id = get_item_id(silver_workspace_name, f\"{silver_lakehouse_name}.Lakehouse\", \"id\")\n", + "\n", + " if not (gold_workspace_id and gold_lakehouse_id and silver_workspace_id and silver_lakehouse_id):\n", + " print(f\" - Could not resolve Gold/Silver lakehouse IDs for '{business_domain_name}' ({short}), skip\")\n", + " continue\n", + "\n", + " set_variable_library_values(code_workspace_name, \"VAR_GOLD_SHORTCUTS_FMD\", {\n", + " \"SourceWorkspaceId\": gold_workspace_id,\n", + " \"SourceLakehouseId\": gold_lakehouse_id,\n", + " \"SourceSchema\": SourceSchema,\n", + " \"Shortcut_TargetWorkspaceId\": silver_workspace_id,\n", + " \"Shortcut_TargetLakehouseId\": silver_lakehouse_id,\n", + " \"Shortcut_TargetSchema\": Shortcut_TargetSchema,\n", + " })" + ], + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "for workspace_business_domain in business_domain_deployment:\n", + " for business_domain_name in business_domain_names:\n", + " for workspace in [workspace_business_domain['workspaces']['code']]:\n", + " for it in item_deployment_code_business_domain:\n", + " new_id = None\n", + " if it['type'] not in ('VariableLibrary'):\n", + " name = it[\"name\"]\n", + " type = it[\"type\"]\n", + " deploy_item(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix, name,mapping_table, workspace_business_domain['environment_name'], tasks, lakehouse_schema_enabled,it)" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "c321c689-995e-4943-ad05-db9db2d885f8" + }, + { + "cell_type": "markdown", + "id": "d9deda98", + "source": "### Create Deployment Pipelines (Business Domains)", + "metadata": {} + }, + { + "cell_type": "code", + "id": "e117937f", + "source": "for business_domain_name in business_domain_names:\n for group in [\"code\", \"data\", \"reporting\", \"semantic\"]:\n pipeline_name = f\"{business_domain_name} {group.upper()}\"\n stage_workspace_names = [\n (bd[\"environment_name\"].capitalize(),\n business_domain_name + \" \" + group.upper() + \" (\" + bd[\"environment_short\"] + \")\" + framework_post_fix)\n for bd in business_domain_deployment\n ]\n deploy_deployment_pipeline(pipeline_name, stage_workspace_names)", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "5d591c63", + "source": "### Wire Business Domains into PL_FMD_ORCHESTRATION_TEMPLATE", + "metadata": {} + }, + { + "cell_type": "code", + "id": "2d0a6f9a", + "source": "##### Adds an InvokePipeline activity to the integration domain's orchestrator per business domain and stage. #####\n##### Skips a domain/stage if it's already wired in (idempotent). #####\nfor business_domain_name in business_domain_names:\n for bd in business_domain_deployment:\n orchestration_workspace_name = f\"{integration_workspace_prefix} CODE ({bd['environment_short']}){framework_post_fix}\"\n gold_workspace_name = f\"{business_domain_name} CODE ({bd['environment_short']}){framework_post_fix}\"\n activity_name = f\"PL_{business_domain_name}_LOAD_GOLD\"\n wire_domain_into_orchestration(orchestration_workspace_name, gold_workspace_name, activity_name)", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## Workspace Icons" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "5d91a5c1-bd07-46b6-a88f-ba7fcb305c4f" + }, + { + "cell_type": "code", + "source": [ + "if assign_icons: \n", + " seen = set()\n", + " workspaces = []\n", + " # Get cluster URL for use in metadata endpoints\n", + " cluster_base_url = get_cluster_url()\n", + "\n", + " for item in mapping_table:\n", + " if item['ItemType'] == 'Workspace':\n", + " key = (item['Description'], item['new_id'])\n", + " if key not in seen:\n", + " seen.add(key)\n", + " workspaces.append({'displayName': item['Description'], 'id': item['new_id']})\n", + " fabric_icons = fabric_icons_fmd \n", + "\n", + " for workspace in workspaces:\n", + " display_name = workspace['displayName'].lower()\n", + " \n", + " # Assign icon\n", + " for icon_key, icon_value in workspace_icon_def['icons'].items():\n", + " if icon_key in display_name:\n", + " workspace[\"icon\"] = icon_value\n", + " workspace_icon = fabric_icons.get(icon_value)\n", + " break\n", + " else:\n", + " workspace[\"icon\"] = None\n", + " workspace_icon = None\n", + "\n", + " workspace[\"icon_base64img\"] = workspace_icon" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "662a019b-f9dc-48f5-9fad-14ca06c30193" + }, + { + "cell_type": "code", + "source": [ + "if assign_icons:\n", + " # Dry run - Display pre and post icons based on specified workspace filters and workspace icon definition. Will NOT update any icons!\n", + " display_workspace_icons(workspaces)\n", + " sleep(2)" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "f0bdcf88-cac2-4546-8c7f-cbfdb390461e" + }, + { + "cell_type": "markdown", + "source": [ + "### Deploy Workspace Icons" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "ca909ec2-916a-4400-82bc-acf434bc4550" + }, + { + "cell_type": "code", + "source": [ + "if assign_icons:\n", + " for workspace in workspaces:\n", + " set_workspace_icon(workspace.get('id'), workspace.get('icon_base64img'))" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "806f42d6-e74c-4909-8ac0-ab14d8f68e0e" + }, + { + "cell_type": "markdown", + "source": [ + "## Create SQL deployment Manifest" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "4cc252ff-6f50-41f9-9c6d-165300837bb6" + }, + { + "cell_type": "markdown", + "source": [ + "### Add Workspaces to Fabric Database" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "8ed267d7-4aca-445c-9189-243307a972e6" + }, + { + "cell_type": "code", + "source": [ + "#add all created workspace to database\n", + "custom_sql_deployment = {\"queries_stored_procedures\": []}\n", + "unique_items = {}\n", + "for item in mapping_table:\n", + " if item.get(\"ItemType\") == \"Workspace\":\n", + " unique_items[item[\"new_id\"]] = item[\"Description\"]\n", + "\n", + "# Convert to list of tuples or dicts\n", + "workspaces = [{\"Description\": desc, \"new_id\": nid} for nid, desc in unique_items.items()]\n", + "\n", + "for workspace in workspaces:\n", + " print(f'EXEC [integration].[sp_UpsertWorkspace](@WorkspaceId = \"{workspace[\"new_id\"]}\" ,@Name = \"{workspace[\"Description\"]}\")')\n", + " custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertWorkspace] @WorkspaceId = \"{workspace[\"new_id\"]}\", @Name = \"{workspace[\"Description\"]}\"')" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "5ee6f1cd-8aaa-4c94-be39-6721312e6cde" + }, + { + "cell_type": "code", + "source": [ + "for workspace_business_domain in business_domain_deployment:\n", + " for business_domain_name in business_domain_names:\n", + " for workspace in [workspace_business_domain['workspaces']['code']]:\n", + " workspace_id = get_workspace_id_by_name(business_domain_name + ' CODE ('+ workspace_business_domain['environment_short']+')'+framework_post_fix)\n", + " result = run_fab_command(f\"api -X get workspaces/{workspace_id}/items\", capture_output=True, silently_continue=True)\n", + " existing_items = json.loads(result)['text']\n", + " for item in existing_items.get('value', []):\n", + " if item['type'] == 'DataPipeline':\n", + " print(f'EXEC [integration].[sp_UpsertPipeline] @PipelineId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n", + " custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertPipeline] @PipelineId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "5a9c1eb2-b4a4-448b-82a8-fdeef790d1e8" + }, + { + "cell_type": "markdown", + "source": [ + "### Add Lakehouses to Fabric Database" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "4867745a-f0e7-4a9a-b326-5609b1b5bb3a" + }, + { + "cell_type": "code", + "source": [ + "for workspace_business_domain in business_domain_deployment:\n", + " for business_domain_name in business_domain_names:\n", + " for workspace in [workspace_business_domain['workspaces']['data']]:\n", + " workspace_id = get_workspace_id_by_name(business_domain_name + ' DATA ('+ workspace_business_domain['environment_short']+')'+framework_post_fix)\n", + " result = run_fab_command(f\"api -X get workspaces/{workspace_id}/items\", capture_output=True, silently_continue=True)\n", + " existing_items = json.loads(result)['text']\n", + " for item in existing_items.get('value', []):\n", + " if item['type'] == 'Lakehouse':\n", + " print(f'EXEC [integration].[sp_UpsertLakehouse] @LakehouseId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n", + " custom_sql_deployment[\"queries_stored_procedures\"].append(f'EXEC [integration].[sp_UpsertLakehouse] @LakehouseId = \"{item[\"id\"]}\", @WorkspaceId = \"{workspace_id}\" ,@Name = \"{item[\"displayName\"]}\"')\n", + " " + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "3144899e-74f6-4eb0-a203-525437d8c127" + }, + { + "cell_type": "markdown", + "source": [ + "## Deploy SQL Code" + ], + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + }, + "nteract": { + "transient": { + "deleting": false + } + } + }, + "id": "31e3bfba-b56e-47c7-a1e5-c764c2c3e622" + }, + { + "cell_type": "code", + "source": [ + "#Make sure out token is still valid before we continou, sometime it can take a bit longer that's we will check it again\n", + "token = notebookutils.credentials.getToken('pbi')\n", + "os.environ['FAB_TOKEN'] = token\n", + "os.environ['FAB_TOKEN_ONELAKE'] = token" + ], + "outputs": [], + "execution_count": null, + "metadata": { + "microsoft": { + "language": "python", + "language_group": "jupyter_python" + } + }, + "id": "6ac7b58e-1240-4110-a4a1-c7c767bad1ee" + }, + { + "cell_type": "code", + "source": [ + "for deployment_item in data_deployment:\n", + " if deployment_item['type'] in ('SQLDatabase'):\n", + " name = configuration['DatabaseName']\n", + " type = deployment_item[\"type\"]\n", + " name=name+'.'+type\n", + " if not server: #it was already set, just in case\n", + " server=get_item_id(configuration['workspace']['name'], name, 'properties.serverFqdn')\n", + " if not database_name: \n", + " database_name=get_item_id(configuration['workspace']['name'], name, 'properties.databaseName')\n", + "\n", + "try:\n", + " i = 0\n", + "\n", + " token = notebookutils.credentials.getToken('pbi').encode('utf-16-le')\n", + " token_struct = struct.pack(f' principal-dicts. Nul-ID's worden weggelaten: die\n", + " betekenen 'nog niet bekend', en fab acl set loopt erop stuk.\"\"\"\n", + " out = []\n", + " for principal_id, principal_type in specs:\n", + " if principal_id and str(principal_id).replace(\"-\", \"\").strip(\"0\"):\n", + " out.append({\"id\": str(principal_id), \"type\": principal_type})\n", + " return out\n", + "\n", + "\n", + "def roles(*specs):\n", + " \"\"\"[(id, type, role), ...] -> rollijst voor deploy_workspaces.\"\"\"\n", + " out = []\n", + " for principal_id, principal_type, role in specs:\n", + " for principal in principals((principal_id, principal_type)):\n", + " out.append({\"principal\": principal, \"role\": role})\n", + " return out\n", + "\n", + "\n", + "# De bootstrap-cel en het manifest moeten dezelfde repo aanwijzen; anders komen\n", + "# src/ en config/ straks ergens anders vandaan dan het manifest beschrijft.\n", + "for _key, _bootstrap_value in (\n", + " (\"repository.owner\", repo_owner),\n", + " (\"repository.name\", repo_name),\n", + " (\"repository.branch\", branch),\n", + "):\n", + " if str(require(_key)) != str(_bootstrap_value):\n", + " raise RuntimeError(\n", + " f\"Bootstrap en manifest spreken elkaar tegen: {_key} is \"\n", + " f\"'{require(_key)}' in manifest.yaml maar '{_bootstrap_value}' in de \"\n", + " f\"bootstrap-cel hierboven.\"\n", + " )\n", + "\n", + "print(f\"Manifest geladen van {MANIFEST_URL}\")\n" + ] + }, { "cell_type": "code", "execution_count": null, @@ -67,11 +210,13 @@ "source": [ "assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons\n", "load_demo_data= True # Set to True if you want to load the demo data, otherwise set to False\n", - "lakehouse_schema_enabled = True # Set to True if you want to use the lakehouse schema, otherwise set to False\n", "\n", "driver = '{ODBC Driver 18 for SQL Server}' # Change this if you use a different driver\n", "overwrite_variable_library=True # By default the Library is overwritten, change this to \"False\" if you have custom changes\n", - "spark_version='2.0'" + "\n", + "# Uit het manifest - pas deze aan in manifest.yaml, niet hier.\n", + "lakehouse_schema_enabled = require('framework.lakehouse_schema_enabled')\n", + "spark_version = str(require('spark.runtime_version'))\n" ] }, { @@ -141,9 +286,10 @@ "source": [ "reassign_capacity= True # If set to False existing assigned capacities to workspaces will no be overwritten\n", "\n", - "capacity_name_dvlm = '' # Which capacity will be used for these workspaces in development\n", - "capacity_name_prod = '' # Which capacity will be used for these workspaces in production\n", - "capacity_name_config = '' # Which capacity will be used for this workspace for the Metadata Database" + "# De capaciteit per omgeving komt uit manifest.environments[].capacity en wordt\n", + "# in de cel \"Workspace configuration\" toegepast. De CONFIG-workspace staat daar\n", + "# los van: die bestaat een keer per klant, niet een keer per omgeving.\n", + "capacity_name_config = require('config_workspace.capacity')\n" ] }, { @@ -176,34 +322,36 @@ }, "outputs": [], "source": [ - "framework_post_fix= '' # post fix to be added at the end of workspace for example INTEGRATION CODE(D) FMD\n", + "framework_post_fix= require('naming.framework_post_fix') # post fix to be added at the end of workspace for example INTEGRATION CODE(D) FMD\n", "if framework_post_fix != '':\n", " framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility\n", "\n", "##Domains\n", "create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed\n", "\n", - "domain_name='INTEGRATION' # Main Domain for Integration for example INTEGRATION CODE(D) \n", + "domain_name= require('naming.domain_name') # Main Domain for Integration for example INTEGRATION CODE(D)\n", "\n", - "domain_contributor_role = {\"type\": \"Contributors\",\"principals\": [{\"id\": \"00000000-0000-0000-0000-000000000000\",\"type\": \"Group\"} ]} # Which group(Object ID) can add or remove workspaces to this domain\n", + "# Groep die workspaces aan dit domein mag toevoegen of verwijderen.\n", + "domain_contributor_role = {\"type\": \"Contributors\", \"principals\": principals(\n", + " (require('security.contributor_group_id'), \"Group\"),\n", + ")}\n", "\n", - "##Connections\n", + "##Connections - naamconventie van het framework, niet klantspecifiek.\n", "connection_fabric_datapipelines_name='CON_FMD_FABRIC_PIPELINES'\n", "connection_fabric_notebooks_name='CON_FMD_FABRIC_NOTEBOOKS'\n", "connection_fabric_database_name='CON_FMD_FABRIC_SQL'\n", "connection_fabric_adf_name='CON_FMD_ADF_PIPELINES'\n", "\n", - "##Service principal used to create CON_FMD_FABRIC_SQL automatically.\n", - "##Leave these empty to keep the manual step. FabricSql accepts only OAuth2-family\n", - "##credentials, so a WorkspaceIdentity cannot be used here, but a service principal can.\n", - "##The principal needs access to the CONFIG workspace that holds the database.\n", - "key_vault=''\n", - "fabric_sql_sp_tenant_id = 'tenantid'\n", - "fabric_sql_sp_client_id = 'sp-fmd-fabric-pipelines-app-id'\n", - "fabric_sql_sp_secret = 'sp-fmd-fabric-pipelines-secret'\n", + "##Key Vault - dit zijn de NAMEN van de secrets; de waarden worden hieronder opgehaald.\n", + "key_vault= require('key_vault.name')\n", + "fabric_sql_sp_tenant_id = require('key_vault.secrets.sql_sp_tenant_id')\n", + "fabric_sql_sp_client_id = require('key_vault.secrets.sql_sp_client_id')\n", + "fabric_sql_sp_secret = require('key_vault.secrets.sql_sp_secret')\n", "\n", - "connection_role = {\"role\": \"owner\",\"principals\": [{\"id\": \"00000000-0000-0000-0000-000000000000\",\"type\": \"Group\"} ]} # Which group(Object ID) can add or remove workspaces to this domain\n", - "\n" + "# Groep die eigenaar wordt van de aangemaakte connections.\n", + "connection_role = {\"role\": \"owner\", \"principals\": principals(\n", + " (require('security.admin_group_id'), \"Group\"),\n", + ")}\n" ] }, { @@ -242,45 +390,6 @@ "# Configuration\n" ] }, - { - "cell_type": "markdown", - "id": "a8e67871-6536-43c0-ac2e-21ff8571d022", - "metadata": { - "microsoft": { - "language": "python", - "language_group": "jupyter_python" - }, - "nteract": { - "transient": { - "deleting": false - } - } - }, - "source": [ - "## Repo Configuration" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e3945911-68db-4961-99db-5020ac37e1b7", - "metadata": { - "microsoft": { - "language": "python", - "language_group": "jupyter_python" - } - }, - "outputs": [], - "source": [ - "#FMD Framework code\n", - "##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE ####\n", - "repo_owner = \"edkreuk\" # Owner of the repository\n", - "repo_name = \"FMD_FRAMEWORK\" # Name of the repository\n", - "branch = \"main\" #\"main\" is default \n", - "folder_prefix = \"\"\n", - "###################################################\n" - ] - }, { "cell_type": "markdown", "id": "0443c89a-6aec-46c1-9456-37e8f4ab93aa", @@ -311,45 +420,18 @@ }, "outputs": [], "source": [ - "workspace_roles_code = [ # Keep emtpy [] if you only want to assign this to your personal account\n", - " {\n", - " \"principal\": {\n", - " \"id\": '00000000-0000-0000-0000-000000000000',\n", - " \"type\": \"Group\"\n", - " },\n", - " \"role\": \"admin\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n", - " },\n", - " {\n", - " \"principal\": {\n", - " \"id\": '00000000-0000-0000-0000-000000000000',\n", - " \"type\": \"ServicePrincipal\"\n", - " },\n", - " \"role\": \"contributor\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n", - " }\n", - " ]\n", - "workspace_roles_data = [ # Keep emtpy [] if you only want to assign this to your personal account\n", - " {\n", - " \"principal\": {\n", - " \"id\": '00000000-0000-0000-0000-000000000000',\n", - " \"type\": \"Group\"\n", - " },\n", - " \"role\": \"admin\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n", - " }, \n", - " {\n", - " \"principal\": {\n", - " \"id\": '00000000-0000-0000-0000-000000000000',\n", - " \"type\": \"Group\"\n", - " },\n", - " \"role\": \"contributor\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n", - " },\n", - " {\n", - " \"principal\": {\n", - " \"id\": '00000000-0000-0000-0000-000000000000',\n", - " \"type\": \"ServicePrincipal\"\n", - " },\n", - " \"role\": \"contributor\" #(choose from 'admin', 'member', 'contributor', 'viewer')\n", - " }\n", - " ]\n" + "# Rollen komen uit manifest.security. Een ID dat nog op nullen staat betekent\n", + "# \"nog niet bekend\" en wordt overgeslagen; fab acl set loopt daar anders op stuk.\n", + "workspace_roles_code = roles(\n", + " (require('security.admin_group_id'), 'Group', 'admin'),\n", + " (optional('security.service_principal_id'), 'ServicePrincipal', 'contributor'),\n", + ")\n", + "\n", + "workspace_roles_data = roles(\n", + " (require('security.admin_group_id'), 'Group', 'admin'),\n", + " (require('security.contributor_group_id'), 'Group', 'contributor'),\n", + " (optional('security.service_principal_id'), 'ServicePrincipal', 'contributor'),\n", + ")\n" ] }, { @@ -382,41 +464,27 @@ }, "outputs": [], "source": [ - "\n", - "##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE, FE ADDING NEW ENVIRONMENTS ####\n", + "##### Opgebouwd uit manifest.environments - voeg omgevingen daar toe, niet hier ####\n", "environments = [\n", - " {\n", - " 'environment_name' : 'development', # Name of target environment\n", - " 'workspaces': {\n", - " 'data' : {\n", - " 'name' : domain_name + ' DATA (D)' + framework_post_fix, # Name of target code workspace for development\n", - " 'roles' : workspace_roles_data, # Roles to assign to the workspace\n", - " 'capacity_name' : capacity_name_dvlm # Name of target data workspace for development\n", - " },\n", - " 'code' : {\n", - " 'name' : domain_name + ' CODE (D)' + framework_post_fix, # Name of target data workspace for development\n", - " 'roles' : workspace_roles_code, # Roles to assign to the workspace\n", - " 'capacity_name' : capacity_name_dvlm # Name of target code workspace for development\n", - " }\n", - " }\n", - " },\n", - " {\n", - " 'environment_name' : 'production', # Name of target environment\n", - " 'workspaces': {\n", - " 'data' : {\n", - " 'name' : domain_name + ' DATA (P)' + framework_post_fix, # Name of target data workspace for production\n", - " 'roles' : workspace_roles_data, # Roles to assign to the workspace\n", - " 'capacity_name' : capacity_name_prod # Name of target data workspace for production \n", - " },\n", - " 'code' : {\n", - " 'name' : domain_name + ' CODE (P)' + framework_post_fix, # Name of target code workspace for production\n", - " 'roles' : workspace_roles_code, # Roles to assign to the workspace\n", - " 'capacity_name' : capacity_name_prod # Name of target code workspace for production\n", - " } \n", - " }\n", - " }\n", - " ]\n", - "###################################################" + " {\n", + " 'environment_name': env['name'].lower(), # Sleutel in mapping_table; lowercase zoals deploy_item verwacht\n", + " 'environment_short': env['short'], # D / T / A / P\n", + " 'workspaces': {\n", + " 'data': {\n", + " 'name': f\"{domain_name} DATA ({env['short']}){framework_post_fix}\",\n", + " 'roles': workspace_roles_data,\n", + " 'capacity_name': env['capacity'],\n", + " },\n", + " 'code': {\n", + " 'name': f\"{domain_name} CODE ({env['short']}){framework_post_fix}\",\n", + " 'roles': workspace_roles_code,\n", + " 'capacity_name': env['capacity'],\n", + " },\n", + " },\n", + " }\n", + " for env in require('environments')\n", + "]\n", + "###################################################\n" ] }, { @@ -984,6 +1052,27 @@ " assign_identity_role_to_different_workspace(environment['workspaces']['code']['name'],configuration['workspace']['name'] ) ##Assign workspace identity to Configuration workspace while Identity is used to connect\n" ] }, + { + "cell_type": "markdown", + "id": "d6931af9", + "source": "### Create Deployment Pipelines (Code and Data)", + "metadata": {} + }, + { + "cell_type": "code", + "id": "8f569535", + "source": [ + "stage_names = [env['name'] for env in require('environments')]\n", + "\n", + "for group in [\"code\", \"data\"]:\n", + " pipeline_name = f\"{domain_name} {group.upper()}\"\n", + " stage_workspace_names = list(zip(stage_names, [e[\"workspaces\"][group][\"name\"] for e in environments]))\n", + " deploy_deployment_pipeline(pipeline_name, stage_workspace_names)\n" + ], + "metadata": {}, + "execution_count": null, + "outputs": [] + }, { "cell_type": "markdown", "id": "20cb4578-f42d-447a-9407-03cb328c1eb0", @@ -1118,7 +1207,10 @@ " new_id = None\n", " name = it[\"name\"]\n", " type = it[\"type\"]\n", - " deploy_item(configuration['workspace']['name'],name,mapping_table,environment['environment_name'], tasks, lakehouse_schema_enabled,it)" + " # ENV_FMD lands in the CONFIG workspace, once, shared by every environment.\n", + " # 'config' matches how that workspace itself is deployed; this used to pass\n", + " # environment['environment_name'], left over from an earlier loop.\n", + " deploy_item(configuration['workspace']['name'],name,mapping_table,'config', tasks, lakehouse_schema_enabled,it)" ] }, { @@ -1150,18 +1242,7 @@ } }, "outputs": [], - "source": [ - "for environment in environments:\n", - " print(f\"\\n--------------------------\")\n", - " print(f\"Processing: {environment['environment_name']}\")\n", - " for workspace in [environment['workspaces']['code']]:\n", - " for it in item_deployment:\n", - " if it['type'] in ('Notebook','VariableLibrary','Environment'):\n", - "\n", - " name = it[\"name\"]\n", - " type = it[\"type\"]\n", - " deploy_item(workspace['name'],name,mapping_table,environment['environment_name'], tasks, lakehouse_schema_enabled,it)" - ] + "source": "for environment in environments:\n print(f\"\\n--------------------------\")\n print(f\"Processing: {environment['environment_name']}\")\n data_workspace_name = environment['workspaces']['data']['name']\n variable_parameters[\"fmd_data_workspace_guid\"] = get_workspace_id_by_name(data_workspace_name)\n variable_parameters[\"fmd_landingzone_lakehouse_guid\"] = get_item_id(data_workspace_name, 'LH_DATA_LANDINGZONE.Lakehouse', 'id')\n for workspace in [environment['workspaces']['code']]:\n for it in item_deployment:\n if it['type'] in ('Notebook','VariableLibrary','Environment'):\n\n name = it[\"name\"]\n type = it[\"type\"]\n deploy_item(workspace['name'],name,mapping_table,environment['environment_name'], tasks, lakehouse_schema_enabled,it)" }, { "cell_type": "code", @@ -1875,4 +1956,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql b/src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql index 41f4fcca..406d184e 100644 --- a/src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql +++ b/src/Config_Database/execution/StoredProcedures/sp_GetBronzelayerEntity.sql @@ -1,6 +1,7 @@ - CREATE PROCEDURE [execution].[sp_GetBronzelayerEntity] - ( @WorkspaceId UNIQUEIDENTIFIER + CREATE PROCEDURE [execution].[sp_GetBronzelayerEntity] + ( @WorkspaceId UNIQUEIDENTIFIER + ,@LoadGroup VARCHAR(50) = '' ) WITH EXECUTE AS CALLER AS @@ -26,7 +27,9 @@ BEGIN ',"DataSourceNamespace" : ' , '"' , LOWER(convert(NVARCHAR(20), [DataSourceNamespace])) , '"' , '}}'),', ') WITHIN GROUP (ORDER BY [EntityId]) ,']') AS NotebookParams - FROM (SELECT TOP 100 PERCENT * FROM [execution].[vw_LoadToBronzeLayer] WHERE SourceWorkspaceId = @WorkspaceId + FROM (SELECT TOP 100 PERCENT * FROM [execution].[vw_LoadToBronzeLayer] + WHERE SourceWorkspaceId = @WorkspaceId + AND (@LoadGroup = '' OR LoadGroup = @LoadGroup) ORDER BY [SourceFileName] ASC) AS [vw_LoadToBronzeLayer] END diff --git a/src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql b/src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql index 742e340f..d5aaa7dd 100644 --- a/src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql +++ b/src/Config_Database/execution/StoredProcedures/sp_GetSilverlayerEntity.sql @@ -1,6 +1,7 @@ CREATE PROCEDURE [execution].[sp_GetSilverlayerEntity] -( @WorkspaceId UNIQUEIDENTIFIER +( @WorkspaceId UNIQUEIDENTIFIER + ,@LoadGroup VARCHAR(50) = '' ) WITH EXECUTE AS CALLER AS @@ -35,7 +36,9 @@ BEGIN ) WITHIN GROUP (ORDER BY EntityId), ']' ) AS NotebookParams - FROM [execution].[vw_LoadToSilverLayer] WHERE SourceWorkspaceId = @WorkspaceId + FROM [execution].[vw_LoadToSilverLayer] + WHERE SourceWorkspaceId = @WorkspaceId + AND (@LoadGroup = '' OR LoadGroup = @LoadGroup) END; GO diff --git a/src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql b/src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql index c88d7dd5..59e5698a 100644 --- a/src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql +++ b/src/Config_Database/execution/Views/vw_LoadSourceToLandingzone.sql @@ -8,6 +8,7 @@ DS.[Name] AS [DataSourceName], DS.[Namespace] AS [DataSourceNamespace], DS.[Type] AS [DataSourceType], + DS.[LoadGroup] AS [LoadGroup], C.[Type] AS [ConnectionType], C.[ConnectionGuid] AS [ConnectionGuid], [SourceSchema], diff --git a/src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql b/src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql index 17876300..8fffe9b3 100644 --- a/src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql +++ b/src/Config_Database/execution/Views/vw_LoadToBronzeLayer.sql @@ -16,8 +16,9 @@ BLH.[Name] AS [TargetLakehouseName], LZH.[Name] AS [SourceLakehouseName], LZE.[IsIncremental], - BLE.[PrimaryKeys], - DS.[Namespace] AS [DataSourceNamespace] + BLE.[PrimaryKeys], + DS.[Namespace] AS [DataSourceNamespace], + DS.[LoadGroup] AS [LoadGroup] FROM [integration].[BronzeLayerEntity] BLE INNER JOIN [integration].[LandingzoneEntity] LZE ON LZE.[LandingzoneEntityId] = BLE.[LandingzoneEntityId] diff --git a/src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql b/src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql index 0aaa13af..55491e96 100644 --- a/src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql +++ b/src/Config_Database/execution/Views/vw_LoadToSilverLayer.sql @@ -25,7 +25,8 @@ , CHAR(10), '' ) -- remove LF , CHAR(9), '' ) -- (optional) remove TABs AS CleansingRules, - DS.[Namespace] AS [DataSourceNamespace] + DS.[Namespace] AS [DataSourceNamespace], + DS.[LoadGroup] AS [LoadGroup] FROM [integration].[SilverLayerEntity] SLE INNER JOIN [integration].[BronzeLayerEntity] BLE ON SLE.BronzeLayerEntityId = BLE.BronzeLayerEntityId diff --git a/src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql b/src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql index 4637c540..1ffdd697 100644 --- a/src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql +++ b/src/Config_Database/integration/StoredProcedures/sp_UpsertDataSource.sql @@ -9,6 +9,7 @@ ,@Namespace VARCHAR(100) ,@Type VARCHAR(30) ,@Description NVARCHAR(200) + ,@LoadGroup VARCHAR(50) = '' ,@IsActive BIT = 1 ) WITH EXECUTE AS CALLER @@ -33,6 +34,7 @@ BEGIN ,[ConnectionId] ,[Type] ,[Description] + ,[LoadGroup] ,[IsActive]) OUTPUT INSERTED.[DataSourceId] INTO @OutputTable VALUES (@Name @@ -40,6 +42,7 @@ BEGIN ,@InternalConnectionId ,@Type ,@Description + ,@LoadGroup ,@IsActive); END ELSE @@ -50,6 +53,7 @@ BEGIN ,[Namespace] = @Namespace ,[Type] = @Type ,[Description] = @Description + ,[LoadGroup] = @LoadGroup ,[IsActive] = @IsActive OUTPUT INSERTED.[DataSourceId] INTO @OutputTable WHERE [DataSourceId] = @DataSourceId; diff --git a/src/Config_Database/integration/Tables/DataSource.sql b/src/Config_Database/integration/Tables/DataSource.sql index d7526d3b..fe642575 100644 --- a/src/Config_Database/integration/Tables/DataSource.sql +++ b/src/Config_Database/integration/Tables/DataSource.sql @@ -5,6 +5,7 @@ CREATE TABLE [integration].[DataSource] ( [Namespace] VARCHAR (100) NOT NULL, [Type] VARCHAR (30) NULL, [Description] NVARCHAR (200) NULL, + [LoadGroup] VARCHAR (50) DEFAULT ('') NOT NULL, [IsActive] BIT DEFAULT ((1)) NOT NULL, CONSTRAINT [PK_integration_DataSource] PRIMARY KEY CLUSTERED ([DataSourceId] ASC), CONSTRAINT [UC_integration_DataSource] UNIQUE NONCLUSTERED ([ConnectionId] ASC, [Name] ASC, [Type] ASC) diff --git a/src/ENV_FMD.Environment/Setting/Sparkcompute.yml b/src/ENV_FMD.Environment/Setting/Sparkcompute.yml index 45f3febf..6aa15398 100644 --- a/src/ENV_FMD.Environment/Setting/Sparkcompute.yml +++ b/src/ENV_FMD.Environment/Setting/Sparkcompute.yml @@ -1,8 +1,13 @@ enable_native_execution_engine: true -driver_cores: 8 -driver_memory: 56g -executor_cores: 8 -executor_memory: 56g +# Conservative default that starts on any capacity; raise it on a bigger one. +# Worst case here is driver(4,28g) + 2 executors(4,28g) = 12 cores / 84g. +# Do not edit these four values here: the setup notebook overwrites them from +# manifest.yaml (spark.default) at deploy time. One Environment item serves every +# environment, so this sizing applies everywhere. +driver_cores: 4 +driver_memory: 28g +executor_cores: 4 +executor_memory: 28g dynamic_executor_allocation: enabled: true min_executors: 1 diff --git a/src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py b/src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py index 8e8b5e22..63ba955c 100644 --- a/src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py +++ b/src/NB_UTILITIES_SETUP_FMD.Notebook/notebook-content.py @@ -275,8 +275,20 @@ def update_variable_library(folder_path, it_variables): variable_file = f"{folder_path}/variables.json" variables_table = [] for it_variable in it_variables: - if it_variable["type"] == "variable": - new_variable = variable_parameters[it_variable["source"]] + if it_variable["type"] != "variable": + raise ValueError( + f"Unsupported variable type '{it_variable['type']}' for " + f"'{it_variable.get('name')}' in the deployment config." + ) + source = it_variable["source"] + if source not in variable_parameters: + raise KeyError( + f"variable_parameters has no '{source}', needed for variable " + f"'{it_variable.get('name')}'. Every source declared in " + f"config/item_deployment*.json must be set before the Variable " + f"Libraries are deployed." + ) + new_variable = variable_parameters[source] variables_table.append( { "name": it_variable.get("name"), @@ -583,6 +595,75 @@ def deploy_workspaces(domain_name,workspace, workspace_name, environment_name, o tasks.append({"task_name": f"Create or Update workspace {workspace_name}","task_duration": int(time() - start),"status": "success" }) +# ------------------------------- +# Manifest applied at deploy time +# ------------------------------- +# Spark sizing and the active value sets differ per environment, while src/ holds +# one copy of each item. Both are therefore applied to the extracted tmp folder +# just before import, from the manifest that the setup notebook already loaded. + +def apply_spark_settings(folder_path): + """ + Rewrites Setting/Sparkcompute.yml from manifest spark.default. Line-based on + purpose: keys that the manifest does not mention (enable_native_execution_engine, + dynamic_executor_allocation, ...) are left exactly as they are, comments and + line endings included. + + No per-environment variant: ENV_FMD.Environment is deployed once, into the + CONFIG workspace, and shared by every environment. Sizing per environment + needs an Environment item per environment first. + """ + settings = dict(require('spark.default')) + settings['runtime_version'] = require('spark.runtime_version') + + path = f"{folder_path}/Setting/Sparkcompute.yml" + if not os.path.exists(path): + print(f" - No Sparkcompute.yml in {folder_path}, skip spark settings") + return + + with open(path, "r", encoding="utf-8", newline="") as file: + lines = file.readlines() + + unseen = dict(settings) + for index, line in enumerate(lines): + key = line.split(":", 1)[0].strip() + if key in unseen and not line.startswith((" ", "\t", "#")): + newline = "\r\n" if line.endswith("\r\n") else ("\n" if line.endswith("\n") else "") + lines[index] = f"{key}: {unseen.pop(key)}{newline}" + for key, value in unseen.items(): + if lines and not lines[-1].endswith(("\n", "\r\n")): + lines[-1] += "\n" + lines.append(f"{key}: {value}\n") + + with open(path, "w", encoding="utf-8", newline="") as file: + file.writelines(lines) + print(f" - Spark settings applied: {settings}") + + +def apply_value_sets(folder_path): + """ + Narrows a Variable Library to the value sets the manifest declares active. + src/ ships all four (Development/Test/Acceptance/Production); a customer on + DTP should not get an empty Acceptance set deployed alongside. + """ + active = [environment['name'] for environment in require('environments')] + + settings_path = f"{folder_path}/settings.json" + if os.path.exists(settings_path): + with open(settings_path, "r", encoding="utf-8") as file: + settings = json.load(file) + settings["valueSetsOrder"] = active + with open(settings_path, "w", encoding="utf-8") as file: + json.dump(settings, file, indent=2) + + value_sets_path = f"{folder_path}/valueSets" + if os.path.isdir(value_sets_path): + for file_name in os.listdir(value_sets_path): + if file_name.endswith(".json") and file_name[:-len(".json")] not in active: + os.remove(os.path.join(value_sets_path, file_name)) + print(f" - Value sets limited to {active}") + + # ------------------------------- # Item deployment # ------------------------------- @@ -654,6 +735,7 @@ def deploy_item(workspace_name,name, mapping_table, environment_name, tasks, lak if VariableLibraryExists != "* true" or overwrite_variable_library: try: print(f"Creating or updating VariableLibrary: {name}") + apply_value_sets(tmp_path) result = update_variable_library(tmp_path, it.get("variables")) result = run_fab_command(f"import {workspace_name}.Workspace/{name} -i {tmp_path} -f",capture_output=True, silently_continue=True) print(f"✅ {name} Created/Imported'") @@ -665,9 +747,10 @@ def deploy_item(workspace_name,name, mapping_table, environment_name, tasks, lak assign_item_to_folder(workspace_name=workspace_name, item_id=new_id, folder_name='VariableLibraries') mapping_type='VariableLibrary' - elif "Environment" in name: + elif "Environment" in name: try: print(f"Creating or updating Environment: {name}") + apply_spark_settings(tmp_path) result = run_fab_command(f"import {workspace_name}.Workspace/{name} -i {tmp_path} -f",capture_output=True, silently_continue=True) print(f"✅ {name} Created/Imported'") except Exception as e: @@ -718,6 +801,178 @@ def deploy_item(workspace_name,name, mapping_table, environment_name, tasks, lak # CELL ******************** +# ------------------------------- +# Deployment Pipeline Management +# ------------------------------- +# Native Fabric ALM (Deployment Pipelines), one per workspace-group (CODE/DATA/REPORTING/ +# SEMANTIC), 3 stages Development/Test/Production, each stage assigned to the matching +# (D)/(T)/(P) workspace. Every domain built by NB_SETUP_FMD/NB_SETUP_BUSINESS_DOMAINS ends +# up needing this, so it is scripted here instead of clicked together by hand per customer. + +def get_deployment_pipeline_id_by_name(name): + """ + Retrieves the deployment pipeline ID by its display name. + """ + response = invoke_fabric_api_request("get", "deploymentPipelines") + pipelines = response.json().get("value", []) + normalized_name = name.strip().lower() + match = next((p for p in pipelines if p['displayName'].strip().lower() == normalized_name), None) + return match['id'] if match else None + +def ensure_deployment_pipeline(pipeline_name, stage_names=("Development", "Test", "Production")): + """ + Ensures a deployment pipeline exists with the given stages. Does not touch an existing + pipeline of this name (there is no API to change stage count after creation), so a + manually-adjusted pipeline is left alone on re-run. + """ + pipeline_id = get_deployment_pipeline_id_by_name(pipeline_name) + if pipeline_id: + print(f" - Deployment pipeline '{pipeline_name}' already exists, skip creation") + return pipeline_id + + payload = { + "displayName": pipeline_name, + "stages": [{"displayName": s, "isPublic": s == stage_names[-1]} for s in stage_names], + } + try: + response = invoke_fabric_request("post", FABRIC_API_BASE_URL + "deploymentPipelines", payload) + pipeline_id = response.json()["id"] + print(f"✅ Deployment pipeline '{pipeline_name}' created") + except Exception as e: + print(f"❌ Failed to create deployment pipeline '{pipeline_name}': {e}") + return pipeline_id + +def assign_deployment_pipeline_stage(pipeline_id, stage_name, workspace_name): + """ + Assigns a workspace to a pipeline stage by stage display name. Skips if the stage + already has a workspace (assign fails on an already-assigned stage) or the workspace + does not exist yet. + """ + if not pipeline_id: + return + stages = invoke_fabric_api_request("get", f"deploymentPipelines/{pipeline_id}/stages").json().get("value", []) + stage = next((s for s in stages if s['displayName'] == stage_name), None) + if not stage: + print(f"❌ Stage '{stage_name}' not found in pipeline {pipeline_id}") + return + if stage.get("workspaceId"): + print(f" - Stage '{stage_name}' already assigned, skip") + return + workspace_id = get_workspace_id_by_name(workspace_name) + if not workspace_id: + print(f"❌ Workspace '{workspace_name}' not found, skip stage assignment") + return + try: + response = invoke_fabric_request("post", f"{FABRIC_API_BASE_URL}deploymentPipelines/{pipeline_id}/stages/{stage['id']}/assignWorkspace", {"workspaceId": workspace_id}) + if response.status_code in (200, 201): + print(f"✅ '{workspace_name}' assigned to stage '{stage_name}'") + else: + print(f"❌ Failed to assign '{workspace_name}' to stage '{stage_name}': HTTP {response.status_code}") + except Exception as e: + print(f"❌ Failed to assign '{workspace_name}' to stage '{stage_name}': {e}") + +def deploy_deployment_pipeline(pipeline_name, stage_workspace_names): + """ + Ensures a deployment pipeline exists and its stages are assigned. + stage_workspace_names: ordered dict/list of (stage_name, workspace_name), e.g. + [("Development", "FINANCE CODE (D)"), ("Test", "FINANCE CODE (T)"), ("Production", "FINANCE CODE (P)")] + """ + stage_names = [s for s, _ in stage_workspace_names] + pipeline_id = ensure_deployment_pipeline(pipeline_name, stage_names) + for stage_name, workspace_name in stage_workspace_names: + assign_deployment_pipeline_stage(pipeline_id, stage_name, workspace_name) + return pipeline_id + +def wire_domain_into_orchestration(orchestration_workspace_name, gold_workspace_name, activity_name, + orchestration_pipeline_name="PL_FMD_ORCHESTRATION_TEMPLATE", + gold_pipeline_name="PL_FMD_LOAD_GOLD_TEMPLATE"): + """ + Adds an InvokePipeline activity to PL_FMD_ORCHESTRATION_TEMPLATE (in orchestration_workspace_name) + that calls PL_FMD_LOAD_GOLD_TEMPLATE in gold_workspace_name. + Idempotent: skips if an activity with this name is already wired in. + """ + orch_workspace_id = get_workspace_id_by_name(orchestration_workspace_name) + orch_pipeline_id = get_item_id(orchestration_workspace_name, f"{orchestration_pipeline_name}.DataPipeline", "id") + + gold_workspace_id = get_workspace_id_by_name(gold_workspace_name) + gold_pipeline_id = get_item_id(gold_workspace_name, f"{gold_pipeline_name}.DataPipeline", "id") + + if not (orch_workspace_id and orch_pipeline_id and gold_workspace_id and gold_pipeline_id): + print(f" - Could not resolve IDs to wire '{gold_workspace_name}' into orchestration, skip") + return + + response = invoke_fabric_api_request("post", f"workspaces/{orch_workspace_id}/items/{orch_pipeline_id}/getDefinition") + response.raise_for_status() + parts = response.json()["definition"]["parts"] + content_part = next(p for p in parts if p["path"] == "pipeline-content.json") + content = json.loads(base64.b64decode(content_part["payload"]).decode("utf-8")) + + activities = content["properties"]["activities"] + if any(a["name"] == activity_name for a in activities): + print(f" - Orchestration already wired for '{gold_workspace_name}', skip") + return + + template_activity = activities[0] + activities.append({ + "type": "InvokePipeline", + "typeProperties": { + "waitOnCompletion": True, + "workspaceId": gold_workspace_id, + "pipelineId": gold_pipeline_id, + "operationType": "InvokeFabricPipeline", + }, + "externalReferences": template_activity["externalReferences"], + "policy": template_activity["policy"], + "name": activity_name, + "dependsOn": [{"activity": template_activity["name"], "dependencyConditions": ["Succeeded"]}], + }) + + content_part["payload"] = base64.b64encode(json.dumps(content).encode("utf-8")).decode("utf-8") + response = invoke_fabric_api_request("post", f"workspaces/{orch_workspace_id}/items/{orch_pipeline_id}/updateDefinition", {"definition": {"parts": parts}}) + if response.status_code in (200, 201, 202): + print(f"✅ Wired '{gold_workspace_name}' Gold pipeline into orchestration template") + else: + print(f"❌ Failed to wire '{gold_workspace_name}': HTTP {response.status_code} {response.text}") + +def set_variable_library_values(workspace_name, library_name, values): + """ + Sets default values on a deployed Variable Library (values.json), e.g. so + VAR_GOLD_SHORTCUTS_FMD's workspace/lakehouse IDs don't have to be looked up + and pasted in by hand. `values` is {variable_name: new_value}; variables not + present in `values` are left untouched. + """ + workspace_id = get_workspace_id_by_name(workspace_name) + library_id = get_item_id(workspace_name, f"{library_name}.VariableLibrary", "id") + if not (workspace_id and library_id): + print(f" - Could not resolve '{library_name}' in '{workspace_name}', skip") + return + + response = invoke_fabric_api_request("post", f"workspaces/{workspace_id}/items/{library_id}/getDefinition") + response.raise_for_status() + parts = response.json()["definition"]["parts"] + variables_part = next(p for p in parts if p["path"] == "variables.json") + content = json.loads(base64.b64decode(variables_part["payload"]).decode("utf-8")) + + for variable in content["variables"]: + if variable["name"] in values: + variable["value"] = values[variable["name"]] + + variables_part["payload"] = base64.b64encode(json.dumps(content).encode("utf-8")).decode("utf-8") + response = invoke_fabric_api_request("post", f"workspaces/{workspace_id}/items/{library_id}/updateDefinition", {"definition": {"parts": parts}}) + if response.status_code in (200, 201, 202): + print(f"✅ Set {list(values.keys())} on '{library_name}' in '{workspace_name}'") + else: + print(f"❌ Failed to update '{library_name}' in '{workspace_name}': HTTP {response.status_code} {response.text}") + +# METADATA ******************** + +# META { +# META "language": "python", +# META "language_group": "jupyter_python" +# META } + +# CELL ******************** + # ------------------------------- # Connections # ------------------------------- @@ -820,6 +1075,10 @@ def create_or_get_fmd_connection(connection_name,connection_role, type): print("FabricSql needs a service principal to be created automatically. " "Set fabric_sql_sp_tenant_id / _client_id / _secret, or create it manually.") elif type =='AzureDataFactory': + # ponytail: when no manual connection exists yet, the get below returns an + # error/empty string that silently becomes this connection's mapping-table + # new_id, corrupting every GUID it replaces. Harden by failing loudly here + # once a tenant with real ADF needs this path. print("AzureDataFactory can't created automated yet to CLI limitations, please create manual") elif type =='FabricDataPipelines': run_fab_command(f"""create .connections/{connection_name}.Connection diff --git a/src/PL_FMD_LDZ_COMMAND_ADF.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_ADF.DataPipeline/pipeline-content.json index a079141a..2fbde0ad 100644 --- a/src/PL_FMD_LDZ_COMMAND_ADF.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_ADF.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ADF' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ADF' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_ADLS.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_ADLS.DataPipeline/pipeline-content.json index 55007777..392c36d3 100644 --- a/src/PL_FMD_LDZ_COMMAND_ADLS.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_ADLS.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DataSourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ADLS' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' group by DataSourceType", + "value": "SELECT DataSourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ADLS' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DataSourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_ASQL.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_ASQL.DataPipeline/pipeline-content.json index 02fb1d06..0407d2c1 100644 --- a/src/PL_FMD_LDZ_COMMAND_ASQL.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_ASQL.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='SQL' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='SQL' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_FTP.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_FTP.DataPipeline/pipeline-content.json index b4070841..4103a59b 100644 --- a/src/PL_FMD_LDZ_COMMAND_FTP.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_FTP.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='FTP' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='FTP' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_NOTEBOOK.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_NOTEBOOK.DataPipeline/pipeline-content.json index ac8ca30a..78147efe 100644 --- a/src/PL_FMD_LDZ_COMMAND_NOTEBOOK.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_NOTEBOOK.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='NOTEBOOK' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='NOTEBOOK' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_ONELAKE.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_ONELAKE.DataPipeline/pipeline-content.json index 513cc2bf..eace5080 100644 --- a/src/PL_FMD_LDZ_COMMAND_ONELAKE.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_ONELAKE.DataPipeline/pipeline-content.json @@ -234,7 +234,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ONELAKE' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ONELAKE' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -306,6 +306,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -339,6 +343,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -519,6 +527,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_ORACLE.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_ORACLE.DataPipeline/pipeline-content.json index 3dbb75e8..8004559d 100644 --- a/src/PL_FMD_LDZ_COMMAND_ORACLE.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_ORACLE.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ORACLE' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='ORACLE' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_SFTP.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_SFTP.DataPipeline/pipeline-content.json index ac811fd3..b778f231 100644 --- a/src/PL_FMD_LDZ_COMMAND_SFTP.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_SFTP.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='SFTP' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='SFTP' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COMMAND_SQLMI.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COMMAND_SQLMI.DataPipeline/pipeline-content.json index c4890126..8eb0d0ac 100644 --- a/src/PL_FMD_LDZ_COMMAND_SQLMI.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COMMAND_SQLMI.DataPipeline/pipeline-content.json @@ -245,6 +245,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -283,7 +287,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='AZURESQLMI' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'group by DatasourceType", + "value": "SELECT DatasourceType FROM [execution].[vw_LoadSourceToLandingzone] where ConnectionType='AZURESQLMI' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}') group by DatasourceType", "type": "Expression" }, "partitionOption": "None", @@ -467,6 +471,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ADF.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ADF.DataPipeline/pipeline-content.json index 66e575a7..c35511b5 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ADF.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ADF.DataPipeline/pipeline-content.json @@ -434,8 +434,9 @@ } }, "externalReferences": { - "connection": "02e107b8-e97e-4b00-a28c-668cf9ce3d9a" - } + "connection": "d18305e6-f633-4506-982c-53ee1c9dff16" + }, + "description": "PLACEHOLDER connection: er is geen Azure Data Factory in de Bitmetric-tenant. De upstream ADF-connection bestaat niet, waardoor de deploy-import faalde. Vervang deze GUID door een echte Azure Data Factory-connection voordat je deze pipeline draait -- nu faalt hij at runtime." }, { "name": "SP_UPDATE_PROCESS", @@ -805,7 +806,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ADF'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ADF'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -978,6 +979,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ADLS_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ADLS_01.DataPipeline/pipeline-content.json index 47135625..93b72bca 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ADLS_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ADLS_01.DataPipeline/pipeline-content.json @@ -851,7 +851,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ADLS_01'and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ADLS_01'and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1023,6 +1023,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ASQL_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ASQL_01.DataPipeline/pipeline-content.json index 19cda2e5..c168f307 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ASQL_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ASQL_01.DataPipeline/pipeline-content.json @@ -1037,7 +1037,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ASQL_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ASQL_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1092,7 +1092,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ASQL_02'and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ASQL_02'and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -2180,6 +2180,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_CUSTOM_NB.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_CUSTOM_NB.DataPipeline/pipeline-content.json index d98e7f00..b6101fac 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_CUSTOM_NB.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_CUSTOM_NB.DataPipeline/pipeline-content.json @@ -847,7 +847,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='NOTEBOOK'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='NOTEBOOK'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "partitionOption": "None", @@ -1036,6 +1036,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_FTP_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_FTP_01.DataPipeline/pipeline-content.json index 0442cf8f..34db2568 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_FTP_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_FTP_01.DataPipeline/pipeline-content.json @@ -134,7 +134,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='FTP_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='FTP_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1205,6 +1205,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_FILES_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_FILES_01.DataPipeline/pipeline-content.json index e0adb6a4..8045f1a5 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_FILES_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_FILES_01.DataPipeline/pipeline-content.json @@ -858,7 +858,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ONELAKE_FILES_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ONELAKE_FILES_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1030,6 +1030,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_TABLES_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_TABLES_01.DataPipeline/pipeline-content.json index 461fe111..1ec75992 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_TABLES_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ONELAKE_TABLES_01.DataPipeline/pipeline-content.json @@ -862,7 +862,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ONELAKE_TABLES_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ONELAKE_TABLES_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1034,6 +1034,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_ORACLE_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_ORACLE_01.DataPipeline/pipeline-content.json index fa0aff60..9c7beffc 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_ORACLE_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_ORACLE_01.DataPipeline/pipeline-content.json @@ -1057,7 +1057,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ORACLE_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='ORACLE_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1230,6 +1230,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_SFTP_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_SFTP_01.DataPipeline/pipeline-content.json index 549c9864..9a60a38d 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_SFTP_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_SFTP_01.DataPipeline/pipeline-content.json @@ -134,7 +134,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='SFTP_01' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='SFTP_01' and WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1203,6 +1203,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LDZ_COPY_FROM_SQLMI_01.DataPipeline/pipeline-content.json b/src/PL_FMD_LDZ_COPY_FROM_SQLMI_01.DataPipeline/pipeline-content.json index dedbb73e..949b21b5 100644 --- a/src/PL_FMD_LDZ_COPY_FROM_SQLMI_01.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LDZ_COPY_FROM_SQLMI_01.DataPipeline/pipeline-content.json @@ -1005,7 +1005,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='SQLMI_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT * FROM [execution].[vw_LoadSourceToLandingzone] where DataSourceType='SQLMI_01'\nand WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "queryTimeout": "02:00:00", @@ -1177,6 +1177,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json b/src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json index 1e650eae..94e2760f 100644 --- a/src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LOAD_ALL.DataPipeline/pipeline-content.json @@ -8,6 +8,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -42,6 +46,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -538,6 +546,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -741,6 +753,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json b/src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json index bc096aa7..65cbcbd9 100644 --- a/src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LOAD_BRONZE.DataPipeline/pipeline-content.json @@ -241,6 +241,13 @@ "type": "Expression" }, "type": "Guid" + }, + "LoadGroup": { + "value": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" + }, + "type": "String" } }, "partitionOption": "None", @@ -513,6 +520,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json b/src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json index 793b2a5d..a8704410 100644 --- a/src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LOAD_LANDINGZONE.DataPipeline/pipeline-content.json @@ -32,6 +32,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -68,6 +72,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -104,6 +112,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -140,6 +152,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -176,6 +192,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -212,6 +232,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -248,6 +272,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -284,6 +312,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -320,6 +352,10 @@ "Data_WorkspaceGuid": { "value": "@pipeline().parameters.Data_WorkspaceGuid", "type": "Expression" + }, + "LoadGroup": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" } }, "waitOnCompletion": true, @@ -606,7 +642,7 @@ "source": { "type": "FabricSqlDatabaseSource", "sqlReaderQuery": { - "value": "SELECT distinct [ConnectionType] FROM [execution].[vw_LoadSourceToLandingzone] where WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}'", + "value": "SELECT distinct [ConnectionType] FROM [execution].[vw_LoadSourceToLandingzone] where WorkspaceGuid='@{pipeline().parameters.Data_WorkspaceGuid}' and (''='@{pipeline().parameters.LoadGroup}' or LoadGroup='@{pipeline().parameters.LoadGroup}')", "type": "Expression" }, "partitionOption": "None", @@ -775,6 +811,10 @@ "Data_WorkspaceGuid": { "type": "string", "defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json b/src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json index d94842be..0501d307 100644 --- a/src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json +++ b/src/PL_FMD_LOAD_SILVER.DataPipeline/pipeline-content.json @@ -14,6 +14,13 @@ "type": "Expression" }, "type": "Guid" + }, + "LoadGroup": { + "value": { + "value": "@pipeline().parameters.LoadGroup", + "type": "Expression" + }, + "type": "String" } }, "partitionOption": "None", @@ -517,6 +524,10 @@ "parameters": { "Data_WorkspaceGuid": { "type": "string" + }, + "LoadGroup": { + "type": "string", + "defaultValue": "" } }, "libraryVariables": { diff --git a/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/.platform b/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/.platform new file mode 100644 index 00000000..393381d8 --- /dev/null +++ b/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/.platform @@ -0,0 +1,12 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json", + "metadata": { + "type": "DataPipeline", + "displayName": "PL_FMD_ORCHESTRATION_TEMPLATE", + "description": "Template top-of-day orchestrator: runs PL_FMD_LOAD_ALL (Landingzone/Bronze/Silver), then fans out to one Gold-layer pipeline per business domain. Not part of the core FMD load chain: copy/rename per customer and add one InvokePipeline activity per business domain Gold pipeline (e.g. PL_FMD_LOAD_GOLD_TEMPLATE), each dependsOn this one Succeeded." + }, + "config": { + "version": "2.0", + "logicalId": "14bb3cb5-26dc-4840-bccc-41556d9a22ec" + } +} diff --git a/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/pipeline-content.json b/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/pipeline-content.json new file mode 100644 index 00000000..04d440ee --- /dev/null +++ b/src/PL_FMD_ORCHESTRATION_TEMPLATE.DataPipeline/pipeline-content.json @@ -0,0 +1,28 @@ +{ + "properties": { + "activities": [ + { + "type": "InvokePipeline", + "typeProperties": { + "waitOnCompletion": true, + "workspaceId": "00000000-0000-0000-0000-000000000000", + "pipelineId": "1b6d7d0d-d04b-4bc2-a6da-7a876216561d", + "operationType": "InvokeFabricPipeline" + }, + "externalReferences": { + "connection": "6d8146c6-a438-47df-94e2-540c552eb6d7" + }, + "policy": { + "timeout": "0.12:00:00", + "retry": 0, + "retryIntervalInSeconds": 30, + "secureInput": false, + "secureOutput": false + }, + "name": "PL_FMD_LOAD_ALL", + "dependsOn": [] + } + ], + "annotations": [] + } +} diff --git a/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/.platform b/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/.platform new file mode 100644 index 00000000..068a95f3 --- /dev/null +++ b/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/.platform @@ -0,0 +1,12 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json", + "metadata": { + "type": "DataPipeline", + "displayName": "PL_FMD_SCHEDULE_INTRADAY", + "description": "Schedule wrapper: invokes PL_FMD_LOAD_ALL with a fixed LoadGroup. Attach this pipeline's own Fabric schedule to run this group's sources on their own cadence. See wiki/LoadGroup_Scheduling.md." + }, + "config": { + "version": "2.0", + "logicalId": "6beb329c-f284-41b9-9d1b-bd292059e075" + } +} \ No newline at end of file diff --git a/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/pipeline-content.json b/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/pipeline-content.json new file mode 100644 index 00000000..e0edec1c --- /dev/null +++ b/src/PL_FMD_SCHEDULE_INTRADAY.DataPipeline/pipeline-content.json @@ -0,0 +1,40 @@ +{ + "properties": { + "activities": [ + { + "type": "InvokePipeline", + "typeProperties": { + "parameters": { + "Data_WorkspaceGuid": { + "value": "@pipeline().parameters.Data_WorkspaceGuid", + "type": "Expression" + }, + "LoadGroup": "INTRADAY" + }, + "waitOnCompletion": true, + "workspaceId": "00000000-0000-0000-0000-000000000000", + "pipelineId": "1b6d7d0d-d04b-4bc2-a6da-7a876216561d", + "operationType": "InvokeFabricPipeline" + }, + "externalReferences": { + "connection": "6d8146c6-a438-47df-94e2-540c552eb6d7" + }, + "policy": { + "timeout": "0.12:00:00", + "retry": 0, + "retryIntervalInSeconds": 30, + "secureInput": false, + "secureOutput": false + }, + "name": "PL_FMD_LOAD_ALL", + "dependsOn": [] + } + ], + "parameters": { + "Data_WorkspaceGuid": { + "type": "string", + "defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc" + } + } + } +} \ No newline at end of file diff --git a/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/.platform b/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/.platform new file mode 100644 index 00000000..bd063da0 --- /dev/null +++ b/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/.platform @@ -0,0 +1,12 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json", + "metadata": { + "type": "DataPipeline", + "displayName": "PL_FMD_SCHEDULE_NIGHTLY", + "description": "Schedule wrapper: invokes PL_FMD_LOAD_ALL with a fixed LoadGroup. Attach this pipeline's own Fabric schedule to run this group's sources on their own cadence. See wiki/LoadGroup_Scheduling.md." + }, + "config": { + "version": "2.0", + "logicalId": "03ba9db4-d5b4-49b5-a37d-c389a3e0a5dd" + } +} \ No newline at end of file diff --git a/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/pipeline-content.json b/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/pipeline-content.json new file mode 100644 index 00000000..c6aae60a --- /dev/null +++ b/src/PL_FMD_SCHEDULE_NIGHTLY.DataPipeline/pipeline-content.json @@ -0,0 +1,40 @@ +{ + "properties": { + "activities": [ + { + "type": "InvokePipeline", + "typeProperties": { + "parameters": { + "Data_WorkspaceGuid": { + "value": "@pipeline().parameters.Data_WorkspaceGuid", + "type": "Expression" + }, + "LoadGroup": "NIGHTLY" + }, + "waitOnCompletion": true, + "workspaceId": "00000000-0000-0000-0000-000000000000", + "pipelineId": "1b6d7d0d-d04b-4bc2-a6da-7a876216561d", + "operationType": "InvokeFabricPipeline" + }, + "externalReferences": { + "connection": "6d8146c6-a438-47df-94e2-540c552eb6d7" + }, + "policy": { + "timeout": "0.12:00:00", + "retry": 0, + "retryIntervalInSeconds": 30, + "secureInput": false, + "secureOutput": false + }, + "name": "PL_FMD_LOAD_ALL", + "dependsOn": [] + } + ], + "parameters": { + "Data_WorkspaceGuid": { + "type": "string", + "defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc" + } + } + } +} \ No newline at end of file diff --git a/src/PL_TOOLING_POST_ASQL_TO_FMD.DataPipeline/pipeline-content.json b/src/PL_TOOLING_POST_ASQL_TO_FMD.DataPipeline/pipeline-content.json index 98ccdd5a..e67929f9 100644 --- a/src/PL_TOOLING_POST_ASQL_TO_FMD.DataPipeline/pipeline-content.json +++ b/src/PL_TOOLING_POST_ASQL_TO_FMD.DataPipeline/pipeline-content.json @@ -40,8 +40,8 @@ "annotations": [], "type": "Lakehouse", "typeProperties": { - "workspaceId": "40e27fdc-775a-4ee2-84d5-48893c92d7cc", - "artifactId": "4b4840bc-5a9f-434e-8345-3f528d1f39bf", + "workspaceId": "@pipeline().libraryVariables.VAR_CONFIG_FMD_fmd_data_workspace_guid", + "artifactId": "@pipeline().libraryVariables.VAR_CONFIG_FMD_fmd_landingzone_lakehouse_guid", "rootFolder": "Files" } } @@ -166,7 +166,7 @@ "WorkspaceGuid": { "type": "Guid", "value": { - "value": "@pipeline().parameters.Data_WorkspaceGuid", + "value": "@pipeline().libraryVariables.VAR_CONFIG_FMD_fmd_data_workspace_guid", "type": "Expression" } }, @@ -260,8 +260,8 @@ "annotations": [], "type": "Lakehouse", "typeProperties": { - "workspaceId": "40e27fdc-775a-4ee2-84d5-48893c92d7cc", - "artifactId": "4b4840bc-5a9f-434e-8345-3f528d1f39bf", + "workspaceId": "@pipeline().libraryVariables.VAR_CONFIG_FMD_fmd_data_workspace_guid", + "artifactId": "@pipeline().libraryVariables.VAR_CONFIG_FMD_fmd_landingzone_lakehouse_guid", "rootFolder": "Files" } } @@ -511,10 +511,6 @@ "type": "string", "defaultValue": "cf673e6a-13f6-4ebb-9cbb-4ba4ab390818" }, - "Data_WorkspaceGuid": { - "type": "string", - "defaultValue": "40e27fdc-775a-4ee2-84d5-48893c92d7cc" - }, "DatasourceName": { "type": "string", "defaultValue": "WideWorldImporters" @@ -557,6 +553,16 @@ "type": "String", "variableName": "fmd_config_database_guid", "libraryName": "VAR_CONFIG_FMD" + }, + "VAR_CONFIG_FMD_fmd_data_workspace_guid": { + "type": "String", + "variableName": "fmd_data_workspace_guid", + "libraryName": "VAR_CONFIG_FMD" + }, + "VAR_CONFIG_FMD_fmd_landingzone_lakehouse_guid": { + "type": "String", + "variableName": "fmd_landingzone_lakehouse_guid", + "libraryName": "VAR_CONFIG_FMD" } } } diff --git a/src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac b/src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac index 152f8d73..79ad8421 100644 Binary files a/src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac and b/src/SQL_FMD_FRAMEWORK.SQLDatabase/SQL_FMD_FRAMEWORK.dacpac differ diff --git a/src/VAR_CONFIG_FMD.VariableLibrary/settings.json b/src/VAR_CONFIG_FMD.VariableLibrary/settings.json index 818709f1..27ecd626 100644 --- a/src/VAR_CONFIG_FMD.VariableLibrary/settings.json +++ b/src/VAR_CONFIG_FMD.VariableLibrary/settings.json @@ -1,8 +1,9 @@ { "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/settings/1.0.0/schema.json", "valueSetsOrder": [ + "Development", "Test", "Acceptance", "Production" ] -} \ No newline at end of file +} diff --git a/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Acceptance.json b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Acceptance.json index 45a67812..c01dc2b8 100644 --- a/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Acceptance.json +++ b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Acceptance.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Acceptance", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Prod.json b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Development.json similarity index 50% rename from src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Prod.json rename to src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Development.json index 0c23f905..44a59070 100644 --- a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Prod.json +++ b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Development.json @@ -1,10 +1,5 @@ { "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", - "name": "Prod", - "variableOverrides": [ - { - "name": "key_vault_uri_name", - "value": "keyvaultname" - } - ] -} \ No newline at end of file + "name": "Development", + "variableOverrides": [] +} diff --git a/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Test.json b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Test.json index 9f703875..b254c845 100644 --- a/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Test.json +++ b/src/VAR_CONFIG_FMD.VariableLibrary/valueSets/Test.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Test", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/VAR_CONFIG_FMD.VariableLibrary/variables.json b/src/VAR_CONFIG_FMD.VariableLibrary/variables.json index 8caee29c..3732a2e5 100644 --- a/src/VAR_CONFIG_FMD.VariableLibrary/variables.json +++ b/src/VAR_CONFIG_FMD.VariableLibrary/variables.json @@ -24,6 +24,18 @@ "note": "", "type": "String", "value": "" + }, + { + "name": "fmd_data_workspace_guid", + "note": "", + "type": "String", + "value": "" + }, + { + "name": "fmd_landingzone_lakehouse_guid", + "note": "", + "type": "String", + "value": "" } ] } \ No newline at end of file diff --git a/src/VAR_FMD.VariableLibrary/settings.json b/src/VAR_FMD.VariableLibrary/settings.json index 818709f1..27ecd626 100644 --- a/src/VAR_FMD.VariableLibrary/settings.json +++ b/src/VAR_FMD.VariableLibrary/settings.json @@ -1,8 +1,9 @@ { "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/settings/1.0.0/schema.json", "valueSetsOrder": [ + "Development", "Test", "Acceptance", "Production" ] -} \ No newline at end of file +} diff --git a/src/VAR_FMD.VariableLibrary/valueSets/Acceptance.json b/src/VAR_FMD.VariableLibrary/valueSets/Acceptance.json index 45a67812..c01dc2b8 100644 --- a/src/VAR_FMD.VariableLibrary/valueSets/Acceptance.json +++ b/src/VAR_FMD.VariableLibrary/valueSets/Acceptance.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Acceptance", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/VAR_FMD.VariableLibrary/valueSets/Development.json b/src/VAR_FMD.VariableLibrary/valueSets/Development.json new file mode 100644 index 00000000..44a59070 --- /dev/null +++ b/src/VAR_FMD.VariableLibrary/valueSets/Development.json @@ -0,0 +1,5 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", + "name": "Development", + "variableOverrides": [] +} diff --git a/src/VAR_FMD.VariableLibrary/valueSets/Test.json b/src/VAR_FMD.VariableLibrary/valueSets/Test.json index 9f703875..b254c845 100644 --- a/src/VAR_FMD.VariableLibrary/valueSets/Test.json +++ b/src/VAR_FMD.VariableLibrary/valueSets/Test.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Test", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/VAR_FMD.VariableLibrary/variables.json b/src/VAR_FMD.VariableLibrary/variables.json index 4c49c971..f02e992c 100644 --- a/src/VAR_FMD.VariableLibrary/variables.json +++ b/src/VAR_FMD.VariableLibrary/variables.json @@ -11,8 +11,8 @@ "name": "lakehouse_schema_enabled", "note": "", "type": "Boolean", - "value": "" - } , + "value": true + }, { "name": "purview_account_name", "note": "", diff --git a/src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql b/src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql index 9e3912e1..5f38eb0e 100644 --- a/src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql +++ b/src/business_domain/NB_MLV_EXAMPLE.Notebook/notebook-content.sql @@ -24,8 +24,8 @@ -- CELL ******************** --- Welcome to your new notebook --- Type here in the cell editor to add code! +-- Welcome to your new notebook +-- Type here in the cell editor to add code! -- CREATE MATERIALIZED LAKE VIEW AS select_statement -- METADATA ******************** @@ -34,3 +34,33 @@ -- META "language": "sparksql", -- META "language_group": "synapse_pyspark" -- META } + +-- MARKDOWN ******************** + +-- ## Refresh the materialized lake views +-- Creating/replacing an MLV above does not refresh it. Run this cell after the CREATE statements +-- (or schedule it after them in a pipeline) so the views actually recompute. Fill in your Gold +-- lakehouse id below. + +-- CELL ******************** + +%%pyspark +import requests + +workspace_id = notebookutils.runtime.context.get('currentWorkspaceId') +lakehouse_id = "" + +token = notebookutils.credentials.getToken("https://api.fabric.microsoft.com") +headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"} + +url = f"https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/lakehouses/{lakehouse_id}/jobs/RefreshMaterializedLakeViews/instances" +response = requests.post(url, headers=headers) +print(response.status_code) +print(response.headers.get("Location")) + +-- METADATA ******************** + +-- META { +-- META "language": "python", +-- META "language_group": "synapse_pyspark" +-- META } diff --git a/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/.platform b/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/.platform new file mode 100644 index 00000000..6ecf114c --- /dev/null +++ b/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/.platform @@ -0,0 +1,12 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json", + "metadata": { + "type": "DataPipeline", + "displayName": "PL_FMD_LOAD_GOLD_TEMPLATE", + "description": "Template Gold-layer load pipeline for a business domain: NB_LOAD_GOLD, then NB_MLV_EXAMPLE. Swap NB_MLV_EXAMPLE for your domain's own MLV notebook(s), and add a PBISemanticModelRefresh activity pointing at your domain's semantic model after deployment. Not part of the core FMD chain: one per business domain, invoked from your orchestration pipeline (see PL_FMD_ORCHESTRATION_TEMPLATE)." + }, + "config": { + "version": "2.0", + "logicalId": "838bc064-4221-48e3-ae54-0fdb0ce5a703" + } +} diff --git a/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/pipeline-content.json b/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/pipeline-content.json new file mode 100644 index 00000000..6934b240 --- /dev/null +++ b/src/business_domain/PL_FMD_LOAD_GOLD_TEMPLATE.DataPipeline/pipeline-content.json @@ -0,0 +1,43 @@ +{ + "properties": { + "activities": [ + { + "name": "NB_LOAD_GOLD", + "type": "TridentNotebook", + "dependsOn": [], + "policy": { + "timeout": "0.12:00:00", + "retry": 0, + "retryIntervalInSeconds": 30, + "secureOutput": false, + "secureInput": false + }, + "typeProperties": { + "notebookId": "8b6fabb1-f9a2-af87-4260-23061c74f547", + "workspaceId": "00000000-0000-0000-0000-000000000000" + } + }, + { + "name": "NB_MLV_EXAMPLE", + "type": "TridentNotebook", + "dependsOn": [ + { + "activity": "NB_LOAD_GOLD", + "dependencyConditions": ["Succeeded"] + } + ], + "policy": { + "timeout": "0.12:00:00", + "retry": 0, + "retryIntervalInSeconds": 30, + "secureOutput": false, + "secureInput": false + }, + "typeProperties": { + "notebookId": "7f3419e4-1e18-9166-43d1-de9529d88dc3", + "workspaceId": "00000000-0000-0000-0000-000000000000" + } + } + ] + } +} diff --git a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/settings.json b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/settings.json index 818709f1..27ecd626 100644 --- a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/settings.json +++ b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/settings.json @@ -1,8 +1,9 @@ { "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/settings/1.0.0/schema.json", "valueSetsOrder": [ + "Development", "Test", "Acceptance", "Production" ] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Acceptance.json b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Acceptance.json index 45a67812..c01dc2b8 100644 --- a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Acceptance.json +++ b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Acceptance.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Acceptance", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Development.json b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Development.json new file mode 100644 index 00000000..44a59070 --- /dev/null +++ b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Development.json @@ -0,0 +1,5 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", + "name": "Development", + "variableOverrides": [] +} diff --git a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Test.json b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Test.json index 9f703875..b254c845 100644 --- a/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Test.json +++ b/src/business_domain/VAR_GOLD_CODE_FMD.VariableLibrary/valueSets/Test.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Test", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/settings.json b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/settings.json index 818709f1..27ecd626 100644 --- a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/settings.json +++ b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/settings.json @@ -1,8 +1,9 @@ { "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/settings/1.0.0/schema.json", "valueSetsOrder": [ + "Development", "Test", "Acceptance", "Production" ] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Acceptance.json b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Acceptance.json index 45a67812..c01dc2b8 100644 --- a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Acceptance.json +++ b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Acceptance.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Acceptance", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Development.json b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Development.json new file mode 100644 index 00000000..44a59070 --- /dev/null +++ b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Development.json @@ -0,0 +1,5 @@ +{ + "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", + "name": "Development", + "variableOverrides": [] +} diff --git a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Test.json b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Test.json index 9f703875..b254c845 100644 --- a/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Test.json +++ b/src/business_domain/VAR_GOLD_SHORTCUTS_FMD.VariableLibrary/valueSets/Test.json @@ -2,4 +2,4 @@ "$schema": "https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json", "name": "Test", "variableOverrides": [] -} \ No newline at end of file +} diff --git a/wiki/LoadGroup_Scheduling.md b/wiki/LoadGroup_Scheduling.md new file mode 100644 index 00000000..9759cc6d --- /dev/null +++ b/wiki/LoadGroup_Scheduling.md @@ -0,0 +1,132 @@ +# LoadGroup Scheduling — Different Cadences per Data Source + +## Overview + +`LoadGroup` lets you run **different data sources on different schedules** — +for example, two sources loaded only overnight and two other sources loaded +every two hours — without duplicating any pipeline. Every source keeps a +`LoadGroup` tag in the metadata; `PL_FMD_LOAD_ALL` (and its children) accept a +`LoadGroup` parameter and only pick up sources tagged with that value. + +--- + +## Why this exists + +Microsoft Fabric's own schedule trigger cannot carry parameters: every +scheduled run of an item uses that item's design-time default values, and the +job-scheduler API's `parameters` field is explicitly **not supported** for +Data Pipelines ([Job Scheduler REST API](https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/create-item-schedule)). +So one pipeline cannot have two schedules that load two different sets of +sources — the schedule can't tell it which set to use. + +`LoadGroup` works around this by moving the "which sources" decision into the +metadata (a column on `integration.DataSource`) and having each schedule +target its own thin wrapper pipeline that hard-codes only the group name. The +schedule still can't carry a parameter — but the wrapper pipeline it points +to needs exactly one. + +--- + +## How it works + +`DataSource.LoadGroup` (`VARCHAR(50)`, default `''`) tags each source. +`vw_LoadSourceToLandingzone`, `vw_LoadToBronzeLayer` and `vw_LoadToSilverLayer` +all expose it, and `sp_GetBronzelayerEntity` / `sp_GetSilverlayerEntity` take +a `@LoadGroup` parameter that filters by it. + +Every pipeline in the load chain (`PL_FMD_LOAD_ALL`, `PL_FMD_LOAD_LANDINGZONE`, +`PL_FMD_LOAD_BRONZE`, `PL_FMD_LOAD_SILVER`, and every `PL_FMD_LDZ_COMMAND_*` / +`PL_FMD_LDZ_COPY_FROM_*` pipeline underneath them) now takes a `LoadGroup` +parameter, defaulting to `''`, and passes it down to the next pipeline in the +chain the same way it already passes `Data_WorkspaceGuid`. + +**The empty string is not a group — it means "no filter".** A source with +`LoadGroup = ''` (the default) is picked up by *every* run regardless of what +`LoadGroup` the caller passed in, and calling the top-level pipeline with +`LoadGroup = ''` loads *all* sources exactly as before. This is what keeps +existing deployments working unchanged: nothing has to be tagged for the +framework to behave exactly as it did before this feature. + +A non-empty `LoadGroup` is an exact match: a run with `LoadGroup = 'NIGHTLY'` +only picks up sources tagged `NIGHTLY`, never `''` or any other group. + +--- + +## Setting it up + +### 1. Tag your data sources + +Pass `@LoadGroup` when registering or updating a source: + +```sql +EXEC [integration].[sp_UpsertDataSource] + @ConnectionId = 4, + @Name = 'Customers', + @Namespace = 'crm', + @Type = 'ASQL_01', + @Description = 'CRM customer export', + @LoadGroup = 'NIGHTLY'; +``` + +Leave `@LoadGroup` unset (or `''`) for sources that should load on every run, +regardless of which schedule triggered it. + +### 2. Create one thin wrapper pipeline per schedule + +The framework ships two ready-to-use examples of this pattern: +`PL_FMD_SCHEDULE_NIGHTLY` and `PL_FMD_SCHEDULE_INTRADAY`. Each is a single +`InvokePipeline` activity calling `PL_FMD_LOAD_ALL` with `LoadGroup` hard-coded +to a literal: + +| Wrapper pipeline | Invokes | `LoadGroup` | +|---|---|---| +| `PL_FMD_SCHEDULE_NIGHTLY` | `PL_FMD_LOAD_ALL` | `"NIGHTLY"` | +| `PL_FMD_SCHEDULE_INTRADAY` | `PL_FMD_LOAD_ALL` | `"INTRADAY"` | + +Rename or duplicate these for whatever group names and cadences your +implementation actually needs — `NIGHTLY`/`INTRADAY` are examples, not +reserved values. A new one is a copy-paste of the folder plus one entry in +`config/item_deployment.json` placed after `PL_FMD_LOAD_ALL`'s entry (so its +real object ID is already known when the wrapper's placeholder reference to +it gets resolved at deploy time) — the same recipe as any other new pipeline +entering the framework. If you only need one layer filtered rather than the +full chain, point the wrapper at `PL_FMD_LOAD_LANDINGZONE` / `_BRONZE` / +`_SILVER` instead. + +`Data_WorkspaceGuid` still has to be passed through as usual. + +### 3. Schedule each wrapper independently + +In the Fabric portal, open each wrapper pipeline's **Settings → Schedule** +and configure its own cadence — e.g. `PL_FMD_SCHEDULE_NIGHTLY` daily at +02:00, `PL_FMD_SCHEDULE_INTRADAY` every 2 hours. Each schedule triggers its +own wrapper with its own baked-in `LoadGroup`, so the two cadences never +interfere with each other. + +--- + +## Design notes + +- **Bronze and Silver are filtered too, not just Landingzone.** A group's + intraday run only reprocesses Bronze/Silver entities belonging to that + group's sources — it does not re-touch the nightly sources' Bronze/Silver + tables. This keeps a frequent schedule cheap and keeps the audit log + readable per run. +- **Gold is untouched by this feature.** If a Gold model joins a nightly + source with an intraday one, decide separately whether Gold refreshes per + group or once after the nightly run — `LoadGroup` only reaches through + Landingzone/Bronze/Silver. +- **Groups can overlap in time.** Two schedules with different groups share + the same Spark capacity (`NB_FMD_PROCESSING_PARALLEL_MAIN`); staggering + cadences avoids them competing for the same pool. +- **A source not yet tagged behaves exactly as before.** `LoadGroup = ''` is + the column default, so upgrading the framework requires no metadata + migration for existing sources. + +--- + +## Related Resources + +- [FMD Business Domain Deployment Guide](../FMD_BUSINESS_DOMAIN_DEPLOYMENT.md) +- [Job Scheduler — Create Item Schedule (REST API)](https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/create-item-schedule) +- [FMD Framework documentation](https://erwindekreuk.com/fmd-framework/)