Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
1cb68ba
Add fabric-cicd deployment pipeline for INTEGRATION code workspace
MitchellBitmetric Sep 4, 2026
0d054a8
Merge pull request #1 from bitmetric-fabric/feature/fabric-cicd-integ…
MitchellBitmetric Sep 4, 2026
e6b525e
Add TST environment across setup notebooks, variable libraries, and I…
MitchellBitmetric Sep 4, 2026
ef0248a
Merge pull request #2 from bitmetric-fabric/feature/add-tst-environment
MitchellBitmetric Sep 4, 2026
55e67d5
Fix pilot deploy script: enable required fabric-cicd feature flags
MitchellBitmetric Sep 4, 2026
a5feeff
Fix two more issues found by the first successful pilot run
MitchellBitmetric Sep 4, 2026
65da3fc
Fix VAR_FMD Boolean value: JSON boolean, not string
MitchellBitmetric Sep 4, 2026
c2dbe90
Fix ENV_FMD Spark compute settings exceeding the Dev/Test capacity's …
MitchellBitmetric Sep 4, 2026
286e824
Add parameter.yml to fix stale connection GUIDs in DataPipeline files
MitchellBitmetric Sep 4, 2026
0d0c2e9
Exclude two pipelines that fail for reasons outside this CI/CD setup
MitchellBitmetric Sep 4, 2026
68561b3
Retire fabric-cicd deployment for INTEGRATION CODE
MitchellBitmetric Sep 4, 2026
effbbe8
Add LoadGroup: schedule data sources on different cadences
MitchellBitmetric Sep 12, 2026
669844d
Ship the two example wrapper pipelines, and rebuild the dacpac
MitchellBitmetric Sep 12, 2026
b6e63c2
Rebuild dacpac from an LF checkout to match CI's build environment
MitchellBitmetric Sep 12, 2026
1434d2a
temp: upload built dacpac artifact for local recovery
MitchellBitmetric Sep 12, 2026
0ac7e1a
Use the CI-built dacpac and revert the temporary artifact upload
MitchellBitmetric Sep 12, 2026
a16a076
Add reusable fork extensions: deployment pipelines, orchestration/Gol…
MitchellBitmetric Sep 13, 2026
cb748d8
Point setup notebooks at bitmetric-fabric fork instead of upstream ed…
MitchellBitmetric Sep 13, 2026
b512303
Merge remote-tracking branch 'origin/feature/loadgroup-scheduling' in…
MitchellBitmetric Sep 13, 2026
7c24d82
Wire business domain Gold pipelines into PL_FMD_ORCHESTRATION_TEMPLATE
MitchellBitmetric Sep 14, 2026
1911cd8
Auto-fill VAR_GOLD_SHORTCUTS_FMD and add set_variable_library_values …
MitchellBitmetric Sep 14, 2026
8f94a42
Read per-customer configuration from manifest.yaml
MitchellBitmetric Sep 15, 2026
827692e
Drop spark.overrides: one Environment item cannot vary per environment
MitchellBitmetric Sep 15, 2026
ff0b7be
Optional capacity per business domain, and deploy from main
MitchellBitmetric Sep 15, 2026
b9d6323
Fix VAR_GOLD_SHORTCUTS_FMD never being deployed
MitchellBitmetric Sep 15, 2026
e4cec43
Add the GUID guard, with one allowlist entry
MitchellBitmetric Sep 15, 2026
be80bb6
Skip notebook cell ids, and align four item ids with their logicalId
MitchellBitmetric Sep 15, 2026
d499a79
WIP: tooling pipelines via library variables, and an ADF placeholder
MitchellBitmetric Sep 15, 2026
c34f4a9
Finish PL_TOOLING_POST_ASQL_TO_FMD library-variable wiring
MitchellBitmetric Sep 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/guid-guard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: guid-guard

on:
pull_request:
paths:
- 'src/**'
- 'setup/**'
- 'wiki/**'
- 'config/**'
- 'scripts/guid_guard.py'
- '.github/workflows/guid-guard.yml'
push:
branches: [main, dev]
workflow_dispatch:

jobs:
guid-guard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Controleer op niet-gemapte GUID's
run: python scripts/guid_guard.py
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,11 @@ Thumbs.db

# OMC (oh-my-claudecode)
.omc/

# FMD klant-manifest
# Het ingevulde manifest hoort niet in dit template-repo: het bevat de namen,
# capaciteiten en Entra-groep-ID's van een specifieke klant.
# LET OP: in een KLANT-fork moet manifest.yaml juist wel gecommit worden - de
# setup-notebooks halen hem daar via raw.githubusercontent.com op. Haal deze
# regel daar dus weg. manifest.example.yaml blijft altijd in git.
manifest.yaml
154 changes: 23 additions & 131 deletions FMD_BUSINESS_DOMAIN_DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,19 @@ Make sure the following settings are enabled:

### 1. Download deployment assets

Download the deployment notebook from the setup folder to your local machine:
If you already deployed the integration framework, `NB_SETUP_BUSINESS_DOMAINS`
is already in your configuration workspace: the bootstrap notebook places both
setup notebooks at once. In that case skip to step 3.

- `NB_SETUP_BUSINESS_DOMAINS.ipynb` – Automates artifact creation for FMD_FRAMEWORK in Fabric, based on your configuration.
Otherwise download a single file from the `setup` folder:

- `NB_BOOTSTRAP_FMD.ipynb` – fetches the setup notebooks from your fork and places them in the workspace.

### 2. Create a configuration workspace

- Create a new workspace (for example, `FMD_FRAMEWORK_CONFIGURATION`).
- Import the deployment notebook into the workspace (ensure you are in the Fabric Experience):
- `NB_SETUP_BUSINESS_DOMAINS.ipynb`
- Import the bootstrap notebook into the workspace (ensure you are in the Fabric Experience):
- `NB_BOOTSTRAP_FMD.ipynb`

> [!NOTE]
> Make sure you set Spark session timeout to at least 1 hour in the workspace settings/Data Engineering/Jobs.
Expand All @@ -87,138 +90,27 @@ Download the deployment notebook from the setup folder to your local machine:

### 3. Configure deployment settings

Open `NB_SETUP_BUSINESS_DOMAINS.ipynb` and navigate to the configuration cell. Update the following parameters as needed.
All per-customer configuration lives in **`manifest.yaml`** in the root of your
fork — the same file the integration framework uses. Nothing is configured by
editing notebook cells any more.

#### Key configuration parameters
The keys this notebook needs on top of the integration ones:

**Framework settings**
| Key | What it holds |
|---|---|
| `naming.business_domains` | The business domains to create, for example `[FINANCE, SALES]`. Each gets a CODE, DATA, REPORTING and SEMANTIC workspace per environment. |
| `naming.domain_name` | Used to find the integration domain's CONFIG workspace and metadata database, so they are not configured separately here. |
| `environments[].capacity_business_domain` | Optional. The capacity the business-domain workspaces land on, per environment. Left out, they share `environments[].capacity` with the integration layer. |
| `security` | Object IDs of the Entra groups that get access to the business-domain workspaces. |

> [!NOTE]
> Fabric Administrator Role is required to create a domain. Otherwise, disable domain creation in the next step.


**settings**

```python
assign_icons = True # Set to True to assign default icons to workspaces; set to False if you have already assigned custom icons

lakehouse_schema_enabled = True # Set to True if you want to use the lakehouse schema, otherwise set to False

driver = '{ODBC Driver 18 for SQL Server}'# Change this if you use a different driver
overwrite_variable_library = True # By default the Library is overwritten, change this to "False" if you have custom changes
```

**Variable settings**

```python
SourceWorkspaceId=''
SourceLakehouseId='' # Your Gold lakehouse
SourceSchema=''
Shortcut_TargetSchema=''
Shortcut_TargetWorkspaceId=''
Shortcut_TargetLakehouseId='' # Your Silver Lakehouse
```

**Capacity settings**

Specify the unique name for the capacity:

```python
capacity_name_dvlm = 'Name of your capacity'
reassign_capacity= True # Set to False if you don't want to reassign the capacity to an existing workspace in case you set the capacity manually
```

**Domain settings**

Define the name for the Main Domain, and you can add 1 or more business domains.

```python
framework_post_fix= '' # postfix to be added at the end of workspace for example INTEGRATION CODE(D) FMD
if framework_post_fix != '':
framework_post_fix= ' '+ framework_post_fix #If empty leave as is else add a space before for better visibility

##Domains
create_domains= True # If you do not have a Fabric Admin role, you need to set this option to False. For domain creation the Fabric Admin role is needed
business_domain_names= ['FINANCE','SALES'] # Define business domains
domain_contributor_role = {"type": "Contributors","principals": [{"id": "00000000-0000-0000-0000-000000000000","type": "Group"} ]} # Which group(Object ID) can add or remove workspaces to this domain
configuration_database_workspace= 'Name of the workspace of your SQL Server database which holds the configuration database'
configuration_database_name= 'Name of the configuration database'

```

You need to create workspace roles for the different workspaces:
See `manifest.example.yaml` for the full structure, and
[FMD_FRAMEWORK_DEPLOYMENT.md](FMD_FRAMEWORK_DEPLOYMENT.md#4-configure-deployment-settings)
for how to fill in and commit the manifest.

> [!NOTE]
> The id of the User, Group or Service Principal is the Object ID in Microsoft Entra ID. For a Service Principal, you can find the Object ID in the Azure Portal under 'Enterprise applications'. Don't use the Object ID of the App Registration.'

workspace_roles_reporting
workspace_roles_gold

Check the examples below

```python

# Replace placeholder IDs with real Object IDs from Microsoft Entra ID.
# Use "Group", "User" or "ServicePrincipal" for "type" as appropriate.

workspace_roles_code = [
{
"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"},
"role": "Member"
},
{
"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "ServicePrincipal"},
"role": "Contributor"
}
]

workspace_roles_data = [
{
"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"},
"role": "Member"
},
{
"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"},
"role": "Admin"
}
]

workspace_roles_configuration = [
{
"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"},
"role": "Contributor"
}
]



# Optional: business-domain specific role lists
workspace_roles_data_business_domain = [
{"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"}
]

workspace_roles_code_business_domain = [
{"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"}
]

workspace_roles_reporting_business_domain = [
{"principal": {"id": "00000000-0000-0000-0000-000000000000", "type": "Group"}, "role": "Member"}
]
```

**Repo Configuration**

Location of the FMD Framework repository. Unless you have a forked version, do not change these settings. If you want to use another branch, you can change the branch name to your own branch.

```python
#FMD Framework code
##### DO NOT CHANGE UNLESS SPECIFIED OTHERWISE ####
repo_owner = "edkreuk" # Owner of the repository
repo_name = "FMD_FRAMEWORK" # Name of the repository
branch = "main" #"main" is default
folder_prefix = ""
###################################################
```
> Business-domain workspaces share `environments[].capacity` with the integration
> workspaces unless you set `capacity_business_domain` on that environment. Either
> way it is a manifest change, not a notebook change.

### 4. Run the deployment

Expand Down
Loading