Skip to content

Latest commit

 

History

History
263 lines (248 loc) · 15.3 KB

AGENTS.md

File metadata and controls

263 lines (248 loc) · 15.3 KB

FIWARE Data Space Connector

Overview

Umbrella Helm chart repository that bundles all components of the FIWARE Data Space Connector (DSC) -- identity, DSP (EDC/Rainbow), contract management, marketplace -- into a single deployable artifact. No application code lives here; this repo is packaging and integration glue.

Tech Stack

  • Packaging: Helm 3 (umbrella chart with subchart dependencies).
  • Components deployed: EDC-based IdentityHub (Java 21), fdsc-edc (Java 21), Rainbow DSP (Rust), Keycloak 26.7.x (CloudPirates chart 0.21.19; upstream image docker.io/keycloak/keycloak:26.7.0), Scorpio Broker (Quarkus/Java), TMForum API (Java), Contract Management, Business API Ecosystem, MongoDB, HashiCorp Vault, cert-manager, OpenTelemetry Collector.
  • Build / Release: GitHub Actions (.github/workflows/), helm package + publish to gh-pages. Top-level pom.xml drives local k3s-based integration tests (it/).
  • Test: helm lint, helm template, helm-unittest (charts/data-space-connector/tests/), k3s integration tests under it/.

Project Structure

charts/
  data-space-connector/       # umbrella chart (THE thing)
    Chart.yaml                # dependencies: decentralizedIam, keycloak,
                              # scorpio, tm-forum-api, contract-management,
                              # marketplace, fdsc-edc, vault, did, cert-manager,
                              # opentelemetry-collector
    values.yaml               # ~4190 lines -- every knob lives here
    templates/                # in-chart manifests (see below)
    tests/                    # helm-unittest suites
      tracing_test.yaml       # OTEL tracing tests (33 tests, 361 asserts)
  trust-anchor/               # secondary chart: TIR/TIL, Zalando postgres, and
                              # the consent authority (consent-manager +
                              # consent-facade subcharts, NetworkPolicies).
                              # No values.schema.json; tests in tests/

charts/data-space-connector/templates/
  identityhub-deployment.yaml # EDC IdentityHub (Java, tractusx/identityhub)
  rainbow-deployment.yaml     # Rainbow DSP (Rust, quay.io/wi_stefan/rainbow)
  dsconfig-deployment.yaml    # static /.well-known/data-space-configuration
  dsconfig-cm.yaml / -service.yaml
  identityhub-cm.yaml / -datasource-cm.yaml / -logging.yaml / -ingress.yaml
  identityhub-service.yaml
  participant-registration-job.yaml
  participant-registration.yaml
  mongodb.yaml                # internal MongoDB (when not using operator)
  cert-manager-issuer.yaml, elsi-secret.yaml, issuance-secrets.yaml
  bae-prep-cm.yaml, ca-script.yaml, credentials-script.yaml
  extra-manifests.yaml, realm.yaml
  _helpers.tpl, _realm.tpl

doc/
  README.md                   # docs index
  deployment-integration/     # role-based and scenario-based guides
    observability/README.md   # OTEL tracing guide (architecture, per-component)
  flows/                      # mermaid sequence diagrams
  release-notes/              # per-version changelogs
    9-x.md                    # 9.0.0 breaking changes + 9.1.0 OTEL tracing
  adr/                        # architecture decision records

it/                           # k3s-based integration tests (Maven-driven)
k3s/                          # k3s bootstrap manifests for local dev
  provider.yaml               # 1300 lines, no observability section yet
helpers/                      # reference assets (e.g. Gaia-X trust anchors)
schema/                       # JSON schemas used by components
.github/workflows/            # check.yaml, test.yaml, pre-release.yml,
                              # release.yml, stale-issues.yaml
pom.xml                       # drives local-deploy + integration tests

Build & Test

  • Dependency fetch: helm dependency update charts/data-space-connector
  • Lint: helm lint charts/data-space-connector
  • Render: helm template test charts/data-space-connector -f <values>.yaml
  • Unit tests: helm unittest charts/data-space-connector and helm unittest charts/trust-anchor (both run in CI)
  • Values schema: .github/scripts/values-schema.sh regenerates values.schema.json from values.yaml; --check fails when it is stale (requires helm-unittest plugin, pinned to v0.6.3 in CI)
  • Integration (local k3s, slow): mvn -f pom.xml verify
  • CI entry points: .github/workflows/check.yaml (lint) and .github/workflows/test.yaml (helm-unittest + integration).

Key Conventions

  • Every values.yaml key carries a # -- comment (helm-docs format) so the values table in README.md regenerates cleanly. Keep new values documented the same way.
  • values.schema.json is generated from values.yaml and validates this chart's own keys: a key that is not in values.yaml is refused. Every key a template reads must therefore be declared, with a default the template treats as unset. Regenerate the schema after any change to values.yaml. Add a # @schema block right above a key when its default does not tell the generator enough:
    • an empty default (key:) or one that also takes another type: type: [string, "null"], type: [string, integer], ...;
    • a map that takes arbitrary keys (annotations, labels, resources, security contexts, env maps, ...), and every subchart's key: type: object, additionalProperties: true and properties: {}. A value removed from under a subchart's key, which the schema cannot refuse, gets a fail in templates/removed-values.yaml.
  • Feature toggles follow <component>.enabled and {{- if ... }} gates wrap the entire template file.
  • Container env blocks always end with three extension points in this order: env: (map), envValueFrom: (map), envFrom: (secrets/ configmaps lists). New env vars should respect that pattern and not clobber user-provided maps -- append, never overwrite.
  • Subchart-specific knobs live under the subchart's alias (fdsc-edc:, scorpio:, keycloak:, contract-management:, marketplace:, decentralizedIam:, did:, vault:, cert-manager:, opentelemetry-collector:). The umbrella chart only overrides values; it does not template the subcharts' manifests.
  • Pod security context is strict (runAsNonRoot, readOnlyRootFilesystem, drop ALL capabilities). Any new container must match that.
  • Chart version (charts/data-space-connector/Chart.yaml version:) follows semver; bump minor for additive features.
  • Comment density: values keys get one # -- line (two-three when a contract needs it); templates explain why in {{- /* ... */}} blocks above non-obvious logic; helm-unittest suites open with a short header comment and one it: per behaviour. Java IT classes carry one-line Javadoc on public constants and step methods. Match that, no more.

Pod Scheduling Fields

  • The chart templates 11 workloads of its own (6 Deployments, 5 Jobs). Pods that come from subcharts are configured by overriding the subchart's own values, never by templating them here -- see the subchart rule above.
  • Deployments carry the full block: nodeSelector, affinity, tolerations, topologySpreadConstraints, priorityClassName. Jobs carry priorityClassName only; spreading a single-pod Job is meaningless.
  • Always gate with {{- with }}, never a bare if plus interpolation. An unset value ("", [], {}) renders nothing, so upgrading an existing release is a no-op. tests/scheduling_test.yaml pins that behaviour.
  • Field indentation is 6 spaces with nindent 8 in every one of the 11 files, Jobs included: their podSpec is spec.template.spec, same depth as a Deployment's. Watch out for backoffLimit, which sits at 2 spaces on the Job spec -- appending after it puts the field on the wrong object.
  • Each workload hangs off the key it already uses for the rest of its config: identityhub, identityhub.bootstrap (shared by the bootstrap and rotation Jobs), rainbow, dataSpaceConfig, statusListServer, statusListServer.redis, registration, tracing (both otel helper Jobs), vault.unattended.
  • The last one sits inside a subchart's values namespace. That is deliberate: vault-unsealer-deployment.yaml reads $p := .Values.vault.unattended. The subchart does not declare the key and ships no additionalProperties: false schema, so the extra key is inert for it.
  • status-list-server-redis.yaml holds a Deployment and a Service in one file. The fields belong to the Deployment, before the ---.
  • CRs are not templated workloads: scheduling for mongodb.yaml goes through managedMongo.spec.statefulSet, and for trust-anchor/postgres.yaml through managedPostgres.config (the Zalando CR has its own podPriorityClassName).
  • identityhub-rotation-job.yaml is deliberately not a Helm hook and its name is pinned by bootstrap.rotation.instance. Changing identityhub.bootstrap.priorityClassName while a completed rotation Job of the same name exists fails the upgrade with "field is immutable" until the instance is bumped. The other five Jobs are hooks with before-hook-creation, so they are recreated.

Documentation

  • Every architectural decision should be documented in the ADR format in the folder doc/adr
  • Every new/modified feature should be documented in the README.md or its corresponding markdown document in doc/

OpenTelemetry Tracing (from ticket-28)

The chart already has a full OTEL tracing integration (chart version 9.1.0):

  • Global tracing: block in values.yaml (line ~2731) controls OTEL_* env vars on all workloads.
  • opentelemetry-collector subchart (v0.152.0) deploys an in-cluster Collector; its config lives in values.yaml under opentelemetry-collector.config: (line ~2806).
  • _helpers.tpl has six helpers: dsc.otel.defaultEndpoint, dsc.otel.endpoint, dsc.otel.resourceAttributes, dsc.otel.env, plus per-subchart variants (dsc.otel.scorpio.envList, dsc.otel.fdscEdc.envList, dsc.otel.keycloak.secretName, etc.).
  • The Collector currently uses the subchart's built-in ConfigMap (opentelemetry-collector.configMap.create: true is the default) with a debug exporter only. Commented-out examples for Tempo, Jaeger, and Honeycomb exporters are in values.yaml (line ~2850).
  • The subchart supports configMap.create: false + configMap.existingName: "<name>" to use a custom ConfigMap; the existingName field is tpl-rendered so template expressions work. The ConfigMap must use the key relay for the collector config YAML.
  • Tracing tests live in charts/data-space-connector/tests/tracing_test.yaml.

Keycloak / OID4VCI wallet issuance (10.x)

10.x migrated Keycloak to the CloudPirates chart + KC 26.6.x/26.7.x with the post-26.4 OID4VCI model. Hard-won gotchas (full detail in doc/release-notes/10-x.md):

  • Image: since 10.4.0 the chart runs the upstream default docker.io/keycloak/keycloak:26.7.0 (CloudPirates chart 0.21.19) and sets no keycloak.image override. The old quay.io/seamware/keycloak:26.6.3 fork is obsolete — 26.7.0 carries both the Liquibase changeset 26.7.0-verifiable-credential (fixes column ... version does not exist on a reused pre-26.4 DB) and the OID4VCI QR-endpoint fix.
  • Keystore path: since KC 26.6.4 the java-keystore realm key provider only reads keystores from /opt/keycloak/data/<realm-name>/. Both the Keycloak container's extraVolumeMounts entry and keycloak.signingKey.storePath (and elsi.storePath) must sit inside that folder; the init container may keep writing to its own /did-material mount of the same emptyDir.
  • Credential encryption: the issuer advertises request + response encryption. Wallets that encrypt must set the JWE kid and must not send a top-level alg in credential_response_encryption; the EUDI wallet needs eudi-lib-ios-openid4vci-swift ≥ 0.40.0. Lissi does not encrypt (unaffected). Response encryption is not disableable via config (RSA token key always offers RSA-OAEP).
  • DCQL: jwt_vc_json uses meta.type_values (array of string arrays); sd-jwt uses meta.vct_values (flat array); credential_sets must be non-null.
  • Realm import: KC does not re-import an existing realm — recreate it to apply keycloak.realm.* changes. Bitnami→CloudPirates upgrades need the keycloak StatefulSet deleted (immutable-field change).
  • Wallet preset + per-user VCs: keycloak.realm.wallets (Lissi/EUDI clients); wallets.issueCredentialsToUsers: true auto-assigns realm VCs to declared users.

Consent Management (ticket-109)

Migrated from the Prometheus-X consent-manager (Node, MongoDB, shared consent key, authority APISIX facade routes) to dsc-consent-manager (Java, PostgreSQL, validates VCVerifier JWTs itself; chart oci://quay.io/fiware/helm-charts/dsc-consent-manager, API base /v1, ports 8080 API / 9090 management).

  • Pieces: trust-anchor chart (manager, authority consent-facade, NetworkPolicies, setup Job, DB); DSC chart only wires the provider side (consent-facade, consent-owner-resolver) and has no consentManagement key of its own.
  • The consent-manager's database comes from the Zalando operator (managedPostgres.config.databases), which creates it on every sync, on a fresh cluster and an existing volume alike. The setup Job only owns the two Secrets (consent-manager-env, consent-manager-signing-keys); its ServiceAccount and RBAC are optional (consentManagement.setup.serviceAccount / .rbac).
  • k3s scenario: k3s/consent-trust-anchor.yaml (2nd DSC release consent-authority in ns trust-anchor: keycloak issuer, VCVerifier service consent-manager), k3s/consent-trust-anchor-components.yaml (trust-anchor chart values), k3s/consent-provider.yaml (APISIX consent-filter plugin, image quay.io/seamware/consent-plugin). Run: mvn clean deploy -Pconsent; IT: mvn clean integration-test -Ptest,consent,consent-test.
  • IT: it/src/test/java/.../components/Consent{Environment,StepDefinitions}.java, it/src/test/resources/it/consent_management.feature (tag @consent).
  • Docs: doc/CONSENT_MANAGEMENT.md, ADR doc/adr/0008 (which supersedes 0001-0005) + index doc/adr/README.md.
  • The consent-filter plugin source is NOT in this repo; only its tag and route config live here.

Important Files

  • charts/data-space-connector/Chart.yaml -- dependency graph (current version 11.0.0).
  • charts/data-space-connector/values.yaml -- ~4190-line config surface; top-level sections start at these approximate line numbers: decentralizedIam (26), scorpio (291), keycloak (420), tm-forum-api (1441), rainbow (1820), contract-management (1911), dataSpaceConfig (2079), managedMongo (2133), marketplace (2152), fdsc-edc (2362), identityhub (2973), tracing (3752), opentelemetry-collector (3878), consent-facade (4134), consent-owner-resolver (4161).
  • charts/trust-anchor/Chart.yaml -- trust-anchor chart (version 2.0.0).
  • charts/data-space-connector/templates/_helpers.tpl -- shared Helm partials (dsc.labels, dsc.otel.*, etc.).
  • charts/data-space-connector/templates/identityhub-deployment.yaml -- reference for how env/volumes/initContainers/OTEL agent are wired.
  • charts/data-space-connector/templates/rainbow-deployment.yaml -- simpler deployment with init container pattern.
  • pom.xml + it/ -- integration test harness.
  • doc/release-notes/11-x.md -- follow existing format when adding entries.
  • doc/deployment-integration/observability/README.md -- OTEL tracing docs.