Provide database migration job as part of the helm update

charts/gateway/Chart.yaml

@@ -1,8 +1,8 @@
 apiVersion: v2
-appVersion: "11.2.1"
+appVersion: "11.2.2"
 description: This Helm Chart deploys the Layer7 Gateway in Kubernetes.
 name: gateway
-version: 3.1.1
+version: 3.1.2
 type: application
 home: https://github.com/CAAPIM/apim-charts
 maintainers:

charts/gateway/README.md

@@ -85,6 +85,7 @@ Helm Version    Supported Kubernetes Versions
 * [Shared State Provider Configuration](#shared-state-provider-config)
 * [OpenTelemetry Configuration](#opentelemetry-configuration)
 * [Database Configuration](#database-configuration)
+* [Database Migration Job](#database-migration-job-pre-upgrade-schema-updates)
 * [MySQL StatefulSet](#mysql-statefulset-developmenttesting-only)
 * [Cluster-Wide Properties](#cluster-wide-properties)
 * [Enable DualStack(IPv4/IPv6)](#enable-dualstack)
@@ -339,9 +340,8 @@ management:
 
 ### OTK Compatibility with Gateway 11.2
 These below information is relevant for those who are upgrading their Gateway to version 11.2 and utilizing the OAuth Toolkit (OTK)
-1. **OTK 4.6.4 & OTK 4.7.0 ** are presently the only versions that provides seamless support for Gateway versions 11.2 & greater
-2. Upgrading to 4.6.4 & 4.7.0 should be tested & validated in a lower environment prior to production rollout
-3. For older versions (< OTK 4.6.4), the below steps have to be followed to ensure smooth transition to Gateway 11.2
+1. **OTK 4.6.4** is presently the only version that provides seamless support for Gateway 11.2
+2. For older versions (< OTK 4.6.4), the below steps have to be followed to ensure smooth transition to Gateway 11.2
    * After upgrading Gateway to 11.2 If there is a necessity to install or upgrade to OTK version 4.6.3, 4.6.2, 4.6.1, or 4.6.0, please ensure to update the OTK image tag to use one of the below tags corresponding to the specific version being deployed:
        * 4.6.0.1
        * 4.6.1.1
@@ -1628,6 +1628,99 @@ mysql:
 
 [Back to Additional Guides](#additional-guides)
 
+### Database Migration Job (Pre-Upgrade Schema Updates)
+
+Gateway 11.2.2 introduces an opt-in `pre-upgrade` Kubernetes Job that applies Liquibase database schema changes before the new Gateway pods roll out. By running the schema update once, in a dedicated job, before any Gateway pod starts, it avoids multiple pods racing to acquire the `DATABASECHANGELOGLOCK` simultaneously — reducing the risk of contention, stuck locks, and schema update failures that can block pods from starting during a rolling upgrade. The job also supports specifying a dedicated JDBC URL (for example, a primary writer endpoint) so that schema changes are applied directly to the primary database node, bypassing read replicas or load-balancing proxies that could route writes incorrectly.
+
+> **Requirements:** Gateway must be connected to an external MySQL database. The target Gateway image must be **11.2.2 or newer**.
+
+> **Important — for upgrades only:** The `db-migration` job is a `pre-upgrade` Helm hook. It runs **only during `helm upgrade`**, never during `helm install`. Likewise, `-Dgateway.db.schema-update.mode=skip (which starts up Gateway container without Liquibase` must **not** be set during a fresh installation — if Liquibase is skipped on first install, the `ssg` database schema will never be populated and the Gateway will fail to start. Only add `skip` mode after the schema has been fully initialised by either a previous `helm install` (default mode) or a successful migration job.
+
+#### How it works
+
+When `database.migrationJob.enabled: true`, a short-lived `db-migration` Job pod is created as a Helm `pre-upgrade` hook. It runs the Gateway container image configured with a special startup mode that applies all pending schema changes and then exits — without starting the Gateway JVM. Once the Job completes successfully, Helm proceeds to roll out the main Gateway Deployment.
+
+The main Gateway pods can be configured with `javaArgs: ["-Dgateway.db.schema-update.mode=skip"]` so they bypass the Liquibase check entirely and start immediately without touching the database lock. This feature requires 11.2.2 or newer.
+
+#### Fresh install vs upgrade
+
+
+| Operation                   | `config.javaArgs` — `-Dgateway.db.schema-update.mode` | Migration job (`database.migrationJob.enabled`) | Expected behaviour                                                              |
+| --------------------------- | ------------------------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------------------- |
+| `helm install` (first time) | Not set — Gateway runs Liquibase on startup (default)  | Not applicable (`pre-upgrade` hook only)         | Gateway populates the `ssg` schema on first boot — **correct**                  |
+| `helm upgrade`              | `skip` — Gateway bypasses Liquibase on startup         | `true` — migration job applies schema changes first | Migration job populates schema and exits, Gateway pods start fast — **correct** |
+
+> **Warning:** Do not set `-Dgateway.db.schema-update.mode=skip` during `helm install`. The migration job is a `pre-upgrade` hook and does not run on install, so if Liquibase is skipped the `ssg` schema is never populated and the Gateway will fail to start.
+
+
+#### Upgrade workflow
+
+Configure both the migration job and `-Dgateway.db.schema-update.mode=skip` in your `values.yaml` together, then run `helm upgrade` once. The migration job runs as a `pre-upgrade` hook — it applies all pending schema changes and exits before any Gateway pod starts. The new Gateway pods, already configured to bypass Liquibase, can then start immediately without competing for the database lock.
+
+```yaml
+database:
+  enabled: true
+  create: false
+  jdbcURL: jdbc:mysql://myprimaryserver:3306/ssg
+
+  migrationJob:
+    enabled: true
+    # Optional: specify the primary writer endpoint directly.
+    # If omitted, falls back to database.jdbcURL above.
+    jdbcURL: "jdbc:mysql://myprimaryserver:3306/ssg"
+
+config:
+  javaArgs:
+    - "-Dgateway.db.schema-update.mode=skip"
+    # ... your other javaArgs
+```
+
+```bash
+helm upgrade my-release layer7/gateway -f values.yaml
+```
+
+Helm runs the `db-migration` job first. Once it completes successfully, Helm rolls out the new Gateway pods. Because `skip` mode is already set, the pods bypass Liquibase and start immediately.
+
+Once enabled, you can leave `database.migrationJob.enabled: true` permanently. If there are no pending schema changes, the job completes in seconds and exits cleanly, so there is no harm in running it on every upgrade.
+
+#### Recovering from a stuck lock
+
+If a previous upgrade left the `DATABASECHANGELOGLOCK` locked (e.g. due to a crashed Gateway pod), set `clearLocks: true`. The migration job will forcefully release the stuck lock before applying schema changes.
+
+```yaml
+database:
+  migrationJob:
+    enabled: true
+    clearLocks: true
+```
+
+> **Warning:** Use `clearLocks: true` with caution. Forcefully releasing the lock while another process is actively updating the schema can corrupt the database.
+
+#### Configuration
+
+
+| Parameter                                     | Description                                                                                                                                                                    | Default |
+| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
+| `database.migrationJob.enabled`               | Enable the pre-upgrade schema migration Job. For upgrades only — do not enable on `helm install`.                                                                              | `false` |
+| `database.migrationJob.jdbcURL`               | JDBC URL for the migration job. Recommended to point at the primary writer endpoint directly to bypass load balancers or proxies. Falls back to `database.jdbcURL` if not set. | `""`    |
+| `database.migrationJob.clearLocks`            | Forcefully release any stuck Liquibase locks before applying schema changes                                                                                                    | `false` |
+| `database.migrationJob.activeDeadlineSeconds` | Maximum time (in seconds) the job pod is allowed to run before being terminated                                                                                                | `300`   |
+
+
+#### Retry behaviour
+
+The Job is configured with `backoffLimit: 1`. If the first pod fails or times out, Kubernetes creates exactly one retry pod. If the retry also fails, the Job is marked `Failed` and Helm aborts the upgrade, leaving the existing Gateway pods untouched.
+
+To investigate a failed migration, retrieve the job logs to identify the failing changeset:
+
+```bash
+kubectl logs -n <namespace> -l job-name=<release-name>-db-migration --tail=200
+```
+
+Fix the root cause (for example, a missing MySQL privilege) and re-run `helm upgrade`. If the migration continues to fail, contact [Broadcom Support](https://support.broadcom.com) for assistance.
+
+[Back to Additional Guides](#additional-guides)
+
 ### Cluster Wide Properties
 You can specify cluster-wide properties in values.yaml, you can also use the [bundle](#bundle-configuration) to load your own Gateway Bundles.
 

charts/gateway/production-values.yaml

@@ -216,6 +216,7 @@ config:
     - -Djava.util.logging.config.file=/opt/SecureSpan/Gateway/node/default/etc/conf/log-override.properties
     - -Dcom.l7tech.server.pkix.useDefaultTrustAnchors=true
     - -Dcom.l7tech.security.ssl.hostAllowWildcard=true
+    - -Dgateway.db.schema-update.mode=default
     # - -Djava.net.preferIPv4Stack=false
     # - -Djava.net.preferIPv6Addresses=true
   log:
@@ -648,6 +649,24 @@ database:
   # severe/warning/info/fine(debug)/off
   liquibaseLogLevel: "off"
   name: ssg
+  # This pre-upgrade job executes any pending database schema updates and exits. The job executes before the rollout of new Gateways.
+  # Ensure the requirements are met before enabling.
+  # Requirement: The Gateway must be 11.2.2 or newer which supports applying db schema changes and exiting without starting Gateway.
+  # Warning, if this job fails, it may leave the DATABASECHANGELOGLOCK locked in which case  
+  # the lock must be manually removed before a retry can be attempted.
+  migrationJob:
+    # Opt-in: set to true only when running helm upgrade against a Gateway 11.2.2+ image.
+    # This job does not run on helm install (it is a pre-upgrade hook only).
+    enabled: false
+    # Optional: Override the JDBC URL used by the migration job. Recommended to specify the primary writer endpoint
+    # to avoid routing through a load balancer or proxy during schema updates. If not set, falls back to database.jdbcURL.
+    jdbcUrl: ""
+
+    # Set to true to force release of any stuck Liquibase db schema locks before running the schema updates.
+    # Warning: use with caution as another simultaneous db schema update may corrupt the db schema.
+    clearLocks: false
+    # The maximum duration (in seconds) the job is allowed to run before being terminated.
+    activeDeadlineSeconds: 300
 
 ## If loading a TLS Key/Pair
 # This key will become the default ssl key. Can only have one.

charts/gateway/release-notes.md

@@ -7,6 +7,21 @@ The Layer7 API Gateway is now running with Java 21 with the release of v11.2.0.
 
 If you use Policy Manager, you will need to update to v11.2.0.
 
+## 3.1.2 Database Migration Job (Pre-Upgrade Schema Updates)
+
+Requires Gateway image **11.2.2 or newer**.
+
+- Added an opt-in `pre-upgrade` Kubernetes Job (`database.migrationJob`) that applies Liquibase database schema changes before the new Gateway pods roll out, avoiding multiple pods racing to acquire the `DATABASECHANGELOGLOCK` simultaneously during rolling upgrades.
+- The migration job supports a dedicated `jdbcUrl` override, allowing schema changes to be applied directly to the primary writer endpoint and bypassing read replicas or load-balancing proxies.
+- Added `database.migrationJob.clearLocks` option to forcefully release a stuck `DATABASECHANGELOGLOCK` before running schema updates, enabling recovery from a previously failed migration.
+- Added four Gateway startup modes via `-Dgateway.db.schema-update.mode` in `config.javaArgs`:
+  - `default` — runs Liquibase then starts the Gateway JVM (original behaviour, unchanged)
+  - `skip` — bypasses Liquibase and starts the Gateway JVM immediately (use with the migration job on upgrade, not for helm install)
+  - `liquibase-only` — runs Liquibase and exits without starting the Gateway JVM (used by the migration job)
+  - `liquibase-only-with-unlock` — force-releases the Liquibase lock, runs Liquibase, and exits (used by `clearLocks`)
+- The migration job is disabled by default (`database.migrationJob.enabled: false`) and only takes effect during `helm upgrade` — it does not run on `helm install`.
+- See [Database Migration Job](./README.md#database-migration-job-pre-upgrade-schema-updates) for full configuration and upgrade workflow.
+
 ## 3.1.1 OTK 4.7.0 Released
 - The default image tag in values.yaml and production-values.yaml for OTK updated to 4.7.0.
   - otk.job.image.tag: 4.7.0

charts/gateway/templates/NOTES.txt

@@ -26,3 +26,17 @@ Gateway Helm Chart Readme
 
 Thinking in Kubernetes
 - https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/congw-10-1/learning-center/thinking-in-kubernetes.html#thinkingk8s
+
+{{- if and .Values.database.migrationJob.enabled .Release.IsUpgrade }}
+
+----------------------------------------------------------------------------------
+IMPORTANT: DB Migration Job Enabled
+----------------------------------------------------------------------------------
+A pre-upgrade database migration job has been deployed to safely apply schema updates.
+If you have Gateway version 11.2.2 or newer, you can now start your main Gateway deployments with:
+  javaArgs:
+    - "-Dgateway.db.schema-update.mode=skip"
+
+This ensures that the Gateway will start up quickly by skipping the liquibase schema check entirely,
+regardless of whether the DATABASECHANGELOGLOCK is locked.
+{{- end }}

charts/gateway/templates/db-migration-job.yaml

@@ -0,0 +1,104 @@
+# Copyright (c) 2026 Broadcom Inc. and its subsidiaries. All Rights Reserved.
+{{- if .Values.database.migrationJob.enabled }}
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: {{ template "gateway.fullname" . }}-db-migration
+  annotations:
+    chartversion: {{ .Chart.AppVersion | quote }}
+    "helm.sh/hook": pre-upgrade
+    "helm.sh/hook-weight": "-5"
+    {{- if .Values.database.migrationJob.annotations }}
+    {{- range $key, $val := .Values.database.migrationJob.annotations }}
+    {{ $key }}: "{{ $val }}"
+    {{- end }}
+    {{- end }}
+  labels:
+    app: {{ template "gateway.fullname" . }}
+    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+    release: {{ .Release.Name }}
+    heritage: {{ .Release.Service }}
+spec:
+  backoffLimit: 1
+  template:
+    metadata:
+      labels:
+        app: {{ template "gateway.fullname" . }}-db-migration
+        release: {{ .Release.Name }}
+      {{- if .Values.database.migrationJob.podAnnotations }}
+      annotations: {{- toYaml .Values.database.migrationJob.podAnnotations | nindent 8 }}
+      {{- end }}
+    spec:
+      {{- if .Values.database.migrationJob.activeDeadlineSeconds }}
+      activeDeadlineSeconds: {{ .Values.database.migrationJob.activeDeadlineSeconds }}
+      {{- end }}
+      serviceAccountName: {{ include "gateway.serviceAccountName" . }}
+      {{- if .Values.podSecurityContext }}
+      securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }}
+      {{- end }}
+      {{- if .Values.imagePullSecret.enabled }}
+      imagePullSecrets:
+        - name: {{ template "gateway.imagePullSecret" . }}
+      {{- end }}
+      containers:
+        - name: db-migration
+          image: {{.Values.image.registry}}/{{.Values.image.repository}}:{{.Values.image.tag}}
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          {{- if .Values.containerSecurityContext }}
+          securityContext: {{- toYaml .Values.containerSecurityContext | nindent 12 }}
+          {{- end }}
+          envFrom:
+            - configMapRef:
+                name: {{ template "gateway.fullname" . }}-configmap
+            - secretRef:
+                name: {{ template "gateway.secretName" . }}
+          env:
+            {{- if .Values.database.migrationJob.jdbcUrl }}
+            - name: SSG_DATABASE_JDBC_URL
+              value: {{ .Values.database.migrationJob.jdbcUrl | quote }}
+            {{- else if .Values.database.jdbcURL }}
+            - name: SSG_DATABASE_JDBC_URL
+              value: {{ .Values.database.jdbcURL | quote }}
+            {{- end }}
+            - name: EXTRA_JAVA_ARGS
+              {{- if .Values.database.migrationJob.clearLocks }}
+              value: "-Dgateway.db.schema-update.mode=liquibase-only-with-unlock"
+              {{- else }}
+              value: "-Dgateway.db.schema-update.mode=liquibase-only"
+              {{- end }}
+          volumeMounts:
+            - name: {{ template "gateway.fullname" . }}-license-xml
+              mountPath: /opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
+              subPath: license.xml
+            {{- if (not .Values.disklessConfig.enabled) }}
+            {{- if (not .Values.disklessConfig.setSecretByInitContainer) }}
+            - name: {{ template "gateway.fullname" . }}-node-properties
+              mountPath: /opt/SecureSpan/Gateway/node/default/etc/conf/node.properties
+              subPath: node.properties
+            {{- else }}
+            - name: shared-secret
+              mountPath: /opt/SecureSpan/Gateway/node/default/etc/conf/node.properties
+              subPath: node.properties
+            {{- end }}
+            {{- end }}
+      volumes:
+        - name: {{ template "gateway.fullname" . }}-license-xml
+          secret:
+            secretName: {{ template "gateway.license" . }}
+            items:
+            - key: license
+              path: license.xml
+      {{- if and (not .Values.disklessConfig.enabled) (not .Values.disklessConfig.setSecretByInitContainer) }}
+        - name: {{ template "gateway.fullname" . }}-node-properties
+          {{- if .Values.disklessConfig.existingSecret.csi }}
+          csi: {{ toYaml .Values.disklessConfig.existingSecret.csi | nindent 12 }}
+          {{- else }}
+          secret:
+            secretName: {{ template "gateway.node.properties" . }}
+            items:
+            - key: node.properties
+              path: node.properties
+          {{- end }}
+      {{- end }}
+      restartPolicy: "Never"
+{{- end }}

charts/gateway/values.yaml

@@ -646,6 +646,24 @@ database:
   # severe/warning/info/fine(debug)/off
   liquibaseLogLevel: "off"
   name: ssg
+  # This pre-upgrade job executes any pending database schema updates and exits. The job executes before the rollout of new Gateways.
+  # Ensure the requirements are met before enabling.
+  # Requirement: The Gateway must be 11.2.2 or newer which supports applying db schema changes and exiting without starting Gateway.
+  # Warning, if this job fails, it may leave the DATABASECHANGELOGLOCK locked in which case
+  # the lock must be manually removed before a retry can be attempted.
+  migrationJob:
+    # Opt-in: set to true only when running helm upgrade against a Gateway 11.2.2+ image.
+    # This job does not run on helm install (it is a pre-upgrade hook only).
+    enabled: false
+    # Optional: Override the JDBC URL used by the migration job. Recommended to specify the primary writer endpoint
+    # to avoid routing through a load balancer or proxy during schema updates. If not set, falls back to database.jdbcURL.
+    jdbcUrl: ""
+
+    # Set to true to force release of any stuck Liquibase db schema locks before running the schema updates.
+    # Warning: use with caution as another simultaneous db schema update may corrupt the db schema.
+    clearLocks: false
+    # The maximum duration (in seconds) the job is allowed to run before being terminated.
+    activeDeadlineSeconds: 300
 
 ## If loading a TLS Key/Pair
 # This key will become the default ssl key. Can only have one.