Skip to content

Provide database migration job as part of the helm update#508

Open
jchanbcbc wants to merge 20 commits into
stablefrom
feature/enable_db_upgrade
Open

Provide database migration job as part of the helm update#508
jchanbcbc wants to merge 20 commits into
stablefrom
feature/enable_db_upgrade

Conversation

@jchanbcbc

@jchanbcbc jchanbcbc commented Jun 9, 2026

Copy link
Copy Markdown

Description of Change

This PR improves the reliability of database schema upgrades when connected to external MySQL by decoupling the schema migration from the Gateway startup process.

  • During an upgrade, the database schema is now updated first.
  • The new Gateway instance will only spin up after the migration successfully completes.
  • New Gateway instance will no longer execute Liquibase routine of checking for database schema changes and thus will not be blocked on start up due to Liquibase schema lock issues.
  • All of the above are configurable, user can opt-in provided they meet requirements of using Gateway version 11.2.2 or newer.

There are two changes required to implement the solution:

1: Gateway container changes

New system property (-Dgateway.db.schema-update.mode) was introduced in Gateway 11.2.2 that supports four native startup modes:

  1. Default Mode (default): Checks and applies schema updates, then starts the Gateway.
  2. Skip Liquibase Mode (skip): Completely bypasses the Liquibase schema check and starts the Gateway immediately.
  3. Liquibase-Only Mode (liquibase-only): Checks and applies schema updates, then exits cleanly with success (0) without starting the Gateway.
  4. Liquibase-Only with Unlock Mode (liquibase-only-with-unlock): Forcefully releases any existing DATABASECHANGELOGLOCK, applies schema updates, and exits cleanly with success (0).

These options will be used by Helm to execute database schema update job.

2: Helmchart changes

The Helm chart has been updated to include an opt-in pre-upgrade database migration job that utilizes the new liquibase-only mode from the container. Also, new Gateways can be started with gateway.db.schema-update.mode=skip to by-pass the Liquibase db schema check on start up and leave the upgrading of the db schema to the db migration job.

Benefits

  • Cleaner Migration Job Execution: The DB migration job now relies on a native container startup mode that only updates the database schema and exit cleanly without starting up the Gateway. Only after a successful migration, will the new Gateways be able to spin up.

  • Eliminates Stuck Locks Blocking Gateways: By passing the skip mode to new Gateway pods rolling out during an upgrade, they bypass the Liquibase FastCheckService and its related issues.

  • Improved Resilience with Proxies: Isolating the schema update to a single, dedicated migration job allows the upgrade to target a direct primary DB endpoint, avoiding the schema corruption and lock-splitting issues that occur when Liquibase connects through connection-multiplexing DB proxies (like RDS Proxy or ProxySQL).

  • If the pre-upgrade migration job fails, the DATABASECHANGELOGLOCK will remain locked in the database. However, this is mitigated because subsequent Gateway containers can now be configured to use the skip mode, ensuring their startup is not blocked by the stuck lock.

To enable feature, first ensure Gateway to upgrade to is at least Gateway 11.2.2 or later.
Using the latest helm chart, update production-values.yaml:

database: 
  # This pre-upgrade job executes any pending database schema updates and exits. The job executes before the rollout of new Gateways.
  # Ensure the requirements are met before enabling.
  # Requirement: The Gateway must be 11.2.2 or newer which supports applying db schema
  # changes and exiting without starting Gateway.
  # Warning, if this job fails, it may leave the DATABASECHANGELOGLOCK locked in which case  
  # the lock must be manually removed before a retry can be attempted.  
  migrationJob:
    enabled: true
    # Optional: Override the JDBC URL used by the migration job. Recommended to specify the primary writer endpoint
    # to avoid routing through a load balancer or proxy during schema updates. If not set, falls back to database.jdbcURL.
    jdbcUrl: ""

    # Set to true to force release of any stuck Liquibase db schema locks before running the schema updates.
    # Warning: use with caution as another simultaneous db schema update may corrupt the db schema.
    clearLocks: false
    # The maximum duration (in seconds) the job is allowed to run before being terminated.
    activeDeadlineSeconds: 300
  • set database.migrationJob.enabled to true.
  • optionally configure the jdbcUrl property with the destination of the database schema update. To ensure best results, connect directly to the primary writer endpoint rather than routing through a database proxy.
  • look for -Dgateway.db.schema-update.mode under the container's javaArgs section and set to skip.

Console output from running helm upgrade:
`Release "my-gateway" has been upgraded. Happy Helming!
NAME: my-gateway
LAST DEPLOYED: Mon Jun 22 13:47:22 2026
NAMESPACE: jc660465
STATUS: deployed
REVISION: 2
DESCRIPTION: Upgrade complete
TEST SUITE: None
NOTES:
##################################################################################

Success!

##################################################################################

Your gateway deployment has been UPGRADED

##################################################################################

To view the Gateway's services you can use the following command
$ kubectl get svc -n jc660465 | grep gateway

You configured the following ingress hosts

  • dev.ca.com

To learn more about the Gateway Helm Chart check out the following links

Gateway Helm Chart Readme

Thinking in Kubernetes

##################################################################################
IMPORTANT: DB Migration Job Enabled
##################################################################################
A pre-upgrade database migration job has been deployed to safely apply schema updates.
If you have Gateway version 11.2.2 or newer, you can now start your main Gateway deployments with:
javaArgs:
- "-Dgateway.db.schema-update.mode=skip"

This ensures that the Gateway will start up quickly by skipping the liquibase schema check entirely,
regardless of whether the DATABASECHANGELOGLOCK is locked.`

Checklist

  • [] Chart version bumped in Chart.yaml according to semver.
  • [] Variables are documented in the README.md
  • [] Title of the PR starts with chart name (e.g. [charts/gateway])
  • [] If the chart contains a values-production.yaml apart from values.yaml, ensure that you implement the changes in both files

…s Container Gateway 11.2.2 or newer db schema startup options to use.
@jchanbcbc jchanbcbc self-assigned this Jun 10, 2026
jchanbcbc and others added 5 commits June 11, 2026 17:17
…t work with pre-install as it requires resources like license file that hasn't been mounted yet. There is no need to run migrate job on new install as there is no contention.

Also display the DB Migration job enabled note only if it was enabled.
…ility to force unlock when running the db upgrade job. Revised some comments.
@jchanbcbc jchanbcbc changed the title Provide database migration job as part of the helm update. Provide database migration job as part of the helm update Jun 17, 2026
Comment thread charts/gateway/values.yaml Outdated
jdbcUrl: ""
hookWeight: "-5"
# Set to true to force release any stuck Liquibase locks before running the schema updates
clearStuckLocks: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we think of a better naming convention for this?

jchanbcbc added 2 commits June 17, 2026 23:51
Add a timeout for db migration job.
…rl optional and use main jdbcUrl by default.
@jchanbcbc jchanbcbc requested a review from Gazza7205 June 19, 2026 01:01

@emilytzhang emilytzhang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Gazza7205 Gazza7205 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants