sink: retry later when meet unexpected DDL state #3948

wk989898 · 2026-01-07T09:25:44Z

What problem does this PR solve?

What is changed and how it works?

When meeting the unexpected DDL state, instead of immediately failing, CDC now attempts to retry the DDL. This change aims to avoid CDC restart.

Check List

Tests

Unit test
Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Avoid CDC restart when meeting lots of slow DDL.

ti-chi-bot · 2026-01-07T09:25:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lidezhu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gemini-code-assist · 2026-01-07T09:25:59Z

Summary of Changes

Hello @wk989898, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the resilience of the CDC sink by enhancing its DDL error handling mechanism. By marking ErrDDLStateNotFound as a retryable error, the system can now gracefully recover from transient issues where DDL states are not immediately available, thereby preventing disruptive CDC restarts, especially in environments with numerous or slow DDL operations. This change contributes to a more stable and robust data replication process.

Highlights

Enhanced DDL Error Handling: The system now treats ErrDDLStateNotFound as a retryable error, preventing unnecessary CDC restarts when DDL states are unexpectedly not found, particularly during slow DDL operations.
New Error Test File: A new test file, pkg/errors/utils_test.go, has been added to thoroughly test various error utility functions, including the updated IsRetryableDDLError.
Improved DDL Execution Logging: Logging for unexpected DDL states in mysql_writer_ddl.go has been updated to explicitly indicate that a retry will occur, providing clearer diagnostic information.
Refined DDL Execution Error Type: The error returned when DDL execution fails has been changed from ErrMySQLTxnError to ErrExecDDLFailed for better semantic clarity.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

ti-chi-bot · 2026-01-07T09:27:26Z

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456.

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}

gemini-code-assist

Code Review

This pull request correctly makes ErrDDLStateNotFound a retryable error, which will prevent changefeed restarts when encountering slow DDLs. The core logic change is simple and is well-supported by a new test file that improves coverage for error-handling utilities. The pull request also includes some nice refactorings, such as using a more specific error type and updating a log message to reflect the new retry behavior. My review includes a couple of suggestions to improve the quality and readability of the new test cases by using more accurate error messages.

gemini-code-assist · 2026-01-07T09:27:52Z

pkg/errors/utils_test.go

+		{newMysqlErr(tmysql.ErrNoDB, "Error: Duplicate key name 'some_key'"), false},
+		{newMysqlErr(tmysql.ErrParse, "Can't create database"), false},
+		{newMysqlErr(tmysql.ErrAccessDenied, "Access denied for user"), false},
+		{newMysqlErr(tmysql.ErrDBaccessDenied, "Access denied for db"), false},
+		{newMysqlErr(tmysql.ErrNoSuchTable, "table not exist"), false},
+		{newMysqlErr(tmysql.ErrNoSuchIndex, "index not exist"), false},
+		{newMysqlErr(tmysql.ErrWrongColumnName, "wrong column name'"), false},
+		{newMysqlErr(tmysql.ErrDupKeyName, "Duplicate key name 'some_key'"), true},
+		{newMysqlErr(tmysql.ErrPartitionMgmtOnNonpartitioned, "xx"), false},
+		{newMysqlErr(tmysql.ErrNonuniqTable, "xx"), false},
+		{newMysqlErr(tmysql.ErrBadDB, "xx"), false},


The error messages for several test cases in TestIsRetryableDDLError seem to be either placeholders ("xx") or potentially incorrect due to copy-pasting. For example, on line 100, tmysql.ErrNoDB is paired with a message for a duplicate key error.

For better test clarity and correctness, please review and update these messages to be representative of the actual errors being tested. This will make the tests more robust and easier to understand.

gemini-code-assist · 2026-01-07T09:27:52Z

pkg/errors/utils_test.go

+		{newMysqlErr(tmysql.ErrDupKeyName, "Error: Duplicate key name 'some_key'"), false},
+		{newMysqlErr(tmysql.ErrNoDB, "Error: Duplicate key name 'some_key'"), false},
+		{newMysqlErr(tmysql.ErrParse, "Can't create database"), false},


The error messages for these test cases appear to be incorrect, likely due to copy-pasting. For instance, tmysql.ErrNoDB is associated with a 'Duplicate key name' message. To improve test clarity and correctness, please ensure the error messages accurately correspond to the error codes being tested.

ti-chi-bot · 2026-01-07T09:28:07Z

@wk989898: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-check	`62cadf2`	link	true	`/test pull-check`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hongyunyan · 2026-01-11T07:56:03Z

pkg/errors/utils.go

 		return true
 	}

+	if Is(err, ErrDDLStateNotFound) {


Does this error only occur in waitDDLDone queries? If so, it would be better to handle this function directly rather than broadening the scope of the error check.

Actually，I‘m not sure whether this error is return only by getDDLStateFromTiDB, and when will this error really happens?

init

62cadf2

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jan 7, 2026

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 7, 2026

gemini-code-assist bot reviewed Jan 7, 2026

View reviewed changes

hongyunyan reviewed Jan 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sink: retry later when meet unexpected DDL state #3948

sink: retry later when meet unexpected DDL state #3948

wk989898 commented Jan 7, 2026 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

gemini-code-assist bot commented Jan 7, 2026

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 7, 2026

Uh oh!

gemini-code-assist bot Jan 7, 2026

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

hongyunyan Jan 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sink: retry later when meet unexpected DDL state #3948

Are you sure you want to change the base?

sink: retry later when meet unexpected DDL state #3948

Conversation

wk989898 commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

gemini-code-assist bot commented Jan 7, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jan 7, 2026

Uh oh!

hongyunyan Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wk989898 commented Jan 7, 2026 •

edited

Loading