Skip to content

DRAFT KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with GraphNode "keyChangingOperation" #18800

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
appchemist wants to merge 4 commits into
base: trunk
Choose a base branch
from
Open

DRAFT KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with GraphNode "keyChangingOperation" #18800

appchemist wants to merge 4 commits into from
+84 −80

Conversation

@appchemist
Copy link
Contributor

@appchemist appchemist commented Feb 4, 2025

That PR is a POC.

For now, I only focused on modifying KStreamImpl.repartitionRequired to replace it with GraphNode.keyChangingOperation.

So I'll be working on it further to make sure it's the right fix and need to write some test code.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@github-actions github-actions bot added triage PRs from the community streams labels Feb 4, 2025
@appchemist
Copy link
Contributor Author

appchemist commented Feb 4, 2025

Hi, @mjsax
If you have a moment, Please take a look

@mjsax
Copy link
Member

mjsax commented Feb 5, 2025

Thanks for this draft -- might take a few days until I find time to take a look.

@github-actions github-actions bot removed the triage PRs from the community label Feb 6, 2025
@appchemist
Copy link
Contributor Author

appchemist commented Feb 17, 2025 •
edited
Loading

Hi, @mjsax
After reuse InternalStreamsBuilder#getKeyChangingParentNode(). Some Tests are failed.
So, I'll looking for these failed tests
I haven't had much time lately, so I haven't looked at it much.

After Analyzing the cause, I'll request the review again

However, I would appreciate it if you could take a look at the following comment when you have time.
#18800 (comment)

@github-actions github-actions bot added the small Small PRs label Feb 22, 2025
@appchemist
Copy link
Contributor Author

appchemist commented Feb 22, 2025 •
edited
Loading

:streams:testAll now succeed.

@ mjsax
If you have a moment, Please take a look.

If the POC is worth applying, I'll resolve the conflict.

@github-actions github-actions bot removed the small Small PRs label Feb 26, 2025
@appchemist
Copy link
Contributor Author

appchemist commented Mar 5, 2025

@mjsax kindly ping

@mjsax
Copy link
Member

mjsax commented Jun 11, 2025

@appchemist -- Sorry for dropping the ball on this work. -- I would have time now, to support you with with PR, if you are still interested.

@appchemist
Copy link
Contributor Author

appchemist commented Jun 12, 2025 •
edited
Loading

@mjsax Of course, I'm still interested.
Thank you for your time.

@appchemist
Copy link
Contributor Author

appchemist commented Jun 15, 2025 •
edited
Loading

I've resolve the conflict.
./gradlew clean :streams:testAll is all passed

mjsax
mjsax reviewed Jun 24, 2025
View reviewed changes
Copy link
Member

@mjsax mjsax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Finally made a pass. Hope the comments/question make sense and help.

@mjsax mjsax mentioned this pull request Sep 3, 2025
Merged
appchemist
appchemist commented Sep 7, 2025
View reviewed changes
@appchemist
Copy link
Contributor Author

appchemist commented Sep 7, 2025 •
edited
Loading

Since the existing commit had conflicts when rebasing onto the latest trunk, I created a new commit instead.

All streams tests pass, but some integration tests occasionally fail.

@github-actions
Copy link

github-actions bot commented Dec 7, 2025

This PR is being marked as stale since it has not had any activity in 90 days. If you
would like to keep this PR alive, please leave a comment asking for a review. If the PR has
merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions bot added the stale Stale PRs label Dec 7, 2025
@appchemist
KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with…
c40b019 
… GraphNode "keyChangingOperation"
@appchemist
KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with…
307e832 
… GraphNode "keyChangingOperation"

- add findNearestParentNodeMatching that performs BFS
- Determine whether to repartition using BFS
@appchemist
KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with…
0c0863a 
… GraphNode "keyChangingOperation"

- To address the StreamsGraphTest.shouldNotThrowNPEWithMergeNodes test failure case
@appchemist
KAFKA-18692: Consider to unify KStreamImpl "repartitionRequired" with…
f5e96c6 
… GraphNode "keyChangingOperation"

- cleanup
@appchemist
Copy link
Contributor Author

appchemist commented Jan 3, 2026 •
edited
Loading

@mjsax happy new year.
If you have time, please help me to review this pr.

Sorry for the late work, I’ve been a bit busy recently.

@appchemist appchemist requested a review from mjsax January 3, 2026 08:01
@github-actions github-actions bot removed the stale Stale PRs label Jan 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mjsax mjsax Awaiting requested review from mjsax

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

streams

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@appchemist @mjsax