Skip to content

Automatic detection and cleanup of removed topics from consumer groups #854

@karthikraina32

Description

@karthikraina32

Description:
Problem
Currently, when a consumer group removes topics from its subscription list, Burrow continues to track and report these removed topics as errors (stopped partitions) until either:

  • The Burrow service is manually restarted, or
  • The /v3/kafka/(cluster)/consumer/(group) DELETE endpoint is manually called

This creates operational overhead and can lead to false alerts in monitoring systems.

Current Workaround:
As mentioned in #14, the current solutions require manual intervention:

  1. Restart Burrow to refresh consumer group configurations
  2. Use the DELETE API endpoint to manually drop the consumer group

Proposed Enhancement:
Implement automatic detection of topic removal from consumer groups by:

  1. Periodic reconciliation: Burrow should periodically compare the list of topics currently consumed by a group (from Kafka metadata) against the topics it's tracking internally
  2. Topic removal detection: When a topic is no longer present in the consumer group's subscription but still exists in Burrow's internal state, automatically remove it from tracking
  3. Configurable interval: Add a configuration parameter (e.g., topic-reconciliation-interval) to control how frequently this check occurs (default: 5-10 minutes)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions