Skip to content

Kafka external stream / Schema Registry docs: consumer-group behavior, credentials in the URL, schema ID changes, Protobuf column inference #704

Open

Description

@kuiliu88

Summary

While QA-testing Kafka external streams with Schema Registry against Timeplus Enterprise 3.5.1-rc.0 and 3.4.1 (docs main at 4a9a696), I found five behaviours that are correct, or deliberate, but are not described in the docs. Each one led to a confusing symptom (a 401, an empty stream, no consumer group on the broker, columns that cannot show a missing field, a streaming query that stops during a registry outage). Items 1 to 4 have wording in a PR; item 5 does not yet.

1. No Kafka consumer group is used (pages: kafka-source, kafka-sink)

A query or materialized view reading a Kafka external stream registers no consumer group and commits no offsets: rpk group list stays empty before, during and after a streaming read and an MV. The properties group.id, group.instance.id, enable.auto.commit, enable.auto.offset.store, auto.offset.reset and auto.commit.interval.ms are rejected in properties with Code 471 Unsupported property <name> (checked live on 3.5.1-rc.0; the same names are in the unsupported lists in Kafka.cpp from 2.8.49 through 3.2.15, 3.3.3, 3.3.4, 3.4.1 and develop). The docs describe seek_to for choosing the start position but never say that no group / offset is committed, so users look for the consumer in broker tools and for lag.

2. Credentials in the Schema Registry URL are not used (pages: kafka-schema-registry, kafka-source #kafka_schema_registry_credentials)

kafka_schema_registry_credentials (username:password) is the only way to authenticate. With https://user:password@host in kafka_schema_registry_url and no credentials setting, no Authorization header is sent (confirmed with a logging proxy, auth_header_present=false) and the registry answers 401. The docs neither allow nor forbid the URL form. A password may contain further : (split at the first one; tested with pa:ss:wd).

3. Schema IDs: what happens after a registry rebuild (page: kafka-schema-registry)

Messages are matched by the schema ID in the Confluent header, not by schema text. After the subject is deleted and the same schema is registered again (new ID), the old messages keep the old ID. Tested on 3.5.1-rc.0 and 3.4.1 with the identical schema text re-registered under new IDs: a consume_schema_strategy='single' stream returned 0 rows with no error (documented as "Non-matching messages are silently skipped", but the consequence is not spelled out), while an all stream kept decoding from its cache and a fresh all/raw stream failed with 404 Schema <id> not found.

4. Protobuf column inference never creates nullable columns (page: data-formats, Column Inference)

For optional string y / optional int32 w the inferred columns are plain string / int32; a missing field reads as '' / 0. Declaring nullable(...) columns explicitly works: a missing field is NULL, a field sent with its default value is that value (tested on 3.5.1-rc.0 with hand-built frames). The docs say nothing about nullability in inference (nullable appears only for Avro unions).

5. A Schema Registry outage while a stream is running (pages: kafka-schema-registry, kafka-source)

The docs do not say what depends on the registry once a stream is running. Tested on 3.5.1-rc.0 and 3.4.1 by stopping the registry (connection refused) while a streaming query and a materialized view (MV) were reading a single Protobuf stream, with the registry cache cold at the start of the run:

During the outage 3.5.1-rc.0 3.4.1
messages with the stream's own, already-known schema ID decoded and delivered, by the running query and by the MV same
a NEW query on the stream (table() or streaming) fails in about 1 s with Connection refused: while fetching subject versions for <subject> same
a message with a schema ID this stream has not looked up before (for example another subject's ID on a mixed topic) the running streaming query aborts with Connection refused ... Failed to parse message topic=... offset=... and stays stopped after the registry is back the running query keeps going; the message is skipped without a registry call
the same message, seen by an MV Background runtime error: ... Connection refused in system.timeplusd_err_log; after the registry is back the MV continued within 5 s with no restart, and the sink held every id once (no loss, no duplicate) no outage error logged for the MV; continued

Why (source reading): at v3.5.1-rc.0 and v3.4.2-rc.2, KafkaSchemaRegistry::isSchemaIdOfSubject checks whether an ID belongs to the stream's subject, caches a successful answer, and deliberately propagates a failed lookup instead of caching "not a member" (code comment: "a registry outage can never be stored as 'not a member'"). The function is not in v3.4.1, where the single filter reads one ID at start. So on these releases a stream over a topic that contains several schema IDs depends on the registry for each ID it has not seen yet, even for messages it will skip. Messages with the stream's own ID kept flowing during the outage (observed); that IDs seen once stay safe afterwards is inferred from the cache behaviour above, not tested separately.

What a user would want from the docs: a short note under the single strategy that a message with a not-yet-seen ID needs a registry call, that an outage then stops a running ad-hoc query (restart it after the registry is back) while an MV retries by itself, and that the lookup result is kept per registry URL and credentials (inferred from a second run that reused the same URL and credentials and skipped the same message without an error; not observed directly).

Not tested: how long an MV keeps retrying (the MV was dropped before I could tell), all and raw (they read the ID endpoint, not the subject), Avro, a registry that is slow instead of down (a separate test: requests over 5 s fail with Timeout), other releases.

Not covered

Docs for releases other than the ones named above were checked in source only. The 2.8.x / 3.2.x / 3.3.x rows of items 2 to 4 were not run live. Item 3 for all/raw was observed on 3.5.1-rc.0 and 3.4.1 only. Item 5 was run live on 3.5.1-rc.0 and 3.4.1 only; the source was read at v3.5.1-rc.0, v3.4.2-rc.2 and v3.4.1.

Proposed wording for items 1 to 4: see the linked PR #705. Item 5 is not in that PR; I did not add to it while it is in review.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions