Apache Iggy Connectors move data between Apache Iggy streams and
external systems. The runtime iggy-connectors loads sink and source
plugins (cdylib) with dlopen and drives them through a small FFI.
Plugins build on the SDK iggy_connector_sdk. The Iggy server, the
client SDKs and the CLI live in apache/iggy.
This repository builds against one commit of apache/iggy, called the core pin.
Skills live under
.claude/skills/, with per-skill links under.agents/skills/for Codex. Load connectors-overview first for any change in this repository.
- Writing code for an issue not assigned to the user
(
gh issue view <N> --json assignees). If it is not, or if you cannot check it, they must ask on the issue and wait until a maintainer assigns it. - Bumping
iggy_connector_sdkMAJOR version or changing any FFI signature insdk/src/{sink,source}.rs(breaks every pre-built plugin .so). - Touching the state-save path (
runtime/src/state.rs) in a way that changes the atomic-rename protocol (corruption risk). - Renaming default consumer groups, plugin path resolution rules,
environment variables, the image
apache/iggy-connect, the tag patternsconnectors-X.Y.Zandconnectors-sdk-X.Y.Z, or any other operator-facing identifier. - Moving the core pin to an apache/iggy commit that is not a release (see The core pin).
- Force-pushing to
master, amending shipped commits, or skipping hooks with--no-verify/--no-gpg-sign.
Most PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Agent-Driven PRs, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.
- The PR must link an issue the user was assigned to before any work started. See the STOP list above.
- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. When an agent wrote any of the change, never leave the section out or write "None".
- Run
prek runbefore the PR is opened. If prek is not installed, ask the user to install it (cargo install prek, thenprek install). Do not run the individual checks in its place. - If a check cannot run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.
- The user drives the change. They pick the issue, decide the approach, write the rationale in their own words and explain every change without asking you. Maintainers can close agent-driven PRs, where the agent did the work and the user only approved it.
- New contributors keep one open PR at a time.
- If you found a vulnerability, follow SECURITY.md and open no public issue or PR. For any other bug, the user opens an issue and waits until a maintainer agrees on the approach and assigns it. Then you can write one PR with the test and the fix. If the fix is hard, the PR can hold only the test, marked
#[ignore].
# CI enforces this order (.github/actions/rust/pre-merge).
cargo fmt --all # step 1
cargo sort --no-format --grouped --workspace # step 2 (both flags mandatory)
cargo clippy --all-features --all-targets -- -D warnings # step 3
cargo machete --with-metadata # step 4
cargo nextest run -p <package> # step 5
# Integration tests need Docker, the server at the core pin and fresh builds in target/debug/.
./scripts/build-test-server.sh # once per core pin
cargo build # runtime and every plugin
cargo nextest run -p iggy_connector_postgres_sink --test postgres
# Other CI checks (mirror in pre-commit; pass --fix where supported)
./scripts/ci/taplo.sh # TOML
./scripts/ci/markdownlint.sh # Markdown
./scripts/ci/skills.sh # skill metadata, links, invocation policies
./scripts/ci/shellcheck.sh # Shell
./scripts/ci/license-headers.sh # Apache headers
./scripts/ci/trailing-whitespace.sh
./scripts/ci/trailing-newline.sh
./scripts/ci/sync-rustc-version.sh
./scripts/ci/binary-artifacts.sh --check
./scripts/ci/third-party-licenses.sh --validate --manifest <path>
./scripts/ci/check-core-pin.sh # each core `version` matches the crate at the core pin
./scripts/verify-sdk-publish.sh # SDK into a scratch local registry
# Move the core pin, the apache/iggy commit
./scripts/bump-core-pin.sh <core tag or commit>
# Run the runtime with the example configuration
IGGY_CONNECTORS_CONFIG_PATH=runtime/example_config/config.toml cargo run --bin iggy-connectorsPre-commit hooks wire the same scripts. Install them with pre-commit install (Python) or prek install (Rust drop-in). Both read
.pre-commit-config.yaml.
iggy-connectors/
├── runtime/ iggy-connectors: the runtime binary, image apache/iggy-connect
│ ├── config.toml default configuration, compiled in as the fallback
│ ├── example_config/ example runtime and connector configurations
│ └── tests/ runtime/, api/, http_config_provider/, config_env_listing.rs
├── sdk/ iggy_connector_sdk: traits, FFI macros, decoders, encoders, transforms
├── sinks/<name>/ one sink plugin, iggy_connector_<name>, with its tests/
├── sources/<name>/ one source plugin, iggy_connector_<name>, with its tests/
├── testing/ iggy_connector_testing: fixtures that several test targets share
├── scripts/ CI checks, test server build, core pin bump
├── .config/ nextest.toml, rail.toml
├── .github/ workflows, actions, components.yml, publish.yml
└── .claude/skills/ connector and review skills, plus the .agents/skills links
The root Cargo.toml is the only place for the core pin. The first
group of [workspace.dependencies] holds six entries that take core
crates from git, with the core pin as rev: configs, configs_derive, iggy,
iggy_common, integration (the test harness and
#[iggy_harness]) and system_stats. iggy and iggy_common also
carry a version, because the published SDK takes them from
crates.io. Each version must equal the crate version at the core
pin, and scripts/ci/check-core-pin.sh fails otherwise.
- The core pin stays at the last core release. When a connector needs
an unreleased core change, the core pin moves to a core
mastercommit. That change lands in apache/iggy first. scripts/bump-core-pin.shrewrites the core pin and the two versions. Never edit them by hand.- No
[patch]section, no submodule and no sibling checkout of apache/iggy. A local edit under~/.cargo/git/checkouts/is lost on the next fetch. - The nightly workflow tests the tip of core
master. If that run fails, it opens an issue.
| Task | Location |
|---|---|
| Runtime | runtime/src/ |
| Plugin SDK | sdk/src/ |
| Sinks and sources | sinks/<name>/, sources/<name>/ |
| Runtime configuration | runtime/config.toml, runtime/example_config/ |
| Integration tests | <package dir>/tests/<suite>/, runtime/tests/ |
| Shared test fixtures | testing/ |
harness module, server |
core/integration/, core/server/ in apache/iggy |
| CI test scope | .config/rail.toml, .github/config/components.yml |
| Releases and publishing | .github/config/publish.yml |
Each SKILL.md has YAML frontmatter (name, description). Load
connectors-overview first as router.
Claude Code uses /skill-name. Codex uses $skill-name or /skills.
Keep one source under .claude/skills/ and link its directory from
.agents/skills/. Keep templates and references inside the source directory.
- connectors-overview: router and universal connector rules
- connector-runtime: FFI host, lifecycle, state, metrics
- connector-sdk: traits, decoders and encoders, retry
- connector-sink: sink plugin authoring
- connector-source: source plugin authoring
- connector-transform: transform authoring
- connector-testing: unit and integration test patterns
Review skills are user-invoked only. Keep disable-model-invocation: true
for Claude Code and policy.allow_implicit_invocation: false in
agents/openai.yaml for Codex. Do not replicate their steps without an
explicit invocation. When a non-trivial change passes its checks,
suggest /team-review <target> in Claude Code or $team-review <target>
in Codex.
- connector-review: adversarial connectors PR or branch review with validation
- team-review: four expert review of a PR, branch, or ref range
Review skills share execution instructions
for target preparation and client-specific agent tools. team-review
is a copy of the apache/iggy skill. Run
./scripts/ci/skills.sh to check skill metadata, links, and invocation policies.
- Apache 2.0 header on every new source file: Follow the comment style that
licenserc.tomlconfigures. For example,.rsuses// ..., andCargo.tomland shell use# .... - Order of the checks:
fmt --all->sort --no-format --grouped --workspace->clippy --all-features --all-targets -- -D warnings->machete->nextest run. Plustaplo.sh --check. CI enforces all. - Always pass
--no-formatand--groupedtocargo sort. Without--no-formatmulti-line feature arrays flatten andtaplore-expands them, churning the workspace. Without--groupedthe core pin group of the rootCargo.tomlis sorted into the other dependencies. - One virtual workspace. Every dependency of every member is declared once in the root
[workspace.dependencies], and members use{ workspace = true }. - Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries. Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.
- Idiomatic Rust over free helpers.
FromStrnotparse_foo,Displaynotformat_foo,From/TryFromnotto_foo_from_bar. StructuredErrenums, notString. - Imports at the top. No inline
useinside functions or blocks. Group: std, external crates, thencrate::. - Precise names. No
b,p,t,m. Match literal API field names as log labels. - Comments explain WHY, not WHAT. Default to no comment. Add a comment only for a hidden constraint or a workaround that can surprise a future reader. Never reference the current task / PR.
- Never
cargo installwithout authorization. Toolchain pinned inrust-toolchain.toml. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check.scripts/build-test-server.shis the exception: it installsiggy-serverundertarget/, not into~/.cargo/bin. - No
unwrap()/expect()on Results from external I/O outside tests. tokio::sync::Mutex(notstd::sync::Mutex) wherever a lock is held across.await. Hold locks briefly: lock, clone, drop guard, then do I/O.- Code reads top to bottom. Public consts and public fns first, helpers below, deeper helpers below those.
- Zero clone in hot loops.
&self,&str,&[T]. If the size is known, useVec::with_capacity(n). - Forward-compatible configuration. New TOML fields use
#[serde(default)]orOption<T>.
| Tool | Where |
|---|---|
cargo fmt, cargo sort, cargo clippy |
pre-commit + CI |
cargo machete |
CI machete task |
cargo nextest + cargo-rail |
CI test-* DAG-scoped |
cargo llvm-cov |
CI coverage |
cargo http-registry |
scripts/verify-sdk-publish.sh |
taplo, markdownlint-cli, shellcheck |
pre-commit + CI |
typos |
pre-commit |
cargo about |
scripts/ci/third-party-licenses.sh |
HawkEye / license-headers.sh |
pre-commit + CI |
- Unit:
#[cfg(test)] mod testsat EOF ofsrc/lib.rs(or per-module). They need no Docker:cargo nextest run -p <package> --lib(--binsfor the runtime). - Integration: each plugin owns its suite in
<package dir>/tests/<suite>/, for examplesinks/postgres_sink/tests/postgres/. A backend with a sink and a source has a target in both packages. The runtime ownsruntime/tests/runtime/,runtime/tests/api/,runtime/tests/http_config_provider/andruntime/tests/config_env_listing.rs.bigquery_sinkhas no#[iggy_harness]tests. Itstests/bigquery_sink.rsruns the sink against an in-process fake BigQuery and needs no Docker. #[iggy_harness]: the tests use this macro and theharnesstypes of the core crateintegrationat the core pin. A test startstarget/debug/iggy-serverandtarget/debug/iggy-connectors. The runtime loads each plugin by its bare file name, such aslibiggy_connector_stdout_sink, from its own directory.- Dev-dependencies: the macro emits
::tokio::and::serial_test::paths, so each package with integration tests declares both. A test binary installs the per-test log capture withintegration::install_log_capture!(), which needsctoranddtortoo. - Backends: external systems run in Docker through
testcontainers-modules. Container names start withiggy-test-. Fixtures that several test targets share live intesting/. - Logs: per-test logs land in
test_logs/, because.cargo/config.tomlsetsIGGY_TEST_LOGS_DIR. - CI scope:
.config/rail.tomlhas onerequirerow per test target. The row lists the runtime binary and every plugin that the target loads. A new test target needs its row.
BDD test naming: given_X_when_Y_should_Z (3-part). Promote to 4-part given_X_when_Y_then_Z_should_W only with a distinct "then" intermediate. Be consistent inside a file.
Filter a single integration test by its path inside the test target. Build the binaries that #[iggy_harness] starts first. It runs
whatever is already in target/debug/, so a stale one fails as though the change under test broke it.
cargo build -p iggy-connectors -p iggy_connector_stdout_sink -p iggy_connector_random_source
cargo nextest run -p iggy-connectors --test runtime benchmark::given_logging_format_jsonA core test named connectors::<suite>::<rest> became the test <rest> in the target <suite> of its package.
- Docker is required for the integration tests.
testcontainers-modulesspawns containers per test. - The Doris tests need
vm.max_map_countof at least 2000000 on Linux. Runsudo sysctl -w vm.max_map_count=2000000. - The tests run against the server at the core pin.
scripts/build-test-server.shbuilds it intotarget/iggy-server-<core-pin>/and copies it totarget/debug/iggy-server. Run it again after each core pin bump. A server from another commit fails in ways that look like the change under test. #[iggy_harness]runs prebuilt binaries and never builds them. A staletarget/debug/iggy-connectorsor plugin library is used silently. Rebuild the runtime and the plugins that a target loads before you run it.cargo sortwithout--no-formatwill churn the workspace. Use the pre-commit hook.- Rust toolchain is pinned in
rust-toolchain.toml. Older systemcargofails on transitive deps. prefer rustup so the pin auto-resolves. - The first build fetches apache/iggy from git. Later builds use
~/.cargo/git. test_logs/grows quickly. Wipe between major refactors.
Created by the runtime or tests. Gitignored. Safe to delete to reset:
local_state/- runtime state files of the example configurationtest_logs/- per-test logstarget/- cargo build, plus the test server intarget/iggy-server-<core-pin>/
- Conventional commits with scope:
feat(sink): .... Subject <= 72 chars. Allowed types/scopes enforced by.github/workflows/pr-title.yml. - Body: why before what. No bullet lists of files changed.
- If a change has co-authors, add a
Co-authored-bytrailer for each. - The runtime ships only as the image
apache/iggy-connect, taggedconnectors-X.Y.Z. The SDK goes to crates.io, taggedconnectors-sdk-X.Y.Z. Sinks and sources arepublish = false. - Post-merge publishes an
edgeSDK version that has no tag yet, and then tags it. A version without a suffix is a release and goes out only after the vote. - A release that ships with a core release has one vote for both. A connectors-only release has its own vote, with the core pin at the last core release.
- Third-party license enumeration:
scripts/ci/third-party-licenses.sh(ASF policy).