From 44ebabb895935e6228b23b2b16b257e1c557531e Mon Sep 17 00:00:00 2001 From: keemgdeok Date: Wed, 16 Sep 2026 22:09:24 +0900 Subject: [PATCH 1/2] docs: improve README readability --- README.md | 87 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 53 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 1123dc9..84c2ac0 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,15 @@ # metadq -**Data-quality checks from Apache Iceberg metadata, without opening content -data files.** +[![Release](https://img.shields.io/github/v/release/keemgdeok/metadq?include_prereleases&sort=semver)](https://github.com/keemgdeok/metadq/releases) +[![CI](https://github.com/keemgdeok/metadq/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/keemgdeok/metadq/actions/workflows/ci.yml) +[![Java 17+](https://img.shields.io/badge/Java-17%2B-007396?logo=openjdk&logoColor=white)](https://adoptium.net/temurin/releases/) -`metadq` is a read-only Java CLI with two commands: - -- `doctor` explains which metadata evidence is available. -- `check` evaluates four conservative rules for CI and orchestration. +**Fast, conservative data-quality checks for Apache Iceberg—without Spark or +data-file scans.** -No Spark, data-row scan, or service is required. +| Metadata-only | Standalone | Automation-ready | +| --- | --- | --- | +| Reads metadata and manifests | One Java CLI; no Spark or service | Text/JSON and documented exit codes | > **Status:** 0.1.0 release candidate. The demo, automated tests, and local > MinIO/REST workflow are complete; a managed REST catalog with S3 still needs @@ -26,8 +27,6 @@ java -jar metadq.jar doctor --demo The demo creates an Iceberg v2 table in memory with metadata pointing to nonexistent sentinel data files. -A report excerpt: - ```text TABLE demo.events FORMAT v2 @@ -37,7 +36,37 @@ NOTE column.null_ratio(user_id) is not ready: null counts cover 2/3 files GUARD No content data files opened. ``` -## Use a REST catalog +## How it works + +```mermaid +flowchart LR + A["Iceberg catalog"] --> B["Metadata & manifests"] + B --> C["metadq"] + C --> D["PASS · FAIL · UNKNOWN"] +``` + +`metadq` is a read-only Java CLI with two commands: + +- `doctor` explains which metadata evidence is available. +- `check` evaluates four conservative rules for CI and orchestration. + +## Rules and results + +| Rule | Checks | +| --- | --- | +| `table.row_count` | Sum of live data-file record counts | +| `column.null_ratio` | Null ratio when top-level column metrics are complete | +| `table.last_commit_age` | Snapshot commit recency, not event-time freshness | +| `schema.column` | Top-level column presence, type, and nullability | + +Results are `PASS`, `FAIL`, or `UNKNOWN`. `UNKNOWN` means the metadata cannot +prove either outcome—for example, because metrics are incomplete or applicable +delete files exist. Operational and configuration errors return `ERROR`. + +Overall exit codes are `0` when all rules pass, `1` for any failure, `2` for an +error, and `3` for an unknown result when there is no failure or error. + +## Connect a REST catalog Configure the included examples, then run `doctor` or `check` against one table: @@ -46,43 +75,33 @@ table: cp examples/catalog.properties.example catalog.properties cp examples/metadq.yml metadq.yml -java -jar build/libs/metadq.jar doctor \ +java -jar metadq.jar doctor \ --catalog-properties catalog.properties \ --table analytics.events -java -jar build/libs/metadq.jar check \ +java -jar metadq.jar check \ --catalog-properties catalog.properties \ --table analytics.events \ --rules metadq.yml ``` -Both commands support `--format text` and `--format json`. - -For a reproducible MinIO and Iceberg REST Catalog environment, see the +Both commands support `--format text` and `--format json`. For a reproducible +MinIO and Iceberg REST Catalog environment, see the [local end-to-end guide](docs/E2E.md). -The [metadata benchmark](docs/BENCHMARK.md) covers 1k, 10k, and 100k live-file -metadata with no, one, or all column statistics requested. - -On the documented Apple M4 reference run, 100k live files took 75 ms for row -count evidence, 81 ms with one column, and 127 ms with all ten columns. These -are in-memory metadata timings, not a comparison with a data-row scan. - -## Rules and results +## Performance -| Rule | Checks | -| --- | --- | -| `table.row_count` | Sum of live data-file record counts | -| `column.null_ratio` | Null ratio when top-level column metrics are complete | -| `table.last_commit_age` | Snapshot commit recency, not event-time freshness | -| `schema.column` | Top-level column presence, type, and nullability | +The most useful rows from the documented Apple M4 reference run are: -Results are `PASS`, `FAIL`, or `UNKNOWN`. `UNKNOWN` means the metadata cannot -prove either outcome—for example, because metrics are incomplete or applicable -delete files exist. Operational and configuration errors return `ERROR`. +| 100,000 live files | Median runtime | Peak heap | +| --- | ---: | ---: | +| Row count | 75 ms | 71 MiB | +| One column | 81 ms | 128 MiB | +| All ten columns | 127 ms | 247 MiB | -Overall exit codes are `0` when all rules pass, `1` for any failure, `2` for an -error, and `3` for an unknown result when there is no failure or error. +These are in-memory metadata timings, not a comparison with a data-row scan. +See the [benchmark methodology and full results](docs/BENCHMARK.md) for the 1k, +10k, and 100k file runs. ## Scope From f390dc6030dafe1320d4425723dcec70edc6e56a Mon Sep 17 00:00:00 2001 From: keemgdeok Date: Wed, 16 Sep 2026 22:14:41 +0900 Subject: [PATCH 2/2] docs: simplify README terminology --- README.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 84c2ac0..f619d31 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,7 @@ data-file scans.** | --- | --- | --- | | Reads metadata and manifests | One Java CLI; no Spark or service | Text/JSON and documented exit codes | -> **Status:** 0.1.0 release candidate. The demo, automated tests, and local -> MinIO/REST workflow are complete; a managed REST catalog with S3 still needs -> validation. +> **Status:** v0.1.0-rc.1 pre-release. ## Quick start @@ -24,8 +22,8 @@ curl -LO https://github.com/keemgdeok/metadq/releases/download/v0.1.0-rc.1/metad java -jar metadq.jar doctor --demo ``` -The demo creates an Iceberg v2 table in memory with metadata pointing to -nonexistent sentinel data files. +The demo runs against an in-memory Iceberg v2 table and never opens content +data files. ```text TABLE demo.events @@ -91,13 +89,13 @@ MinIO and Iceberg REST Catalog environment, see the ## Performance -The most useful rows from the documented Apple M4 reference run are: +The 100,000-live-file results from the documented Apple M4 reference run are: -| 100,000 live files | Median runtime | Peak heap | +| Requested column statistics | Median runtime | Peak heap | | --- | ---: | ---: | -| Row count | 75 ms | 71 MiB | -| One column | 81 ms | 128 MiB | -| All ten columns | 127 ms | 247 MiB | +| None | 75 ms | 71 MiB | +| 1 column | 81 ms | 128 MiB | +| 10 columns | 127 ms | 247 MiB | These are in-memory metadata timings, not a comparison with a data-row scan. See the [benchmark methodology and full results](docs/BENCHMARK.md) for the 1k,