Skip to content

BUG: Duplicate key causes undefined behaviour, user not warned #31

Open

Description

@corneliusroemer

When trying to figure out what happens in #30 I discovered that when there are rows with duplicated keys, the behaviour is odd.

It would be good to test for duplicate keys and raise an error or warning (maybe allowing user to pick how to deal with this).

What shouldn't happen is that incomplete rows cause some sort of undefined behaviour as here:

printf "a,b,c,d\n1,2,3,4\n1,2,3\n3,2,3,4" >a.csv
printf "a,b,c,d\n1,2,3,4\n1,2,3,4\n3,2,3,4" >b.csv
csv-diff b.csv a.csv --key a

The output makes no sense:

1 column removed

1 column removed

  d

Activity

  1. jhrch7 commented on Nov 2, 2026

    @jhrch7

    The reproducer combines two things that need separate checks before constructing the keyed dictionary:

    • a.csv data record 2 has three fields for a four-column header. The current dict(zip(headings, line)) leaves d absent.
    • Both inputs repeat key 1; constructing {keyfn(r): r for r in rows} keeps the last row for that key. A missing field can therefore affect the later inferred column set.

    For an explicit preflight/review workaround, I tested these exact inputs with Local Row Check v1.1.0, a free MIT tool I publish. From that checkout:

    python3 rowcheck.py a.csv b.csv --key a --compare-columns b c d --out original-report

    Actual result, exit 2, with no report directory created:

    Error: a.csv: data record 2 has 3 fields; expected 4
    

    To test duplicates separately, I made a clearly separate, valid-width variant a-valid.csv where the missing d is an explicit empty field:

    a,b,c,d
    1,2,3,4
    1,2,3,
    3,2,3,4

    With b.csv unchanged:

    python3 rowcheck.py a-valid.csv b.csv --key a --compare-columns b c d --out valid-duplicate-report

    The executed result has all four key-1 source rows in review.csv (two per input), while key 3 is the one unambiguous shared pair. No key-1 row is selected or discarded, and those ambiguous rows are excluded from value comparisons. The command reports zero changed cells only for the one unambiguous pair.

    This is an external validation/review workaround, not a patch to csv-diff. The original malformed input is rejected; adding the trailing comma above is an explicit demonstration variant, not an automatic data repair. A native reader fix could likewise validate width and key uniqueness before converting rows to a dictionary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions