Skip to content

Add opt-in hash DB batching and atomic JSON writes#514

Open
TerryFrench wants to merge 3 commits into
jmathai:masterfrom
TerryFrench:pr/07-hash-batching
Open

Add opt-in hash DB batching and atomic JSON writes#514
TerryFrench wants to merge 3 commits into
jmathai:masterfrom
TerryFrench:pr/07-hash-batching

Conversation

@TerryFrench

@TerryFrench TerryFrench commented Feb 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in hash DB flush batching for import and makes local JSON DB writes atomic.

Changes

  • Adds --hash-db-batch-size to import (default 1, preserving current behavior).
  • Enables periodic hash DB flush when batching is enabled.
  • Makes hash.json / location.json writes atomic (tempfile + os.replace).
  • Adds tests for:
    • batch flush behavior
    • atomic replace write behavior
    • process-file no-flush path when batching

Why

Improves resilience and performance tradeoff options for large imports while keeping default behavior unchanged.

Dependency

This PR is based on #513 (pr/06-interrupt-exiftool) because it builds on nearby CLI/filesystem changes.

Tests

Validated locally with pytest after applying this change.

Targeted coverage added for new behavior

  • elodie/tests/elodie_test.py::test_import_hash_db_batch_size_default_writes_per_file
    • verifies default behavior remains unchanged (--hash-db-batch-size=1, per-file write path).
  • elodie/tests/elodie_test.py::test_import_hash_db_batch_size_flushes_in_batches
    • verifies opt-in batching flushes hash DB at the expected interval.
  • elodie/tests/filesystem_test.py::test_process_file_skips_hash_db_flush_when_write_db_false
    • verifies process_file skips immediate hash DB flush when batching mode is active.

Suite run

  • Ran project test suite
  • Result: all selected tests passed (no regressions in modified areas).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant