Bind the pipeline the batch source stands on

[martin-s-a]

Closes #154. Item 3b of the backlog, and the half with the threads in it.

image_batch_source is the class meant for pipelines: asynchronous, batched,
running its own threads. The slow path stays what it is, for one-off reads.

source = rexlib.em.image.batch_source(workers=8)

destination = rexlib.zeros(
    rexlib.make_contiguous_array_descriptor(
        (len(locations), *rexlib.em.image.query_core_extents(path)),
        rexlib.NumericalType.float32,
    ),
    rexlib.hardware.MemoryResourceAffinity.host,
    context,
)

completion = source.read(destination, locations)
completion.get()

Reaching that meant binding the chain beneath it — Executor and its two
kinds, ImageReaderProvider and its two, ImageSource — and Completion,
which is what read hands back. Executor and Completion live in a new
_binding.concurrency, following src/core/concurrency/.

The GIL, which is the whole point

Completion.wait, Completion.get and ImageBatchSource.read take
py::call_guard<py::gil_scoped_release>().

This is not a refinement. Held through a read, the GIL freezes every other
Python thread for its duration, and there is nothing asynchronous left about
the interface — ordering a batch and blocking the interpreter is what the slow
path already does. pybind11 converts the arguments before the guard is
constructed and the return value after it is destroyed, so nothing touches
Python without it.

read gets the same treatment because it opens files through the provider
before returning, even though it returns before the reads are done.

The destination is shared, not moved

read takes its destination by value and rexlib::array is move-only. Bound
plainly, pybind11 would move the array out of the caller's Python object and
hand them back an empty one — silently, and only noticed later. The binding
takes array& and passes destination.share(), which is a second handle onto
the same storage. A test pins it: the destination still reports its shape
after the read.

Assembly

batch_source(workers=…, cache=…, manager=…, executor=…) wires the formats, a
provider, an executor and the source, so a caller reaches the pipeline without
naming four objects. Every argument has a default and every one can be
overridden; the parts are all exported for anyone who wants to assemble them
by hand, and a test does exactly that.

Each source built this way owns its executor, which is the settled decision:
sharing a pool between sources is the caller's business, done by passing one
executor to both, and per-instance is the direction that stays cheap to
reverse. cache=N swaps the direct provider for a caching one.

The query_extents and query_core_extents overloads over a provider arrive
here too, having waited for the provider to exist.

Checked

• All 47 translation units compile clean under
  g++ -std=c++20 -fsyntax-only -Wall -Wextra -Wpedantic against the pinned
  submodule.
• ruff check . passes; 56 tests collect under tests/em/image/.
• The tests cover a batch read end to end, two batches in flight collected
  together, an empty batch resolving without touching the source, a wrong
  batch size refused, the synchronous executor, a caching provider serving a
  repeated read, and assembly by hand.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud