Skip to content

some bug fixes from fuzzer finds - #289

Open
stephen-dwq wants to merge 1 commit into
mainfrom
fix/fuzzer-found-bugs
Open

stephen-dwq wants to merge 1 commit into
mainfrom
fix/fuzzer-found-bugs

Conversation

@stephen-dwq

Copy link
Copy Markdown
Collaborator

Bugs found using test/fuzzer.

  1. Buffered loop-back edges could requeue nodes.
    ingest_input puts an edge in one slot. It then calls register_ingested_input, which reads both slots. An edge that lands in ready_next_iter still makes the registry read ready_signals, which this ingest does not change. For a node that already ran, that slot is still full, so the node goes back on ready_names. The scheduler can then run the node a second time in one iteration, on a partial input set.

  2. Registry counts completion instead of entities
    Triggered by bug 1, which can complete a node twice.
    The guard holds only after the registry is done. A duplicate completion before that point still raises the count. The registry then reports the iteration complete while one entity never ran. The loop resets early, the inputs of that entity are dropped, and the request will hang.

  3. Finished loops can loop indefinitely with empty outputs
    Previously hidden by bug 2, which would eventually end the loop because each loop execution would raise completion count.
    A loop on its last iteration sets is_done = True and calls inner_registry.clear(). That call returns the registry to its start, so the registry of the body says the body did not start. A later edge is then ingested as usual, the node goes on ready_names, and the body starts again. complete_iter runs a second time and sends the declared outputs again. The first completion cleared _cached_outputs, so those outputs carry no tensor. The body runs without end.

  4. Backlog drain ignores deferred removal
    _filter_cap_and_schedule does not ignore pending_removes and held RIDs.

5, 6. Emptied chunk can be considered work or indefinitely parked
If all RIDs are dropped (e.g. all requests failed), it is still considered work, and scheduled. We could work on things with nothing in them.

  1. Abandoned pre-plans leak streams in KVManager
    Labels made for segments in pre-plans are not added to the list that clear_preplan consults; so their streams remain unclaimed until request end.

  2. Zero-span segments don't get streams in KVManager
    Fresh zero-span segments (intended as readop) make it past admit but cannot actually be plan because no associated stream.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant