Skip to content

Race between Stop() and FetchMe crashes host process (nil deref) or wedges the FFI on never-paired devices #201

Description

@devjotaduo

Summary

There is a race between Stop() and the Neonize() connect flow in goneonize/main.go (tag 0.4.3.post0, still present on master) that either crashes the whole host process with a nil-pointer dereference in FetchMe, or permanently wedges the FFI call — depending on which side wins the race. Any embedder that calls Stop() while a never-paired device is in the QR flow hits one of the two outcomes.

Environment

  • neonize 0.4.3.post0 (also reproduced reading master sources)
  • Linux amd64, Python 3.11, client created via neonize.aioze.client.NewAClient
  • Device with no stored pair (QR flow), host app calls Stop()/reconnect when the QR expires unpaired

The race

Stop() does, in this order (main.go:572):

  1. client.Disconnect() — this closes the QR channel that Neonize() is ranging over;
  2. delete(clients, id);
  3. cancels the context, drops the event channel.

Closing the QR channel wakes the Neonize() goroutine out of for evt := range qrChan (main.go:1123), which then proceeds to CallbackFunction()FetchMe(id):

func FetchMe(id string) *defproto.Device {
	cli := clients[id].Store   // main.go:2481

	// Block until cli.ID is set
	for cli.ID == nil {
		time.Sleep(100 * time.Millisecond)
	}
	...
}
  • If the delete(clients, id) in Stop() wins the race, clients[id] is nil and clients[id].Store segfaults — the whole embedding process dies (Python interpreter included, taking down everything else the host was running).
  • If FetchMe wins, it captures the store of a device that will never pair, and the for cli.ID == nil busy-wait spins forever on a thread that is locked to the FFI call — the connect() call never returns and the client can never be recycled.

Both failure modes are two faces of the same race. We observed both in production: the SIGSEGV killed our multi-agent host every ~3 minutes (the QR channel times out after ~3 min unscanned, our watchdog called Stop(), boom), and the busy-wait was the "FFI stopped responding" wedge we had been chasing for a day.

Crash log (production)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation addr=0x0]

goroutine 34 [running, locked to thread]:
main.FetchMe({0x18e37e0a22a0?, 0x56?})
	goneonize/main.go:2481 +0x41
main.CallbackFunction({0x7fdfbdf11aa8, 0x18e37e0a4a50}, 0x7fe026674090, {0x18e37e0a22a0, 0x56})
	goneonize/main.go:2507 +0xd7
main.Neonize(...)
	goneonize/main.go:1154 +0xb4f

Fix we are running

We patched FetchMe to (a) return nil when the client is already gone from the map, and (b) wait for cli.ID in a select that is interruptible by the per-client context that Stop() already cancels (plus re-checking the map each tick); CallbackFunction treats a nil return by exiting cleanly. Running in production since 2026-07-18 with zero panics.

Patch: devjotaduo@c999239 (branch qwenpaw-0.4.3.1, based on tag 0.4.3.post0; comments are in Portuguese — happy to open a PR with cleaned-up comments if you want it).

Related: while debugging this we also hit a separate use-after-free of the ctypes callbacks on the Python side (filed separately).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions