tProxy HTTP APIs crashing

[plebhash]

not long after launching tProxy, I can no longer curl its HTTP APIs

---

edit:

for context, this happened while testing Braiins new service https://hashpower.braiins.com/

we deployed a few hundred machines to the VPS-hosted tProxy

I'm trying to reproduce this on a local tProxy with 2 CPU miners and it's not happening

[plebhash]

@lucasbalieiro can you try to reproduce?

[plebhash]

sorry I should to correct what I said here:

by paying a few sats, we can deploy a few PH/s via multiple machines (~100 units)

it's not so cheap to deploy hundreds of machines... with 10k sats I just placed an order for ~5h of 1 PH/s, which turned out to be only 3 machines

yesterday we had hundreds of machines because we had a much bigger budget, which I wasn't aware of when I wrote this issue

---

anyways, just taking note because this might not be so "cheap" to reproduce as I originally made it seem to be

[plebhash]

with 10k sats I just placed an order for ~5h of 1 PH/s, which turned out to be only 3 machines

it seems that after ~1h of testing, this was enough to reproduce this issue

it's important to note that there were ~200 client disconnection/reconnections... not sure why, but the braiins machines keep dropping and reconnecting, which might play a key role in this

[plebhash]

found this on logs:

...
2026-01-25T02:56:16.952817Z ERROR translator_sv2::sv1::downstream::downstream: Error receiving downstream message: RecvError
2026-01-25T02:56:16.952907Z ERROR translator_sv2::sv1::downstream::downstream: Downstream 2: error in downstream message handler: TproxyError { kind: ChannelErrorReceiver(RecvError), action: Disconnect(2), _owner: PhantomData<translator_sv2::error::Downstream> }
2026-01-25T02:56:16.952981Z DEBUG translator_sv2::status: Sending status from Downstream [2]: DownstreamShutdown { downstream_id: 2, reason: ChannelErrorReceiver(RecvError) }
2026-01-25T02:56:16.953033Z  WARN translator_sv2::sv1::downstream::downstream: Downstream 2: unified task shutting down
2026-01-25T02:56:16.953061Z DEBUG translator_sv2::sv1::downstream::channel: Dropping downstream channel state
2026-01-25T02:56:16.953058Z  WARN translator_sv2: Downstream 2 disconnected — notifying SV1 server.
2026-01-25T02:56:16.953182Z  INFO translator_sv2::sv1::sv1_server::sv1_server: 🔌 Downstream: 2 disconnected and removed from sv1 server downstreams
2026-01-25T02:56:16.953219Z ERROR translator_sv2::sv1::downstream::downstream: Error receiving downstream message: RecvError
2026-01-25T02:56:16.953243Z DEBUG translator_sv2::sv2::channel_manager::channel_manager: Received UpdateChannel from SV1Server: UpdateChannel { channel_id: 0, nominal_hash_rate: 1000000000000000.0, maximum_target: Owned([220, 157, 232, 163, 155, 244, 147, 123, 230, 200, 157, 168, 151, 114, 172, 239, 63, 46, 230, 134, 189, 136, 167, 172, 52, 7, 0, 0, 0, 0, 0, 0]) }
2026-01-25T02:56:16.953262Z ERROR translator_sv2::sv1::downstream::downstream: Downstream 3: error in downstream message handler: TproxyError { kind: ChannelErrorReceiver(RecvError), action: Disconnect(3), _owner: PhantomData<translator_sv2::error::Downstream> }
2026-01-25T02:56:16.953278Z  INFO translator_sv2::sv2::channel_manager::channel_manager: Sending UpdateChannel message to upstream for channel_id: 2
2026-01-25T02:56:16.953293Z DEBUG translator_sv2::status: Sending status from Downstream [3]: DownstreamShutdown { downstream_id: 3, reason: ChannelErrorReceiver(RecvError) }
2026-01-25T02:56:16.953328Z DEBUG translator_sv2::sv2::upstream::upstream: Upstream: sending sv2 frame from channel manager: Sv2Frame { header: Header { extension_type: 32768, msg_type: 22, msg_length: U24(40) }, payload: Some(Mining(UpdateChannel(UpdateChannel { channel_id: 2, nominal_hash_rate: 1000000000000000.0, maximum_target: Owned([220, 157, 232, 163, 155, 244, 147, 123, 230, 200, 157, 168, 151, 114, 172, 239, 63, 46, 230, 134, 189, 136, 167, 172, 52, 7, 0, 0, 0, 0, 0, 0]) }))), serialized: None }
2026-01-25T02:56:16.953333Z  WARN translator_sv2::sv1::downstream::downstream: Downstream 3: unified task shutting down
2026-01-25T02:56:16.953360Z DEBUG translator_sv2::sv1::downstream::channel: Dropping downstream channel state
2026-01-25T02:56:16.953467Z  WARN translator_sv2: Downstream 3 disconnected — notifying SV1 server.
2026-01-25T02:56:16.953524Z  INFO stratum_apps::monitoring::http_server: Monitoring server received shutdown signal, stopping...
2026-01-25T02:56:16.953720Z  INFO stratum_apps::monitoring::http_server: Monitoring server stopped

let's analyze TranslatorSv2::start:

here we create a notify_shutdown as a tokio::sync::broadcast channel with capacity 1:

sv2-apps/miner-apps/translator/src/lib/mod.rs

Line 65 in 3d99603

let (notify_shutdown, _) = broadcast::channel::<ShutdownMessage>(1);

which we then monitor here, where line 173 is key:

sv2-apps/miner-apps/translator/src/lib/mod.rs

Lines 167 to 182 in 3d99603

	let mut notify_shutdown_monitoring = notify_shutdown.subscribe();
	let shutdown_signal = async move {
	loop {
	match notify_shutdown_monitoring.recv().await {
	Ok(ShutdownMessage::ShutdownAll) => break,
	Ok(_) => continue, // Ignore other shutdown messages
	Err(_) => break,
	}
	}
	};
	task_manager.spawn(async move {
	if let Err(e) = monitoring_server.run(shutdown_signal).await {
	error!("Monitoring server error: {:?}", e);
	}
	});

when downstream miners disconnect rapidly (as seen in the logs - Downstreams 2 and 3 disconnect almost simultaneously), multiple DownstreamShutdown messages are sent on the channel... the monitoring server's receiver lags behind because:

• the channel capacity is 1
• two messages are sent quickly
• we break the monitoring loop because of Err(RecvError::Lagged(_))

finally, shutdown_signal.await; returns and we exit MonitoringServer::run:

sv2-apps/stratum-apps/src/monitoring/http_server.rs

Lines 230 to 235 in 3d99603

	axum::serve(listener, app)
	.with_graceful_shutdown(async move {
	shutdown_signal.await;
	info!("Monitoring server received shutdown signal, stopping...");
	})
	.await?;