[RFC] Magnet & BitTorrent support — design proposal, library/standards survey, and path to collaboration

[goutham80808]

Summary

This is a design RFC, not a request to rush an implementation. It consolidates research that hasn't appeared in the existing threads, proposes a concrete integration architecture that respects the concerns raised in #330, and offers to do the work. Posting it as an issue so the decision has a single citable design document instead of another "+1" comment.

Prior discussion (doing my homework)

• Torrent support #170 — the open tracking issue ("Torrent support"; user notes aria2 has it). 6 👍.
• I am against Torrents being added in surge downloader. #330 — @SuperCoolPencil (member) opens a detailed case against adding torrents to Surge core, quoting @sfzaw's 20+ years of experience. Proposes a separate surgebt TUI.
• Do you plan to support magnet links and ed2k link downloads? #247 — closed as duplicate.
• Cant connect to the extention #159 — passing mention.

I'm not here to relitigate #170 or dismiss #330. Several of #330's points are correct (Part 4). The goal is to bring new, concrete data and a design that separates the protocol from Surge's core identity, so the maintainers can decide with full information.

What's new here (not in #170 / #330)

Those threads are mostly positions ("please add" vs. "please don't"). They don't yet contain:

1. A specific Go library proven for this use case, with maturity/license data.
2. A direct Go precedent — a downloader in the same category as Surge already shipping HTTP+magnet+torrent.
3. The BEP standards an implementation follows.
4. A file-level integration sketch that keeps Surge's HTTP core untouched.
5. A point-by-point response to I am against Torrents being added in surge downloader. #330, with a path that concedes where I am against Torrents being added in surge downloader. #330 is right.

---

Part 1 — How the industry does it

Magnet links are not "HTTP in disguise." They require a real BitTorrent engine. Every major tool that supports magnets embeds one:

Tool	Engine	Approach
Free Download Manager (FDM)	libtorrent (C++)	Embedded native client
aria2	built-in (C++)	Native BitTorrent + DHT + metalink
qBittorrent	libtorrent-rasterbar	Native
Transmission	libtransmission	Native
Motrix	aria2 (bundled)	Native via aria2 daemon
Gopeed	anacrolix/torrent (Go)	Native, pure Go

Flow for a magnet link (same everywhere):

1. Parse magnet:?xt=urn:btih:<infohash>&dn=...&tr=...
2. Resolve metadata via DHT (BEP 5) + tracker announce (BEP 3 / BEP 15 UDP)
3. Exchange metadata with peers (BEP 9 ut_metadata) — no .torrent file needed
4. Discover peers via trackers + DHT + PEX (BEP 11)
5. Download pieces in parallel from the swarm; verify SHA-1 (v1) / SHA-256 (v2)
6. Reassemble; optionally seed

Relevant BEPs (among others): BEP 3 (base spec), BEP 5 (DHT), BEP 9 (magnet + metadata exchange), BEP 10 (extension protocol), BEP 11 (PEX), BEP 15 (UDP trackers), BEP 19 (HTTP/Web seeding — "webseeds"), BEP 6 (Fast extension), BEP 29 (uTP).

Note: Surge's headline speed advantage ("28s vs aria2c") does not apply to swarm downloads — torrent throughput is bounded by seeders, not connection optimization. This RFC does not promise faster torrents. The value is convenience (one tool) and, where webseeds exist, hybrid acceleration (Part 6, Phase 4).

---

Part 2 — The Go ecosystem (and the decisive precedent: Gopeed)

The pure-Go equivalent of libtorrent is github.com/anacrolix/torrent (figures below are from its README):

• Maturity: "used 24/7 in production by downstream services since late 2014." ~6k★, 668 forks, 4,249 commits.
• Design: library-first ("emphasis is on use as a library from other projects") — exactly what Surge needs.
• Features: DHT, PEX, uTP, protocol encryption, WebTorrent, WebSeeds, BitTorrent v2, holepunching.
• Streaming reader with seeking + readahead → maps directly onto Surge's existing Sequential / Streaming Mode.
• Storage backends: file, bolt, mmap, sqlite (Surge already depends on modernc.org/sqlite).
• License: MPL-2.0 (weak, file-scoped copyleft — compatible with Surge's MIT; no virality across files).

Gopeed — the proof point

anacrolix/torrent's README lists downstream projects, including:

Gopeed — "a high-speed downloader developed in Golang + Flutter, supports (HTTP, BitTorrent, Magnet) protocol, and supports all platforms."

Gopeed occupies the same product niche as Surge (high-speed Go downloader with a GUI) and ships HTTP + magnet + torrent on top of this library. Strong evidence this is (a) idiomatic in Go, (b) not out of scope for the category, and (c) achievable without a C/C++ dependency.

---

Part 3 — Proposed integration into Surge (HTTP core stays untouched)

Surge already has the clean branch point. In internal/download/manager.go, TUIDownload chooses between the concurrent and single HTTP downloaders:

useConcurrent := cfg.SupportsRange   // manager.go:157
if useConcurrent {
    // concurrent HTTP downloader     // manager.go:159
} else {
    // single HTTP downloader         // manager.go:215
}

The minimal change is one protocol-level branch upstream of that, so the HTTP path is literally unchanged:

URL ─┬─ "magnet:…"  or  *.torrent file  →  internal/engine/torrent/  (new)
     └─ http(s)://…                      →  existing HTTP path (0 diff)

Design rules (per Surge's conventions, and to contain #330's concerns):

• Isolated package: all BitTorrent code lives in internal/engine/torrent/ + a small resolver in internal/processing/. No torrent symbols leak into the HTTP engine.
• Reuse existing interfaces: the new downloader writes to the same ProgressState, emits the same events.*Msg, and pushes to the same ProgressCh chan<- any — so the TUI, engine pool, and SQLite history need minimal changes.
• Feature-flagged / opt-in: gated by a setting (or a build tag initially) so the default binary and UX are unchanged. Torrent is a guest protocol, not a new identity.
• Reversible: removing the package is one directory delete + one if removal.

⚠️ Honesty note: I have read the anacrolix/torrent README but have not yet run go get against this repo. The spike (Part 6) verifies the build before any commitment.

---

Part 4 — Responding to #330 point by point

@SuperCoolPencil / @sfzaw's concerns are substantive. Where each lands:

#330 concern	Response
Different protocol; IDM doesn't ship it	Fair. Counterpoint: aria2 (which Surge benchmarks against) does, and so does Gopeed. Ultimately a product call, not a technical blocker.
Would be half-baked vs. qBittorrent/Transmission	True if we try to be a full torrent client. Mitigation: target download-only MVP (no RSS, no auto-skip, no rule engine); explicitly document "for power torrent users, qBittorrent remains better."
Torrent TUI ≠ download TUI (design conflict)	Valid and the strongest argument. Mitigation: a distinct torrent view with torrent-specific widgets (peers, swarm speed, file tree), not a forced mash-up. Or concede fully → the surgebt split (Part 5, Path B).
Browser extension / default-app conflicts	Real. Mitigation: opt-in scheme registration; don't grab .torrent/magnet: by default.
Private-tracker stats precision	True — top private trackers whitelist specific clients; Surge wouldn't be whitelisted initially. Concede: private trackers are explicitly out of MVP scope; recommend qBittorrent for that use.
Seeding / cross-seeding / ratio / storage mgmt	Real and large. Mitigation: MVP = leech-only with optional time-limited seeding; full ratio management is explicitly post-MVP or belongs in surgebt.
"Keep them separate — build surgebt"	A genuinely good idea and fully compatible with this RFC (Path B).

Net: #330 is right that a full-featured torrent client doesn't belong in Surge core. Where this RFC differs is on download-only, opt-in, isolated support — and on presenting surgebt as an equal, not subordinate, option.

---

Part 5 — Two collaboration paths (I'll build either)

Path A — In-core, isolated, opt-in. Torrent lives in internal/engine/torrent/, feature-flagged, download-only. Smallest change; one binary. Best if maintainers want "magnet just works" without a second install.

Path B — Shared monorepo, separate surgebt binary (endorses #330). Extract reusable Surge internals (TUI widgets, ProgressState, SQLite history, engine types) into shared internal/ packages consumed by two cmd/ binaries: surge (HTTP, today) and surgebt (torrent, new). Cleanest separation; each tool gets a focused UX; no protocol mash-up. This is the strongest answer to #330.

I'm happy to do the work for either path. The decision is yours.

---

Part 6 — Phased MVP + spike offer

• Spike (read-only, I can run it first): go get github.com/anacrolix/torrent, a ~50-line program downloading a public Ubuntu magnet on Go 1.25 / Windows + Linux, measuring throughput and binary-size delta. Zero Surge changes. I'll post results here.
• Phase 1: Protocol dispatcher + surge <magnet> CLI (single-file). Reuses ProgressState. No TUI changes yet.
• Phase 2: TUI torrent view (metadata-resolving state, peer count, swarm speed, multi-file picker).
• Phase 3: Optional time-limited seeding; storage management.
• Phase 4 (differentiator): BEP 19 webseed URLs piped into Surge's existing concurrent HTTP engine for hybrid download — this is where Surge's speed story applies to torrent content.

Honest risks

• anacrolix/torrent is a heavy dependency (DHT/uTP/bencode) → larger binary. Unverified on Go 1.25; the spike confirms.
• Magnet metadata resolution adds a "resolving…" latency state that doesn't exist today.
• Multi-file torrents need new TUI.
• Private-tracker support is explicitly out of MVP scope.

---

Open questions for maintainers

1. Is Path B (surgebt) acceptable in principle? If so, is a shared-internal/ extraction agreeable?
2. If Path A, is a feature flag (off by default) acceptable as the delivery gate?
3. Any hard "no" on adding anacrolix/torrent (MPL-2.0) as a dependency?
4. Would you like to see the read-only spike results before deciding?

I'm offering to do the implementation work and coordinate here. Not asking for a fast decision — asking for a direction. Thanks for the project, and for the thoughtful pushback in #330; it genuinely shaped this proposal.

(cc @SuperCoolPencil — Part 4 is a point-by-point reply to your #330 concerns.)