Skip to content

Commit 4cee7e0

Browse files
committed
merge origin/main into fix/279-advisory-lock-session-affinity
Reconcile the advisory-lock series with main after #173, #330, and related landings. Conflict resolution in repo_store, repos, issues, pulls, ipfs, and main.
2 parents 9dd71e9 + 73fd747 commit 4cee7e0

49 files changed

Lines changed: 33316 additions & 2462 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -109,9 +109,14 @@ GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false
109109

110110
# Require the authenticated pusher to be the repo owner on git-receive-pack.
111111
# A valid did:key signature is authentication, not authorization: anyone can
112-
# sign as their own DID. When true, pushes from a non-owner DID are rejected.
113-
# Keep false until the repo owner is ready for owner-only writes.
114-
GITLAWB_ENFORCE_OWNER_PUSH=false
112+
# mint a key, derive its DID and sign, so with this off every signed caller may
113+
# push to every repository, private ones included. On by default.
114+
#
115+
# Set to false only for a rolling upgrade whose pushers are not yet the repo
116+
# owner. Note that delegated and CI keys count as non-owners: UCAN git/push is
117+
# verified but not yet honored for authorization, so they cannot push while this
118+
# is on. See docs/RUN-A-NODE.md.
119+
GITLAWB_ENFORCE_OWNER_PUSH=true
115120

116121
# Comma-separated libp2p multiaddrs.
117122
# Example: /ip4/1.2.3.4/udp/7546/quic-v1/p2p/12D3KooW...
@@ -206,8 +211,9 @@ GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
206211
# many WRITE-pool slots. This one is load-bearing for push availability, since
207212
# it is acquired before the global write permit: without it, one host minting
208213
# disposable did:key identities could open enough slow pushes to monopolize the
209-
# write pool and 503 every other source (owner enforcement defaults off, and
210-
# the push rate limiter caps arrival rate, not in-flight concurrency).
214+
# write pool and 503 every other source (the push rate limiter caps arrival
215+
# rate, not in-flight concurrency, and owner enforcement can be turned off for
216+
# a rolling upgrade).
211217
# Keyed on the resolved source IP, never the DID, so a DID farm does not defeat
212218
# them; keying granularity follows GITLAWB_TRUSTED_PROXY like the read cap above.
213219

@@ -228,10 +234,26 @@ GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
228234
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
229235
# Default 4.
230236
GITLAWB_IPFS_WALK_PER_SOURCE=4
237+
# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
238+
# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
239+
# truncated scan sheds a retryable 503, never a false 404. Default 256.
240+
GITLAWB_IPFS_MAX_LEGACY_PROBES=256
241+
# Max repo ROWS one /ipfs request's legacy scan may read from the database. The probe
242+
# ceiling above only counts once a probe runs, and quarantined or private repos are
243+
# denied before that, so without this an all-denying inventory paged the whole repo
244+
# table for one anonymous request. A truncated scan sheds a retryable 503 carrying a
245+
# sealed `continuation` token; echo it as ?scan= to resume, so a holder buried past the
246+
# ceiling is reachable in ceil(repos / ceiling) + 1 requests. Raising this also raises
247+
# the per-caller /ipfs work allowance, since each page is charged to it. Lowering it
248+
# sharpens a coarse oracle: laddering to the end reveals the node's total repo count to
249+
# within one ceiling. Default 2048.
250+
GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS=2048
231251
# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
232252
# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
233253
# skipped without a verdict and the scan continues; if the object is then found
234-
# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
254+
# nowhere the request sheds a retryable 503 instead of a false 404. The effective
255+
# cap is the tighter of this value and the node's internal per-request ceiling of
256+
# 17, so values above 17 have no effect; lower values do tighten it. Default 64.
235257
GITLAWB_IPFS_MAX_REPOS_WALKED=64
236258
# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
237259
# visit costs a repo acquire — on a Tigris cache miss a full archive download,
@@ -252,6 +274,15 @@ GITLAWB_IPFS_MAX_REPO_VISITS=1024
252274
# Must be 1..=3153600000 (100 years): the node derives an Instant deadline from
253275
# this value, and a larger one cannot be represented. Default 600.
254276
GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
277+
# Shorter budget (seconds) for the pre-walk CID resolve: the lookup that maps a
278+
# requested CID to its git oid(s), which runs while the scarce walk admission is
279+
# already held. A well-formed CID with no pin row does no probe and no walk work,
280+
# so without this a stalled lookup could hold a walk slot for the whole request
281+
# budget while nothing walked. The effective deadline is the lesser of this and
282+
# the remaining request budget; walk and probe work stay on the request budget,
283+
# so a slow but progressing scan is never shed by it.
284+
# Must be 1..=3153600000 (100 years). Default 10.
285+
GITLAWB_IPFS_RESOLVE_BUDGET_SECS=10
255286
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
256287
# from the concurrency caps above). 0 disables. Default 600.
257288
GITLAWB_IPFS_RATE_LIMIT=600

‎Cargo.lock‎

Lines changed: 10 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,9 @@ chrono = { version = "0.4", features = ["serde"] }
5353
uuid = { version = "1", features = ["v4"] }
5454
# http client
5555
reqwest = { version = "0.12", features = ["blocking", "json", "multipart", "rustls-tls"], default-features = false }
56+
# URL parsing (what reqwest::Url re-exports, so the shared redirect predicate can
57+
# take a parsed URL without pulling reqwest into gitlawb-core)
58+
url = "2"
5659
# HMAC
5760
hmac = "0.12"
5861

‎README.md‎

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,53 @@ Notes:
229229
sends the bearer token **only** to your configured origin, never to a URL a
230230
node advertises, so a hostile node can't capture the key or redirect the solve.
231231

232+
### Fetching an object by CID
233+
234+
```bash
235+
gl ipfs list # CIDs this node has pinned
236+
gl ipfs get bafkrei... > object.bin # object bytes on stdout
237+
```
238+
239+
Objects that were pinned before the node started recording which repo they came
240+
from are found by scanning its repo inventory, and that scan stops at the
241+
per-request ceilings in the [Configuration](#configuration) table. A stopped scan
242+
answers 503 with a resume token instead of a false "not found", and `gl ipfs get`
243+
follows the token automatically: up to 8 resumes after the first request, so at
244+
most 9 calls to the node, waiting between attempts for as long as the node's
245+
`Retry-After` asks and never longer than 5 seconds.
246+
247+
The whole ladder runs under a 60 second wall-clock deadline. The deadline bounds
248+
the search, not the download: each attempt gets the time left on it to produce
249+
response headers, and once an object is found its bytes stream outside that
250+
deadline. They are not unbounded, though. The client's own 30 second HTTP timeout
251+
is a total request timeout, running from the start of a request until its body
252+
has finished, so a transfer still going 30 seconds after its request began is cut
253+
off. Waits between attempts never run past the deadline either, so a single run
254+
spends at most around 90 seconds on the network: the deadline plus the 30 second
255+
timeout covering the last attempt. Writing the object out sits outside both
256+
bounds, so piping into a reader that stops reading can hold the command open
257+
longer than that.
258+
259+
Two node-side brakes end a ladder early and are reported rather than retried
260+
around. A 429 is terminal, because the node's rate-limit window is an hour and
261+
that wait cannot be honored inside one invocation; a transient overload (a 503
262+
carrying no incomplete-scan code) is retried on the token already held, under the
263+
same cap, clamp and deadline. The per-IP fanout brake can also stop a ladder well
264+
short of the 9 calls, so automatic resumption is not a guarantee of reaching the
265+
object.
266+
267+
When a bound stops the ladder with a usable token in hand, the command prints the
268+
token and the invocation that continues from it before exiting nonzero:
269+
270+
```txt
271+
resume from where this stopped: gl ipfs get bafkrei... --scan <token>
272+
```
273+
274+
Run that to carry on from where the scan stopped. Re-running without `--scan`
275+
restarts at the first row, reproduces the same truncation and spends the node's
276+
per-IP budget again, so the token is the only thing that makes progress. Tokens
277+
are valid for an hour.
278+
232279
---
233280

234281
## Architecture
@@ -345,7 +392,8 @@ Important node settings:
345392
| `GITLAWB_BOOTSTRAP_PEERS` | Comma-separated HTTP peer URLs. |
346393
| `GITLAWB_P2P_BOOTSTRAP` | Comma-separated libp2p multiaddrs. |
347394
| `GITLAWB_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. |
348-
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
395+
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. |
396+
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
349397
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
350398
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
351399
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
@@ -357,16 +405,21 @@ Important node settings:
357405
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
358406
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
359407
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
360-
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Default 64. |
408+
| `GITLAWB_IPFS_MAX_LEGACY_PROBES` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
409+
| `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS` | Max repo rows one `/ipfs/{cid}` request's legacy scan may read from the database. The probe ceiling above only starts counting once a probe runs, and quarantined or private repos are denied before that, so this is what bounds a scan over an inventory that denies the caller everywhere. A truncated scan sheds a retryable 503 carrying an opaque `continuation` token; echoing it as `?scan=` resumes the scan where it stopped. Every per-request ceiling on this path (rows, probes, visits, retained rule bytes) mints one, so each request advances the ladder by at least the rows it read and a holder buried past a ceiling is reached in a bounded number of requests, `ceil(repos / ceiling) + 1` when this row ceiling is the one that binds. No ceiling ever produces a 404. Every page is charged to the caller's `/ipfs` work allowance, so raising this raises that allowance too. Lowering it sharpens an oracle: because a truncation emits a token and a completed wrap does not, laddering to the end reveals the node's total repo count (private and quarantined included) to within one ceiling. Default 2048. |
410+
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks a `/ipfs/{cid}` request may run per phase; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. The effective cap is the tighter of this knob and the node's internal history-walk ceiling of 17 (`MAX_PIN_SOURCES + 1`), so a value above 17 has no effect while a value below it does tighten the cap. It is charged per phase: the provenance lookup and the legacy-scan fallback get separate equal budgets, so one request can run up to twice the cap in total. Default 64. |
361411
| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
362412
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
413+
| `GITLAWB_IPFS_RESOLVE_BUDGET_SECS` | Shorter budget for the pre-walk CID resolve inside an admitted `/ipfs/{cid}` request: the lookup that maps the requested CID to its git oid(s), which runs while the scarce walk admission is already held. A well-formed CID with no pin row does no probe and no walk work, so without this it could hold a walk slot for the whole request budget while nothing walked, and enough such requests shed every real retrieval at admission. The effective deadline is the lesser of this and the remaining request budget, so a value above `GITLAWB_IPFS_REQUEST_BUDGET_SECS` degrades to the request budget. Only the resolve is on this clock; walk and probe work stay on the request budget, so a slow but progressing scan is never shed by it. Default 10. Accepted range is 1 to 3153600000 (100 years). |
363414
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
364415
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
365416
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
366417
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
367418

368419
Production note: change the default Postgres password before exposing a node publicly.
369420

421+
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
422+
370423
---
371424

372425
## Optional node staking

‎crates/git-remote-gitlawb/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ name = "git-remote-gitlawb"
1111
path = "src/main.rs"
1212

1313
[dependencies]
14-
gitlawb-core = { path = "../gitlawb-core" }
14+
gitlawb-core = { path = "../gitlawb-core", features = ["redirect"] }
1515
anyhow = { workspace = true }
1616
reqwest = { workspace = true }
1717
tracing = { workspace = true }

0 commit comments

Comments
 (0)