Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 12 additions & 16 deletions apps/subgraph/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,20 @@

A dealbot-owned Graph Protocol subgraph.

## What it indexes
What it indexes
Comment thread
dennis-tra marked this conversation as resolved.

- **PDPVerifier** — dataset lifecycle, piece add/remove, proving periods.
- **FilecoinWarmStorageService (FWSS)** — payer/service-provider metadata, `withIPFSIndexing` flag, `ipfsRootCID` per piece, service/payment termination.

## Motivation

The motivation comes from the anonymous retrieval check from [#427](https://github.com/FilOzone/dealbot/issues/427): probing SPs for pieces dealbot did not create, so an SP can't preferentially serve pieces when "the teacher is watching". To pick those pieces fairly we need to query any FWSS-managed piece on a given SP, including `withIPFSIndexing` and `ipfsRootCID`, uniformly at random across the SP's entire active piece set. Neither existing subgraph supports that shape:

- [FilOzone/pdp-explorer](https://github.com/FilOzone/pdp-explorer): indexes the PDPVerifier contract only. No FWSS-level fields, so it can't tell us whether a piece is CAR-validatable (`withIPFSIndexing`) or what its `ipfsRootCID` is.
- [FIL-Builders/fwss-subgraph](https://github.com/FIL-Builders/fwss-subgraph): FWSS-centric, but doesn't expose the joined PDP+FWSS view we need (active piece set per SP, with IPFS metadata attached) and isn't on a release cadence we control.

`@dealbot/subgraph` is intentionally a strict functional superset of `pdp-explorer`: same PDPVerifier coverage, plus the FWSS fields we need. Once we're confident in its correct operation it can become a drop-in replacement for `PDP_SUBGRAPH_ENDPOINT`, but for now the basic retrieval and data-retention checks continue to use `pdp-explorer`.

## Contract ABIs

`abis/*.json` is generated by `scripts/sync-abis.mjs` from the canonical
Expand All @@ -31,20 +40,7 @@ pnpm build:calibration
pnpm test
```

## Deploy

Requires `goldsky` CLI authenticated via `GOLDSKY_API_KEY`.

```bash
export VERSION=0.1.0
pnpm build:calibration
pnpm deploy:calibration

pnpm build:mainnet
pnpm deploy:mainnet
```
## Release and Deployment

Goldsky slots (slugs TBD):
The subgraph is versioned and released independently of `apps/backend` and `apps/web`. The end-to-end checklist lives in [`docs/release-subgraph.md`](../../docs/release-subgraph.md).

- `dealbot-mainnet/<version>` — mainnet
- `dealbot-calibration/<version>` — calibration
2 changes: 1 addition & 1 deletion apps/subgraph/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"build:calibration": "pnpm run codegen && graph build --network filecoin-testnet",
"deploy:mainnet": "goldsky subgraph deploy dealbot-mainnet/$VERSION --path ./build",
"deploy:calibration": "goldsky subgraph deploy dealbot-calibration/$VERSION --path ./build",
"test": "graph test"
"test": "graph test && node --test scripts/check-networks.test.mjs"
},
"dependencies": {
"@filoz/synapse-core": "0.3.3",
Expand Down
36 changes: 36 additions & 0 deletions apps/subgraph/scripts/check-networks.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Asserts apps/subgraph/networks.json stays in sync with the contract
// addresses shipped by @filoz/synapse-core (which is generated from
// FilOzone/filecoin-services). Bumping synapse-core is the trigger for any
// address change; this test fails fast if networks.json drifts.

import { readFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { test } from "node:test";
import assert from "node:assert/strict";
import { generated } from "@filoz/synapse-core/abis";

const { filecoinWarmStorageServiceAddress, pdpVerifierAddress } = generated;

const here = dirname(fileURLToPath(import.meta.url));
const networksPath = join(here, "..", "networks.json");
const networks = JSON.parse(await readFile(networksPath, "utf8"));

const cases = {
"filecoin": 314,
"filecoin-testnet": 314159,
};

for (const [network, chainId] of Object.entries(cases)) {
test(`${network} PDPVerifier address matches synapse-core[${chainId}]`, () => {
const actual = networks[network]?.PDPVerifier?.address;
const expected = pdpVerifierAddress[chainId];
assert.equal(actual, expected, `expected ${expected}, got ${actual} for ${network}.PDPVerifier.address`)
});

test(`${network} FilecoinWarmStorageService address matches synapse-core[${chainId}]`, () => {
const actual = networks[network]?.FilecoinWarmStorageService?.address;
const expected = filecoinWarmStorageServiceAddress[chainId];
assert.equal(actual, expected, `expected ${expected}, got ${actual} for ${network}.FilecoinWarmStorageService.address`);
});
}
2 changes: 1 addition & 1 deletion docs/release-process.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Release Process

Dealbot uses [release-please](https://github.com/googleapis/release-please) to drive semver releases from [Conventional Commits](https://www.conventionalcommits.org/), and ArgoCD Image Updater to promote those releases into Kubernetes. For release-please mechanics (commit parsing, version bumps, troubleshooting) see [release-please-flow.md](release-please-flow.md). For container images, manifests, and runtime topology see [deployment.md](deployment.md). For ingress, egress, persistence, secrets, and observability expectations see [infra.md](infra.md).
Dealbot uses [release-please](https://github.com/googleapis/release-please) to drive semver releases from [Conventional Commits](https://www.conventionalcommits.org/), and ArgoCD Image Updater to promote those releases into Kubernetes. For release-please mechanics (commit parsing, version bumps, troubleshooting) see [release-please-flow.md](release-please-flow.md). For container images, manifests, and runtime topology see [deployment.md](deployment.md). For ingress, egress, persistence, secrets, and observability expectations see [infra.md](infra.md). For the subgraph release checklist see [release-subgraph.md](release-subgraph.md).

## Flow

Expand Down
114 changes: 114 additions & 0 deletions docs/release-subgraph.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Subgraph Release Checklist

Source of truth for releasing and deploying [`apps/subgraph/`](../apps/subgraph/).

The dealbot subgraph is versioned independently from `apps/backend` and `apps/web`. Those two go through a release process as described in [release-process.md](release-process.md). The subgraph is published to [Goldsky](https://goldsky.com/) on its own cadence following the steps below.

When cutting a release, you may copy the checklist into a new GitHub issue titled `Release subgraph vX.Y.Z` and tick the boxes as you go. The structure mirrors [filecoin-pay-explorer's release template](https://github.com/FilOzone/filecoin-pay-explorer/blob/main/.github/ISSUE_TEMPLATE/release.md).

## Notes

- Dealbot's backend consumes the subgraph at a `prod`-tagged URL (e.g. `.../dealbot-mainnet/prod/gn`). Re-tagging is what actually promotes a new version to production (no backend redeploy required).
- This checklist assumes a **backward-compatible** change. For a breaking change see [Breaking changes](#breaking-changes) at the bottom.

## 1. Define the new version

```bash
NEW_RELEASE_VERSION="X.Y.Z" # no v prefix — used as the Goldsky version segment
CURRENT_VERSION="X.Y.Z" # version currently tagged as `prod` (will be replaced)
```

- [ ] Version strings agreed and set
- [ ] Bump follows semver: additive → minor, fix → patch, breaking → major

## 2. Create a GitHub release

```bash
gh release create "subgraph-v$NEW_RELEASE_VERSION" \
--title "subgraph v$NEW_RELEASE_VERSION" \
--generate-notes
```

- [ ] GitHub release created with tag `subgraph-vX.Y.Z`

The `subgraph-` prefix disambiguates from backend / web releases, which are tagged on the same repo by release-please.

## 3. Build and publish the subgraph

> **Note:** Tag-triggered CI for subgraph deploys is not wired up yet — for now every release uses the manual fallback. Tracked as follow-up in [#573](https://github.com/FilOzone/dealbot/issues/573); target is to match [filecoin-pay-explorer's `deploy.yml`](https://github.com/FilOzone/filecoin-pay-explorer/blob/main/.github/workflows/deploy.yml).

The following commands need to be run in the `apps/subgraph` directory and require the `goldsky` CLI authenticated via a `GOLDSKY_API_KEY`.

```bash
export VERSION=X.Y.Z # no v prefix; used as the Goldsky version segment

pnpm build:calibration
pnpm deploy:calibration

pnpm build:mainnet
pnpm deploy:mainnet
```

- [ ] `dealbot-calibration/$NEW_RELEASE_VERSION` deployed
- [ ] `dealbot-mainnet/$NEW_RELEASE_VERSION` deployed

## 4. Await subgraph indexing

- [ ] Received confirmation email that `dealbot-calibration` finished indexing
- [ ] Received confirmation email that `dealbot-mainnet` finished indexing

## 5. Smoke-test the candidate in staging

Point dealbot staging at the new versioned URL. The cleanest way is via a `staging` tag so the staging backend's config can stay stable:

```bash
for network in calibration mainnet; do
goldsky subgraph tag delete dealbot-$network/$CURRENT_VERSION --tag staging 2>/dev/null || true
goldsky subgraph tag create dealbot-$network/$NEW_RELEASE_VERSION --tag staging
done
```

- [ ] `dealbot-mainnet/$NEW_RELEASE_VERSION` tagged as `staging`
- [ ] `dealbot-calibration/$NEW_RELEASE_VERSION` tagged as `staging`
- [ ] Dealbot staging backend's subgraph-dependent checks succeed against the new subgraph
- [ ] No new errors in dealbot staging logs

## 6. Promote subgraphs to `prod`

> [!NOTE]
> Re-tagging as `prod` below switches the **production** dealbot backend over to the new subgraph version. Do this only when you're ready to take it live.

```bash
for network in calibration mainnet; do
# `tag create` errors if `prod` already points at another version, so we
# delete first. The window between the two calls is sub-second; consumers
# querying the `prod` URL in that window will get an error.
goldsky subgraph tag delete dealbot-$network/$CURRENT_VERSION --tag prod 2>/dev/null || true
goldsky subgraph tag create dealbot-$network/$NEW_RELEASE_VERSION --tag prod
done
```

- [ ] `dealbot-mainnet` tagged as `prod`
- [ ] `dealbot-calibration` tagged as `prod`

## 7. Verify production

- [ ] Dealbot production backend's subgraph-dependent checks are healthy
- [ ] No regressions in production metrics or dashboards

## 8. Wrap-up

- [ ] Announce the release in `#fil-foc`
- [ ] Capture any improvements needed for next time
- [ ] Close this issue

---

## Breaking changes

When the schema or entity shape changes in a way the current backend can't read, the order changes — re-tagging `prod` first would take the backend down:

1. Steps 1–4 as above (cut & deploy the new version, wait for indexing).
2. Land the backend changes that consume the new shape, configured to read from the new **versioned** URL (not `prod`). Deploy to dealbot staging and validate end-to-end against `vX.Y.Z`.
3. Once dealbot production is on a backend release that understands the new shape (either both shapes during a migration, or new-only), proceed to step 6 to re-tag `prod`.
4. Rollback: re-tag the previous subgraph version back to `prod`. The `prod` tag is the rollback lever — a single Goldsky API call, no backend redeploy required.