Skip to content

feat: fwss dataset/piece index enrichment#100

Closed
dennis-tra wants to merge 1 commit into
FilOzone:mainfrom
probe-lab:fwss-integration
Closed

feat: fwss dataset/piece index enrichment#100
dennis-tra wants to merge 1 commit into
FilOzone:mainfrom
probe-lab:fwss-integration

Conversation

@dennis-tra
Copy link
Copy Markdown

@dennis-tra dennis-tra commented Apr 21, 2026

This PR adds functionality to index the FilecoinWarmStorageService contract alongside PDPVerifier so a single pdp-explorer query can answer both "what PDP state is this dataset in?" and "what FWSS metadata was it created with?"

This is motivated by FilOzone/dealbot#427 (anonymous retrieval check). With this change, dealbot replaces a client-side join of two subgraphs with one native GraphQL query that filters on withIPFSIndexing, fwssPayer, and rawSize all at once.

Changelog

  • FilecoinWarmStorageService contract added to all three manifests handled in fwss.ts
  • ServiceTerminated flips DataSet.isActive = false.
  • PDPPaymentTerminated does not flip isActive. Clients compare to current epoch themselves. Is this correct? Should it also flip isActive?
  • ABI taken from foc-observer

There was one complication with the event ordering from the contract:

PDPVerifier._createDataSet calls listener.dataSetCreated(...) before emitting its own DataSetCreated (see pdp/src/PDPVerifier.sol:589-592), so FWSS's event has the lower log index and fires first. To avoid the FWSS handler running against a missing DataSet entity:

  • fwss.ts:handleFwssDataSetCreated: When first to run, creates a stub with required non-null defaults and sets FWSS fields.
  • pdp-verifier.ts:handleDataSetCreated — changed from new DataSet(...) to load-or-create. When the FWSS handler already created a stub, we load it (preserving FWSS fields) rather than overwrite.

Only DataSetCreated has this inverted ordering. PiecesAdded and StorageProviderChanged emit PDPVerifier-first, so those FWSS handlers are straightforward patch-the-existing-entity.

Downstream example

This unlocks richer filtering on existing dealbot queries. Example: GET_PROVIDERS_WITH_DATASETS in dealbot (apps/backend/src/pdp-subgraph/queries.ts) currently is:

query GetProvidersWithDataSet($addresses: [Bytes!], $blockNumber: BigInt!) {
  providers(where: {address_in: $addresses}) {
    address
    totalFaultedPeriods
    totalProvingPeriods
    proofSets(where: {nextDeadline_lt: $blockNumber, status: PROVING}) {
      nextDeadline
      maxProvingPeriod
    }
  }
}

With this PR, it can filter by FWSS metadata in the same query. Anonymous-retrieval piece selection becomes:

query AnonPieceCandidates(
  $sp: Bytes!, $dealbot: Bytes!, $minSize: BigInt!, $maxSize: BigInt!
) {
  provider(id: $sp) {
    proofSets(
      where: {
        isActive: true,
        status: PROVING,
        withIPFSIndexing: true,     # <-- new
        fwssPayer_not: $dealbot     # <-- new
      }
      first: 1000
    ) {
      setId
      fwssPayer
      roots(
        where: { removed: false, rawSize_gte: $minSize, rawSize_lte: $maxSize }
        first: 100
      ) {
        rootId
        cid
        rawSize
        ipfsRootCID                 # <-- new
      }
    }
  }
}

All filter dimensions (active state, piece size, IPFS indexing, payer exclusion) are native indexed filters.


I have deployed the new subgraphs for testing to Goldsky:


Transparency: Implementation heavily augmented by Claude Opus 4.7

@FilOzzy FilOzzy added this to FOC Apr 21, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Apr 21, 2026
@dennis-tra dennis-tra marked this pull request as ready for review April 21, 2026 09:41
@rvagg
Copy link
Copy Markdown

rvagg commented Apr 21, 2026

FWSS isn't the only user of PDPVerifier. We've held off on doing this because entangling the two creates a false view of the system. PDPVerifier is a utility contract that happens to be used by FWSS to get its work done. On mainnet, there's more volume going through PDPVerifier via the Storacha contract than FWSS at the moment and there's nothing stopping anyone else using it to do whatever they want.

We've debated enriching like this because it would be nice on the PDP Explorer to be able to dive deeper into them, and @juliangruber is currently looking at this from the top-down Filecoin Pay perspective. As it stands though this feels like a bit of a hack and maybe we should be thinking more creatively about how to make the subgraph code in here either reusable or extensible such that someone can take it and deploy their own subgraph based on it but either enriched with their custom context, like FWSS, or to fit in with some larger view.

@dennis-tra
Copy link
Copy Markdown
Author

Hi @rvagg, is your concern mainly that I tried to bolt a dealbot usecase onto the pdp-explorer or is the general direction not right? Remember I'm coming from the retrieval++ use case.

As it stands though this feels like a bit of a hack and maybe we should be thinking more creatively about how to make the subgraph code in here either reusable or extensible such that someone can take it and deploy their own subgraph based on it but either enriched with their custom context, like FWSS, or to fit in with some larger view.

Would it work if we took what's in this PR, moved it to the dealbot repository, and deployed a dedicated subgraph for dealbot that performs the aggregation? Currently dealbot seems to reuse this PDP Explorer's subgraph.

@SgtPooki
Copy link
Copy Markdown

Would it work if we took what's in this PR, moved it to the dealbot repository, and deployed a dedicated subgraph for dealbot that performs the aggregation? Currently dealbot seems to reuse this PDP Explorer's subgraph.

This sounds like a good plan TBH, but i'm not sure if we want to take on owning/deploying our own subgraph. How much work would it be to deploy a dealbot-specific subgraph?

@dennis-tra
Copy link
Copy Markdown
Author

Deploying subgraphs to Goldsky seems very easy (unless I'm missing something). In the PR description you can find two links at the bottom which point to my test deployments.

@dennis-tra dennis-tra closed this Apr 22, 2026
@github-project-automation github-project-automation Bot moved this from 📌 Triage to 🎉 Done in FOC Apr 22, 2026
dennis-tra added a commit to probe-lab/dealbot that referenced this pull request Apr 22, 2026
Imports the goldsky subgraph mappings from FilOzone/pdp-explorer#100 as
an in-tree package. This is the subgraph dealbot will own and deploy for
itself (motivated by dealbot#427 anonymous retrieval check).

Integrated with pnpm workspace, parameterized over networks.json for
mainnet (filecoin) and calibration (filecoin-testnet), and pinned
assemblyscript@0.19.23 so matchstick-as@0.6.0 picks up its binary.

Biome and root test/build scripts intentionally skip this package — it
is AssemblyScript compiled to WASM via graph-cli, and its lifecycle is
"rebuild and redeploy to Goldsky", not per-PR.

Schema, handlers, and tests are currently the unmodified upstream
pdp-explorer content; subsequent commits will trim them to the three
queries dealbot actually uses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dennis-tra added a commit to probe-lab/dealbot that referenced this pull request Apr 22, 2026
Imports the goldsky subgraph mappings from FilOzone/pdp-explorer#100 as
an in-tree package. This is the subgraph dealbot will own and deploy for
itself (motivated by dealbot#427 anonymous retrieval check).

Integrated with pnpm workspace, parameterized over networks.json for
mainnet (filecoin) and calibration (filecoin-testnet), and pinned
assemblyscript@0.19.23 so matchstick-as@0.6.0 picks up its binary.

Biome and root test/build scripts intentionally skip this package — it
is AssemblyScript compiled to WASM via graph-cli, and its lifecycle is
"rebuild and redeploy to Goldsky", not per-PR.

Schema, handlers, and tests are currently the unmodified upstream
pdp-explorer content; subsequent commits will trim them to the three
queries dealbot actually uses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🎉 Done

Development

Successfully merging this pull request may close these issues.

4 participants