Skip to content

feat: Demozoo, Pouët, and CSDb metadata handlers - #4221

Open
axelquack wants to merge 2 commits into
rommapp:masterfrom
axelquack:feat/demozoo-pouet-phase0
Open

feat: Demozoo, Pouët, and CSDb metadata handlers#4221
axelquack wants to merge 2 commits into
rommapp:masterfrom
axelquack:feat/demozoo-pouet-phase0

Conversation

@axelquack

Copy link
Copy Markdown

Description

Closes the first shippable slice of #1796: treat demoscene productions as ROMs on platforms RomM already has (DOS, Amiga, C64, NES, …) and identify them from Demozoo, Pouët, and CSDb.

This is Phase 0–2 of the plan on that issue. No new screens. No HTML scraping of Pouët prod.php. No party browser.

Demos live next to games in a mixed library. Filter → Metadata providers → Demozoo / Pouët / CSDb.

How matching works

Step Source What
1 Filename tags (demozoo-108), (pouet-63), (csdb-75330) — same idea as (hltb-12345)
2 Edit ROM Paste a numeric id or a production URL (https://demozoo.org/productions/108/, Pouët prod.php?which=, CSDb release/?id=)
3 Demozoo title GET /api/v1/productions/?title= (never ?search= — that endpoint is unused / wrong for this). Jaro–Winkler rank, then refetch by id
4 Pouët exact title GET search.php?what=&type=prod with redirects off. Unique hit → 302 Location: …?which=N. Several hits → 200 HTML → leave unmatched (no parse)
5 Follow links Demozoo PouetProduction / CsdbRelease → fetch Pouët JSON / CSDb XML by id

Pouët JSON is GET https://api.pouet.net/v1/prod/?id=N only. CSDb is the public XML webservice, id only (?type=release&id=N&depth=2). No CSDb search.

Scene cover lock

Game catalogs fuzzy-match demo titles onto the wrong box (Desert DreamDesert Fox). Once a ROM has demozoo_id, pouet_id, or csdb_id, only those three sources may write name / summary / cover. Regular games never have those ids, so IGDB / SteamGridDB / Libretro stay as they are.

CSDb exists mainly for C64 stills: Demozoo often has screenshots-needed and an empty shot list; Scan follows the Demozoo CsdbRelease link and uses CSDb’s ScreenShot.

Overview

Demozoo has no prose description. Scan writes one summary line:

Demo by Future Crew (1993) · Assembly 1993 / PC Demo #1 ·
Music: Purple Motion · Graphics: Pixel ·
Pouët 0.941 · #75 ·
https://youtube.com/watch?v=… · https://demozoo.org/productions/108/ ·
https://www.pouet.net/prod.php?which=63

Pouët is later in artwork/metadata priority, so Scan splices score / rank / CdC onto the Demozoo line instead of letting Pouët wipe party and credits. Rank is omitted past #2000.

Short flags when the JSON has them: no sound (tag no-sound), hidden part (tag hidden-part or hidden_parts). Empty notes are skipped.

The blob stores a clean party name + Demozoo party_id and keeps the mashed party_line for Overview. That is not browse — it only means a future virtual-collection type does not need a second rescan.

YouTube: first watch id from Demozoo, else Pouët. Classic embeds it. v2 has no player yet, so Overview shows the watch URL.

Configuration

No API keys. Off with env flags (default on, same as other public sources):

DEMOZOO_API_ENABLED=true
POUET_API_ENABLED=true
CSDB_API_ENABLED=true

Changes

Backend

  • backend/handler/metadata/demozoo_handler.py — id / filename tag / ?title= + rank; Overview builder; party / party_id / party_line; URL parse
  • backend/handler/metadata/pouet_handler.py — id / filename tag / unique search.php 302; JSON ?id= only
  • backend/handler/metadata/csdb_handler.py — id / filename tag / Demozoo CsdbRelease; XML webservice, no search
  • backend/handler/scan_handler.pySCENE_METADATA_SOURCES, Pouët follow + vote splice, CSDb cover fallback
  • backend/models/rom.py + 0108_demozoo_pouet_metadata.py + 0109_csdb_metadata.pydemozoo_id / pouet_id / csdb_id + JSON blobs; facets mirrored
  • backend/endpoints/roms/__init__.py — Edit ROM accepts ids or production URLs (scene_id_or_none; safe_int of a URL would become 0)
  • Identify search already lists Demozoo ?title= hits (Pouët picker is not in this PR)
  • Tests for tag parse, URL parse, production_to_rom, Pouët 302, CSDb XML, scan lock / splice

Frontend

  • Setup wizard + Settings → Metadata sources (logos, flags)
  • Scan settings includes the three sources
  • Edit ROM (Classic + v2): Demozoo / Pouët / CSDb id fields; URL paste
  • Provider chips / Metadata tab extra download links
  • i18n keys in all locales
  • frontend/src/utils/sceneIds.ts + tests

Security notes

  • All outbound fetches go through RomM’s existing SSRF-protected httpx client. Handler URLs are hardcoded hosts + integer ids (no user-controlled fetch target).
  • CSDb XML is untrusted remote input: parsed with defusedxml (same as gamelist / LaunchBox), response size capped.
  • Screenshot and extra links are stored and rendered only when the scheme is http/https (Metadata tab href).
  • Pouët search.php is called with follow_redirects=False; we read Location for which=N and never fetch or parse the HTML body.
  • YouTube embed ids are 11-char [A-Za-z0-9_-] only.

Not in this PR

Left out Why
Pouët Identify picker Ambiguous titles return 200 HTML. We do not parse it. Workaround: (pouet-N), paste URL, or match Demozoo and follow the Pouët link
Party / group browser in RomM Needs filter_roms(party=) + virtual-collection schema. Clean party / party_id are stored for later
Demozoo types/tags in gallery genre filters Already on demozoo_metadata; gallery reads generated_* columns. Alembic, not a handler tweak
v2 YouTube player Id is saved. Classic embeds. v2 shows the URL
Pouët prod.php HTML Out of policy. JSON + 302 only
Demozoo ?search= Wrong endpoint. Title filter is ?title=

Happy to split CSDb into a follow-up if you want this PR smaller.

How I tested

  • Live Demozoo / Pouët / CSDb against a mixed retail + scene library (DOS, Amiga, C64, NES)
  • Filename tags, Demozoo title match, Pouët unique 302, CSDb cover when Demozoo has no stills
  • Scene lock: IGDB left on; demo titles did not pick up game box art
  • Edit ROM URL paste (Demozoo / Pouët / CSDb)
  • Handler unit tests (test_demozoo_handler, test_pouet_handler, test_csdb_handler, scan-priority)

AI assistance disclosure

This PR was developed with substantial AI assistance (Grok, xAI). I wrote the design on #1796, chose the APIs (Demozoo REST, Pouët JSON + exact-match 302, CSDb XML id-only), and tested against live services and my own library. Most of the code was written by the model under that direction and review. Replies on this PR that I paste from the same session will also be AI-assisted; I will say so if that happens.

Checklist

  • I've tested the changes locally
  • I've updated relevant comments
  • I've assigned reviewers for this PR
  • I've added unit tests that cover the changes

Fixes #1796 (first slice; leftover items listed above)

Identify demoscene productions as ROMs via filename tags, Edit-ROM
id/URL paste, Demozoo ?title= + rank, unique Pouët search.php 302,
and CSDb XML by id (no HTML scrape). Scene ids lock name/summary/cover
so game catalogs cannot attach a similarly titled box.

CSDb XML is parsed with defusedxml and capped; screenshot and extra
links are stored only when they are http(s).

Fixes rommapp#1796 (Phase 0–2).
Add the three sources to Classic and v2 settings, Scan, and Edit ROM.
Edit ROM accepts a production URL or a bare id. Metadata tab extra
links render only http(s) hrefs.
@gantoine
gantoine self-requested a review August 17, 2026 12:56
@gantoine gantoine added the on-hold Pending further research or blocked by another issue label Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

on-hold Pending further research or blocked by another issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata support & integration for "demoscene" releases

2 participants