feat: Demozoo, Pouët, and CSDb metadata handlers - #4221
Open
axelquack wants to merge 2 commits into
Open
Conversation
Identify demoscene productions as ROMs via filename tags, Edit-ROM id/URL paste, Demozoo ?title= + rank, unique Pouët search.php 302, and CSDb XML by id (no HTML scrape). Scene ids lock name/summary/cover so game catalogs cannot attach a similarly titled box. CSDb XML is parsed with defusedxml and capped; screenshot and extra links are stored only when they are http(s). Fixes rommapp#1796 (Phase 0–2).
Add the three sources to Classic and v2 settings, Scan, and Edit ROM. Edit ROM accepts a production URL or a bare id. Metadata tab extra links render only http(s) hrefs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes the first shippable slice of #1796: treat demoscene productions as ROMs on platforms RomM already has (DOS, Amiga, C64, NES, …) and identify them from Demozoo, Pouët, and CSDb.
This is Phase 0–2 of the plan on that issue. No new screens. No HTML scraping of Pouët
prod.php. No party browser.Demos live next to games in a mixed library. Filter → Metadata providers → Demozoo / Pouët / CSDb.
How matching works
(demozoo-108),(pouet-63),(csdb-75330)— same idea as(hltb-12345)https://demozoo.org/productions/108/, Pouëtprod.php?which=, CSDbrelease/?id=)GET /api/v1/productions/?title=(never?search=— that endpoint is unused / wrong for this). Jaro–Winkler rank, then refetch by idGET search.php?what=&type=prodwith redirects off. Unique hit →302 Location: …?which=N. Several hits →200HTML → leave unmatched (no parse)PouetProduction/CsdbRelease→ fetch Pouët JSON / CSDb XML by idPouët JSON is
GET https://api.pouet.net/v1/prod/?id=Nonly. CSDb is the public XML webservice, id only (?type=release&id=N&depth=2). No CSDb search.Scene cover lock
Game catalogs fuzzy-match demo titles onto the wrong box (Desert Dream → Desert Fox). Once a ROM has
demozoo_id,pouet_id, orcsdb_id, only those three sources may write name / summary / cover. Regular games never have those ids, so IGDB / SteamGridDB / Libretro stay as they are.CSDb exists mainly for C64 stills: Demozoo often has
screenshots-neededand an empty shot list; Scan follows the DemozooCsdbReleaselink and uses CSDb’sScreenShot.Overview
Demozoo has no prose description. Scan writes one
summaryline:Pouët is later in artwork/metadata priority, so Scan splices score / rank / CdC onto the Demozoo line instead of letting Pouët wipe party and credits. Rank is omitted past #2000.
Short flags when the JSON has them:
no sound(tagno-sound),hidden part(taghidden-partorhidden_parts). Empty notes are skipped.The blob stores a clean
partyname + Demozooparty_idand keeps the mashedparty_linefor Overview. That is not browse — it only means a future virtual-collection type does not need a second rescan.YouTube: first watch id from Demozoo, else Pouët. Classic embeds it. v2 has no player yet, so Overview shows the watch URL.
Configuration
No API keys. Off with env flags (default on, same as other public sources):
Changes
Backend
backend/handler/metadata/demozoo_handler.py— id / filename tag /?title=+ rank; Overview builder; party / party_id / party_line; URL parsebackend/handler/metadata/pouet_handler.py— id / filename tag / uniquesearch.php302; JSON?id=onlybackend/handler/metadata/csdb_handler.py— id / filename tag / DemozooCsdbRelease; XML webservice, no searchbackend/handler/scan_handler.py—SCENE_METADATA_SOURCES, Pouët follow + vote splice, CSDb cover fallbackbackend/models/rom.py+0108_demozoo_pouet_metadata.py+0109_csdb_metadata.py—demozoo_id/pouet_id/csdb_id+ JSON blobs; facets mirroredbackend/endpoints/roms/__init__.py— Edit ROM accepts ids or production URLs (scene_id_or_none;safe_intof a URL would become0)?title=hits (Pouët picker is not in this PR)production_to_rom, Pouët 302, CSDb XML, scan lock / spliceFrontend
frontend/src/utils/sceneIds.ts+ testsSecurity notes
defusedxml(same as gamelist / LaunchBox), response size capped.http/https(Metadata tabhref).search.phpis called withfollow_redirects=False; we readLocationforwhich=Nand never fetch or parse the HTML body.[A-Za-z0-9_-]only.Not in this PR
(pouet-N), paste URL, or match Demozoo and follow the Pouët linkfilter_roms(party=)+ virtual-collection schema. Cleanparty/party_idare stored for laterdemozoo_metadata; gallery readsgenerated_*columns. Alembic, not a handler tweakprod.phpHTML?search=?title=Happy to split CSDb into a follow-up if you want this PR smaller.
How I tested
test_demozoo_handler,test_pouet_handler,test_csdb_handler, scan-priority)AI assistance disclosure
This PR was developed with substantial AI assistance (Grok, xAI). I wrote the design on #1796, chose the APIs (Demozoo REST, Pouët JSON + exact-match 302, CSDb XML id-only), and tested against live services and my own library. Most of the code was written by the model under that direction and review. Replies on this PR that I paste from the same session will also be AI-assisted; I will say so if that happens.
Checklist
Fixes #1796 (first slice; leftover items listed above)