Skip to content

Latest commit

 

History

History
708 lines (578 loc) · 46.5 KB

BACKEND.md

File metadata and controls

708 lines (578 loc) · 46.5 KB

Backend Reference

FastAPI app entry point: backend/main.py.

API Routes

Auth

Method Path Notes
POST /api/auth/login Username/password login
GET /api/auth/me Current authenticated user
GET /api/auth/mode Returns { multi_user: boolean }
PUT /api/auth/mode Admin-only toggle for single-user vs multi-user mode
GET /api/auth/users Admin-only user list
POST /api/auth/users Admin-only user creation
PUT /api/auth/users/{user_id} Admin-only user update
DELETE /api/auth/users/{user_id} Admin-only user delete; cascades owned data cleanup
PUT /api/auth/me/password Change password with current password
PUT /api/auth/me/force-password Complete required first-login password change
PUT /api/auth/me/avatar Update current user's avatar
PUT /api/auth/me/username Update current user's profile name

Cards

Method Path Notes
GET /api/cards/search Local card search
GET /api/cards/custom List the current user's custom cards and shared templates
POST /api/cards/custom Create an owner-scoped custom card
POST /api/cards/custom/{card_id}/clone Copy a shared template into an independent private card
PUT /api/cards/custom/{card_id} Owner-only custom-card update
DELETE /api/cards/custom/{card_id} Owner-only custom-card delete
GET /api/cards/custom/matches Pending custom-card migration matches
POST /api/cards/custom/migrate/{match_id} Migrate custom card to API card
POST /api/cards/custom/dismiss/{match_id} Dismiss match
GET /api/cards/{card_id}/lang/{lang} Resolve equivalent card in another language
GET /api/cards/{card_id}/price-history Price history
PUT /api/cards/{card_id}/custom-image Set/clear a validated HTTPS fallback image for an API card with no TCGdex artwork
GET /api/cards/{card_id} Card detail
POST /api/cards/recognize Card recognition through the user's configured vision provider
POST /api/cards/recognize/jobs Sanitize and enqueue up to 50 persistent scan photos
GET /api/cards/recognize/jobs Current user's active/actionable scan jobs
GET /api/cards/recognize/jobs/{job_id} User-scoped scan job and review items, resolved items included (collapsed row)
GET /api/cards/recognize/jobs/{job_id}/items/{item_id}/image Private sanitized review photo
GET /api/cards/recognize/jobs/{job_id}/items/{item_id}/candidates/{index}/image A candidate's full-resolution artwork, served from the shared image cache
POST /api/cards/recognize/jobs/{job_id}/items/{item_id}/resolve Confirm/dismiss an item and delete its queued photo
POST /api/cards/recognize/jobs/{job_id}/items/{item_id}/resolve-and-add Atomically add the selected card and resolve the queued item
POST /api/cards/recognize/jobs/{job_id}/items/{item_id}/retry Retry one reviewable item individually
DELETE /api/cards/recognize/jobs/{job_id} Delete a job and its queued photos

GET /api/cards/search accepts q (with name as a compatibility alias), number, set_id, type, category, subtype, rarity, artist, rule_text, hp_min, hp_max, dex_id, lang, sort_by, sort_order, page, and page_size. Text filters are accent-insensitive. rule_text searches card effects plus attack/ability names and effects. Card numbers use the same leading-zero and alphanumeric matching as exact lookup, and q also recognizes code-number pairs such as PFL 001 or 30C 001. Set codes may start with a digit. A code that matches no set falls back to the name search, so card names that look like a code-number pair (151 025) stay reachable.

Custom cards belong to exactly one user. Owners may publish a card as a shared template, but other users must clone it before using it in collections, wishlists, binders, products, or trades. Clones have independent IDs, metadata, images, and prices. Manual image URLs must use public HTTPS destinations and are fetched through the size-limited image proxy. During upgrade, existing custom cards become shared templates owned by the first-created admin account, while each other referencing user receives one private clone and keeps their existing references.

Set display metadata

GET /api/sets/?lang=de&name_lang=en keeps each set's original name, lang, release_date, identity and collection counts. It adds nullable translated_name and translated_name_lang fields. Without name_lang, the user's own app-language setting is used; it is independent of the card-language filter.

Translations come from the locally synchronized catalogue and require exactly one set in the requested name language with the same canonical tcg_set_id. Missing, blank or ambiguous names return null for both fields. No upstream requests or catalogue writes are triggered to obtain a translation. Unsynchronized names and UI-only languages without TCGdex catalogue support (such as Swedish and Danish) therefore have no translated label. Regional sets with different identifiers are not matched by similar names, abbreviations or dates. This is not machine translation or an inferred equivalence between Japanese and international sets.

Clients can format the unchanged ISO release date in their display locale. The Sets page uses month/year in the selected app language with UTC calendar handling, so time-zone conversion cannot move a release into the preceding month.

Collection, Sets, Wishlist, Binders

Method Path Notes
GET /api/collection/ User-scoped collection
GET /api/collection/user/{user_id} View another user's collection (read-only, auth required)
POST /api/collection/ Add to collection
POST /api/collection/bulk-add Bulk-add selected cards; commits each item independently and reports added/updated/failed counts
POST /api/collection/import-csv Strict CSV collection import with all-or-nothing validation
GET /api/collection/printing-detail-tags List reusable printing-detail tags for the current user
POST /api/collection/printing-detail-tags Create or reuse a normalized tag
PUT /api/collection/printing-detail-tags/{tag_id} Rename an owned reusable tag
DELETE /api/collection/printing-detail-tags/{tag_id} Delete an owned reusable tag and its associations
PUT /api/collection/{item_id} Update collection item
DELETE /api/collection/{item_id} Delete collection item
GET /api/collection/{item_id}/photo Read the owner's private photo for that card identity
POST /api/collection/{item_id}/photo Store/replace one sanitized owner photo for that card identity
DELETE /api/collection/{item_id}/photo Delete the owner's photo for that card identity
GET /api/collection/stats/summary Collection summary
GET /api/sets/ List sets; optional lang filters catalogue language, optional name_lang selects translated-name metadata (defaults to the authenticated user's app language)
GET /api/sets/new Newly detected sets
POST /api/sets/mark-seen Mark new-set badges seen
GET /api/sets/{set_id} Set detail
GET /api/sets/{set_id}/checklist Set checklist
GET /api/wishlist/ Wishlist
POST /api/wishlist/ Add wishlist item
PUT /api/wishlist/{item_id} Update wishlist quantity and price alerts
DELETE /api/wishlist/{item_id} Remove wishlist item
GET /api/binders/ Binders
POST /api/binders/ Create binder
PUT /api/binders/{binder_id} Update binder
POST /api/binders/{binder_id}/convert-to-collection Atomically convert a Planned Binder to a physical Binder
POST /api/binders/{binder_id}/convert-to-wishlist Convert a physical Binder to a Planned Binder and release allocations
DELETE /api/binders/{binder_id} Delete binder
GET /api/binders/{binder_id}/cards Binder cards
GET /api/binders/{binder_id}/optimize-prints Equivalent-print optimization preview
POST /api/binders/{binder_id}/optimize-prints Apply equivalent-print optimization
POST /api/binders/{binder_id}/cards Add card to binder
POST /api/binders/{binder_id}/collection-items Add owned collection item to binder
POST /api/binders/add-owned-set Create a Card List populated from an owned set
POST /api/binders/{binder_id}/add-owned-set Add owned cards from a set to an existing Card List
PUT /api/binders/{binder_id}/entries/{binder_card_id} Update binder entry quantity
GET /api/binders/{binder_id}/entries/{binder_card_id}/equivalent-prints List equivalent prints for an entry
PUT /api/binders/{binder_id}/entries/{binder_card_id}/card Switch an entry to an equivalent print
POST /api/binders/{binder_id}/entries/{binder_card_id}/wishlist Add the entry's needed copies to the global Wishlist
POST /api/binders/{binder_id}/wishlist Add all missing planned-list copies to the global Wishlist
GET /api/binders/{binder_id}/export-csv Binder CSV export
POST /api/binders/{binder_id}/import-csv Binder CSV import
DELETE /api/binders/{binder_id}/entries/{binder_card_id} Remove binder entry
DELETE /api/binders/{binder_id}/cards/{card_id} Remove card from binder

Decks

Decks are backed by the shared Binder/Card List tables; these compatibility routes expose Deck-specific validation, allocation, comparison, and probability responses.

Method Path Notes
GET /api/decks/ List the current user's Planned and Real Decks
POST /api/decks/ Create a 20-, 40-, or 60-card Deck
GET /api/decks/compare Compare two Decks and their probability analysis
POST /api/decks/{deck_id}/duplicate Duplicate as a new Planned Deck without physical allocations
POST /api/decks/{deck_id}/convert-to-real Atomically reserve every required owned copy
POST /api/decks/{deck_id}/convert-to-planned Release allocations and retain the planned quantities
GET /api/decks/{deck_id} Deck detail with validation, composition, shortages, and allocations
GET /api/decks/{deck_id}/probability Opening hand/draw/prize probability analysis
PATCH /api/decks/{deck_id} Update Deck metadata or its individual public-sharing opt-in
DELETE /api/decks/{deck_id} Delete Deck and release allocations
POST /api/decks/{deck_id}/entries Add/increment a planned card entry
PATCH /api/decks/{deck_id}/entries/{entry_id} Update required quantity
DELETE /api/decks/{deck_id}/entries/{entry_id} Remove planned entry

Dashboard, Analytics, Social, Community

Method Path Notes
GET /api/dashboard/ Dashboard summary
GET /api/analytics/duplicates Duplicate cards
GET /api/analytics/top-movers Price movers
GET /api/analytics/rarity-stats Rarity distribution
GET /api/analytics/trades-summary Aggregate trade counts, totals, and deltas
GET /api/analytics/investment-tracker Portfolio history
GET /api/analytics/new-sets Analytics new sets
GET /api/social/leaderboard Multi-user leaderboard
GET /api/social/compare/{user_id} Multi-user comparison
GET /api/social/achievements/{user_id} Achievement progress
GET /api/github/contributors Public GitHub contributors feed
GET /api/community/supporters Fresh, strictly validated public supporter registry projection; returns 503 with Cache-Control: no-store on any upstream or validation failure
GET /api/github/rescue-donations Rescue donation total from RESCUE_DONATIONS.csv

Pokédex, Profiles, and Public Sharing

Method Path Notes
GET /api/pokedex Grouped or exact-form completion overview with generation/status/search/form filters
GET /api/pokedex/{entry_id} One species/form with ownership, related forms, and printing summary
GET /api/pokedex/images/{kind}/{entry_id}.png Cached sprites or artwork image, including form artwork
GET /api/profile/ Current user's public-profile, value, and Wishlist-visibility preferences and handle
PUT /api/profile/ Publish/unpublish profile, control values, and select Private, Trade matches, or Public Wishlist visibility
GET /api/public/profiles Anonymous directory of published profiles
GET /api/public/profiles/{handle} Anonymous published profile summary
GET /api/public/profiles/{handle}/binders/{binder_id} Anonymous shared collection Binder
GET /api/public/profiles/{handle}/wishlist Anonymous safe Wishlist projection with search, filters, sorting, and pagination
GET /api/public/profiles/{handle}/decks/{deck_id} Anonymous safe Planned or Real Deck projection
GET /api/public/profiles/{handle}/decks/{deck_id}/probability Anonymous probability analysis for a shared Deck

Products, Export, Backup, Sync, Settings

Method Path Notes
GET /api/products/types Product type suggestions
GET /api/products/ Product list
POST /api/products/ Create product
POST /api/products/batch Create up to 200 product records in one batch
PUT /api/products/lifecycle/bulk Change sealed/opened state for multiple products
PUT /api/products/{product_id} Update product
DELETE /api/products/{product_id} Delete product
GET /api/products/summary Product summary
GET /api/products/{product_id} Product detail
POST /api/products/{product_id}/cards Link collection cards to product
POST /api/products/{product_id}/cards/bulk Link up to 200 collection-card selections atomically
DELETE /api/products/{product_id}/cards/{product_card_id} Unlink product card
POST /api/products/{product_id}/cards/{product_card_id}/sell Record product-card sale
POST /api/products/{product_id}/ledger Add product ledger entry
GET /api/trades/ Current user's trade journal
GET /api/trades/{trade_id} Trade detail with immutable item snapshots
POST /api/trades/value Preview incoming/outgoing card values
POST /api/trades/ Create a trade and apply inventory changes atomically
PUT /api/trades/{trade_id} Safely reverse and reapply an editable trade
GET /api/export/csv CSV export
GET /api/export/pdf PDF export
GET /api/backup/download Admin-only SQL backup
POST /api/backup/restore Admin-only SQL restore
POST /api/backup/clear-image-cache Admin-only image cache clear
POST /api/sync/ Admin-only full sync
POST /api/sync/prices Admin-only small price sync
POST /api/sync/prices/all Admin-only forced price sync for all tracked cards
POST /api/sync/reschedule-full Reschedule full sync
POST /api/sync/reschedule-prices Reschedule price sync
GET /api/sync/status Sync status and history
GET /api/images/card/{card_id}/{size} Card image proxy/cache
GET /api/images/set/{set_id}/{image_type} Set logo/symbol proxy/cache
GET /api/images/product/{product_id} Token-gated, size-limited proxy for a validated product image
GET /api/settings/ Effective settings for current user
GET /api/settings/scanner Typed provider/model readiness for the current user
PUT /api/settings/scanner Save an already-verified scanner configuration or remove a key
POST /api/settings/scanner/test Two-image capability test with optional atomic save
GET /api/settings/tcgdex-languages Supported TCGdex language metadata
GET /api/settings/tcgdex-filter-languages Languages currently available for catalogue filtering
GET /api/settings/currencies Authenticated Frankfurter currency metadata from the installation cache
PUT /api/settings/ Update settings
GET /api/settings/debug-log Admin-only debug log download
DELETE /api/settings/scan-diagnostics Delete all persisted scanner diagnostics for the current user
DELETE /api/settings/card-photos Delete all private collection-card photos for the current user
GET /api/settings/telegram_status Whether Telegram is configured for current user
GET /api/settings/exchange-rate Public read-only EUR/USD-base quote from the installation cache, with provider date and stale status
GET /api/settings/{key} Get one setting
POST /api/settings/{key} Set one setting
GET /api/health Unauthenticated service-health probe

Image caching

The database image cache remains the source of downloaded artwork. Public card artwork and set logos/symbols also use the frontend's bounded Nginx disk cache (up to 1 GiB, discarded with the frontend container). Repeated requests can be served without transferring the image from the backend again.

These routes are mutable, not content-addressed assets. New public responses use public, max-age=0, s-maxage=60, must-revalidate: browsers revalidate their copy, while shared caches may reuse it for at most 60 seconds. Changes to artwork, visibility or fallback settings can therefore take up to that interval to reach a shared-cache hit. Existing browser entries created by older releases retain their original expiry. ETags are SHA-256 hashes of the actual image bytes, so clearing/refilling the database image cache cannot incorrectly validate changed bytes just because the source URL stayed the same. Set image cache keys include the selected source URL, including English fallback URLs.

Manual cards and custom fallback artwork use private, no-store. Missing card redirects and set placeholders use no-store, so a temporary failure does not hide artwork that becomes available later. Fingerprints, token-gated product images and private collection photos never use the shared proxy image cache. Requests carrying authorization, cookies, explicit cache controls or a retry parameter bypass it too. Expired entries are not served when revalidation fails. The existing product-token and private-photo authorization rules are unchanged.

For a disposable integration check, run python scripts/test-image-proxy-cache.py --bind 0.0.0.0 --proxy-url http://frontend on an isolated network where its synthetic upstream is reachable as backend:8000 and the frontend uses the candidate frontend/nginx.conf. Publish no host ports and mount no production data. The check counts upstream requests, verifies conditional responses, private-route exclusions, expiry, fallback recovery and security headers. It does not measure production traffic.

Currency data and offline conversion

Frankfurter currency metadata and rates are stored in global, per-installation currency_metadata and currency_rates tables. Rate rows are keyed by base and quote currency and include the provider's observation date and the time they were synced. The backend refreshes a stale base-rate set when a conversion is requested; metadata is refreshed daily. Each response marks a cached rate/list as stale when its refresh window has passed and the provider could not be reached.

The rate feed can also contain assets with longer codes, such as AAVE or USDT. The backend skips quotes outside its three-letter currency format so those records do not block valid currency conversions. A response with no supported quotes is rejected; invalid dates, rates, duplicate supported quotes, and inconsistent bases still fail validation. This does not expand the currency codes accepted by the API.

The frontend requests both EUR-based and USD-based rate sets for the selected currency. Collection, wishlist, analytics, public binder/profile values, exports, and Telegram use the selected currency. If a conversion has never been synced, the API returns unavailable rather than relabeling an EUR or USD amount as the selected currency. A new offline installation therefore needs one successful sync before it can convert to currencies other than its native price currencies. The database cache is local to each self-hosted installation; it is not shared between installations.

The exchange-rate endpoint is public read-only so anonymous public-profile views can use the same cache; it accepts only EUR or USD bases, which are the price sources used by the app. The currency metadata endpoint remains authenticated. Neither endpoint exposes user settings or collection data.

Models

Card

  • Composite primary key: {tcg_card_id}_{lang}, for example sv1-1_de
  • tcg_card_id stores the original TCGdex card id
  • set_id stores the original TCGdex set id, not the composite set row id
  • rarity is read-only API data
  • Variant availability is represented by boolean flags:
    • variants_normal
    • variants_reverse
    • variants_holo
    • variants_first_edition

CollectionItem

  • Stores user-owned copies of cards
  • Active fields: card_id, user_id, quantity, condition, variant, purchase_price, lang
  • Variant values are now the physical print variants only: Normal, Holo, Reverse Holo, First Edition
  • The old grading UI is gone; the database migration history still contains a legacy grade column, but it is not part of the current ORM model or API schema
  • Existing rows are grouped by user, card, variant, language, condition, purchase price, and normalized printing-detail tag set when cards are added through the API
  • printing_detail_tags adds reusable owner-scoped descriptors without expanding the fixed physical-variant enum

CollectionCardPhoto and PrintingDetailTag

  • CollectionCardPhoto stores one private image per user_id + card_id; all grouped collection rows for that card identity share it
  • Photo bytes are returned only through authenticated collection endpoints and never written to the globally visible catalogue-card image cache
  • PrintingDetailTag is unique per user after Unicode normalization and a fixed digest key
  • The same tag may be associated with collection items, product cards, product ledger entries, and trade items

Binder / BinderCard

  • binder_type is collection, wishlist, deck, or physical_deck
  • Planned rows store card_id + required_quantity; physical rows add an exact collection_item_id
  • target_size, when present, is 20, 40, or 60
  • is_public applies to collection Binders and Decks exposed through public profiles
  • Allocation services enforce one shared owned-copy capacity across physical Binders and Real Decks

Public sharing boundary

  • User.wishlist_visibility is constrained to private, trade_matches, or public; migrations normalize every existing row to private
  • anonymous serializers query catalogue cards directly and exclude Wishlist alert state, notification history, collection ownership, physical allocation, purchase, condition, private-photo, and internal-row data
  • private custom cards are excluded from public Wishlists and block Deck sharing
  • every successful anonymous response requires the global feature, a live public owner profile, and the relevant Wishlist or Card List opt-in; responses require cache revalidation so revocation takes effect immediately

Products and trades

  • ProductPurchase tracks batch, image/Cardmarket links, notes, and lifecycle state (sealed, opened, sold, or review)
  • ProductCard links opened-product contents to source collection rows while retaining initial, active, and sold quantities
  • ProductLedgerEntry preserves realized sales, gains, adjustments, and trade outflow history even if an active collection row is later removed
  • Trade records partner/date/notes, cash-adjusted incoming/outgoing values, and value delta
  • TradeItem stores direction and a card/condition/variant/language/printing snapshot plus provenance used to safely edit newer trades

User

  • Fields include role, avatar_id, and must_change_password
  • must_change_password is returned by auth responses and enforced by the frontend after login

Setting

  • Global key/value table
  • Used for admin-only settings such as sync cadence and auth mode

UserSetting

  • Per-user key/value table
  • Used for isolated user preferences and secrets
  • Unique constraint: user_id + key

Other Core Models

  • Set
  • WishlistItem
  • PriceHistory
  • PortfolioSnapshot
  • SyncLog
  • ImageCache
  • CustomCardMatch
  • ScanJob / ScanJobItem / ScanQueueUserState
  • GeminiQuotaState / ScannerProviderLimitState

Settings Scope

Current settings are split in backend/api/settings.py:

  • PER_USER_KEYS
    • language
    • currency
    • price_primary
    • price_display
    • telegram_bot_token
    • telegram_chat_id
    • telegram_enabled
    • price_alerts_enabled
    • price_alert_threshold
    • gemini_api_key
    • openai_api_key
    • provider-specific scanner provider/model settings managed by the dedicated scanner endpoint
    • scan_diagnostics_enabled
    • prefer_own_card_photos
    • set_overview_filters
    • hidden_set_ids
    • portfolio_display_mode
    • provider-specific request timeouts, custom-model selections, capability proofs, and optional Gemini fallback preference
    • trainer_name
  • ADMIN_ONLY_KEYS
    • full_sync_interval_days
    • price_sync_interval_minutes
    • multi_user_mode
    • tcgdex_sync_languages
    • debug_mode
    • cross_language_price_fallback
    • cross_language_image_fallback
    • tcgdex_digital_sets_enabled
    • public_profiles_enabled

Important behavior:

  • Each user only reads and writes their own UserSetting rows
  • Admin-only settings are stored globally in settings
  • Recurring automatic syncs include a full sync cadence and a separate small price sync cadence
  • tcgdex_sync_languages is seeded from TCGDEX_SYNC_LANGUAGES only when the row does not exist yet; afterward the DB value is authoritative. Empty or invalid env values safely fall back to en,de. The env value all expands to every supported TCGdex language during first bootstrap.
  • Supported TCGdex sync language codes are centralized in services/tcgdex_languages.py. Optional extra languages are fr, es, es-mx, it, pt, pt-br, pt-pt, nl, pl, ru, ja, ko, zh-tw, id, th, and zh-cn in addition to the default en,de.
  • English is the preferred cross-language fallback source for missing data, images, and prices by exact TCGdex ID. The backend does not guess English replacements by card name for regional-only cards.
  • Admin users can receive initial fallback values from env vars for Telegram and Gemini
  • recognize.py intentionally reads Gemini only from the current user's UserSetting; there is no cross-user fallback
  • scan_diagnostics_enabled is off by default and is effective only when the server configures SCAN_TRACE_DIR

Sync & Backup Behavior

Sync

  • /api/sync/, /api/sync/prices, and /api/sync/prices/all enforce admin access
  • /api/sync/ runs the full TCGdex set/card sync using the configured tcgdex_sync_languages
  • /api/sync/prices runs the small tracked-card price sync
  • /api/sync/prices/all force-refreshes prices for all tracked cards
  • Sync status returns current flags plus the last 10 sync log rows
  • Full sync and price sync can be rescheduled through dedicated endpoints

Selective Backup

GET /api/backup/download accepts include as a comma-separated query param.

Supported groups:

  • full
  • collection
  • users
  • cards
  • products
  • system
  • images

Current table mapping:

  • collection: collection, wishlist, binders, binder_cards, printing_detail_tags, collection_printing_detail_tags
  • users: users, user_settings, settings, printing_detail_tags
  • cards: cards, sets, price_history, custom_card_matches
  • products: product_purchases, product_cards, product_ledger_entries, portfolio_snapshots, printing_detail_tags, product_card_printing_detail_tags, product_ledger_printing_detail_tags
  • system: sync_log
  • images: image_cache

If include=full, image cache is excluded unless images is also explicitly included.

Use full for disaster recovery. Selective mappings are targeted export groups, not guaranteed standalone restore sets: they may depend on rows in another group and currently do not include every independent newer table, including trades, trade-item tag associations, private collection-card photos, and scan queue/provider state. A full dump includes those database tables. Scanner trace and queued-photo files remain filesystem data and are not part of SQL backups.

Manual restore

POST /api/backup/restore accepts only a non-empty .sql upload, streams it to a temporary file, and invokes psql with ON_ERROR_STOP=1, --no-psqlrc, and --single-transaction. Any SQL error rolls back the whole restore. The temporary file is removed whether the operation succeeds or fails.

Automatic Pre-upgrade Backup

The backend image installs PostgreSQL 18 client tools so pg_dump can back up the default PostgreSQL 18 service and newer external PostgreSQL 18 servers. PostgreSQL requires pg_dump to be at least as new as the server major version.

backend/services/pre_upgrade_backup.py runs before init_db() startup migrations.

Behavior:

  • Reads the current app version from VERSION through backend/main.py.
  • Reads settings.last_successful_app_version from the existing database.
  • Skips fresh installs where the settings table does not exist yet.
  • Creates a full SQL dump in /app/backups when an existing install starts on a new version.
  • Uses filenames like pre_upgrade_1.17.0_to_1.18.0_20260526_010500.sql.
  • Records last_successful_app_version only after startup initialization succeeds.
  • Retains the newest PRE_UPGRADE_BACKUP_KEEP automatic backups, default 10, minimum 1.
  • Writes dumps to a temporary filename first, then atomically renames after a successful non-empty pg_dump so partial files are not treated as valid backups.

Environment controls:

  • PRE_UPGRADE_BACKUP_ENABLED, default true
  • PRE_UPGRADE_BACKUP_REQUIRED, default true; when true, startup fails before migrations if pg_dump fails
  • PRE_UPGRADE_BACKUP_KEEP, default 10, minimum 1

Scanner Notes

backend/api/recognize.py, backend/api/scan_jobs.py, and backend/services/scan_queue.py implement the persistent background queue used by the unified scanner. The direct single-card recognition endpoint remains available for API compatibility:

  1. Uploads are bounded, sanitized, orientation-normalized JPEGs with metadata removed.
  2. Two-to-four batch-eligible photos share one indexed composite provider request. Any missing or uncertain position is retried from its original individual photo.
  3. The selected provider extracts name, split local/total collector number, printed set code, regulation mark, type, HP, language, and artist; uncertain small text stays null.
  4. Candidates are found by querying the locally synced cards table and ranked deterministically by local number, language, printed total, set code, regulation mark, artist, and HP. Missing evidence is neutral and contradictions are negative. Broad substring rows from either the local catalogue or live TCGdex are retained only when their complete names match after accent, case, and whitespace normalization, so an unrelated containing name or different card suffix cannot become a confident number match. A (language, name) search pair falls back to one live TCGdex call (_api_search_fallback in backend/api/recognize.py) when it has no name-compatible local rows, or when a collector number was recognized but none of those rows has that number. The second condition matters when a newly released printing reuses an existing card name before the local sync reaches it. The fallback result is used for that one scan only and is never persisted to cards. When every required live fallback is unavailable and no local candidate exists, the scan reports a transient catalogue outage instead of a false "no matches" result. Queued scans save the parsed vision result before matching and reuse it for catalogue-outage retries, so recovery does not repeat the paid extraction. Scan traces tag each search-pair result's source as local or api_fallback, and cached retries mark the extraction source as queue_cache.
  5. If metadata is inconclusive, conservative pHash can accept a close, clearly separated visual winner without another provider call. It never overrides known contradictions.
  6. Individual scans may use the same provider's visual comparison when pHash abstains; composite scans fall back to individual recognition instead.
  7. Queue results remain reviewable after restarts. Confirming and adding a candidate uses one row-locked database transaction so concurrent tabs cannot increment the collection twice. The frontend retains the source Blob before that request; when no catalogue/custom artwork exists, it uploads the Blob afterward as a best-effort authenticated collection-card photo. Failure does not undo the collection add. Confirming/dismissing deletes the queued photo. Unreviewed jobs expire after 14 days.

Provider error handling:

  • Gemini and OpenAI-compatible providers share the queue contract while retaining provider-specific request formats
  • Permanent authentication, billing, model, and request errors stop immediately with fixed actionable messages; arbitrary upstream prose is not returned, logged, or persisted
  • Transient provider failures are retried with bounded backoff
  • Machine-readable daily-quota 429 responses are separated from short-term limits
  • Provider Retry-After or google.rpc.RetryInfo delays are used exactly when supplied; missing daily delays fall back to one hour and later six-hour intervals
  • Quota state is shared by an API-key fingerprint, so concurrent requests using the same key observe one block while different keys stay independent
  • Quota retries do not consume the three recognition attempts
  • Invalid API keys get a dedicated user-facing message
  • Provider defaults and approved models are administrator-controlled; normal users select only enabled choices, while administrators may test an Advanced custom model
  • Retired or unavailable models return a fixed model-unavailable message without reflecting provider text
  • Temporary Gemini outages are returned clearly instead of leaking as generic backend 500 errors
  • Gemini requests send the API key via header instead of the request URL

Additional matching behavior:

  • Name suffixes like EX, GX, V, VMAX, VSTAR, TAG TEAM, BREAK, and LV.X are stripped before search
  • Search may fall back from detected card language to English
  • Result payload includes recognized metadata and candidate matches, each flagged with printed_total_mismatch when its printed set total contradicts the recognized card (the same signal the ranker already uses to demote it)
  • backend/services/scan_candidate_images.py caches each unresolved review's full-resolution candidate artwork in the shared ImageCache table (see backend/api/images.py); match_card_info fires a bounded, non-blocking prewarm of the top-ranked candidates so the review UI's first look is usually a local cache read. Fetches accept only image responses from the TCGdex HTTPS CDN, and this feature's cache entries are capped.

Scanner diagnostics

backend/services/scan_trace.py is disabled unless SCAN_TRACE_DIR points to storage the backend can create and write. Availability alone does not collect data: each user must opt in with scan_diagnostics_enabled=true, which is off by default. SCAN_TRACE_STORAGE_DIR is the stable cleanup location; standard Docker Compose keeps it at /app/data/scan-traces even when new collection is disabled.

For opted-in attempts, one user-scoped JSON trace and sanitized JPEG are stored. Traces contain the selected provider and model, generic prompt, redacted provider response, parsed fields and usage, TCGdex searches, ranked candidates and rank keys, pHash distances, visual-verification response, final mechanism, and errors. Configured credentials are redacted before persistence and are never returned by the settings API.

When a queued candidate is confirmed, its TCGdex card id labels all stored attempts for that job item as ground truth. backend/scripts/analyse_scan_traces.py reports top-1 accuracy, retrieval/ranking misses, decision-mechanism performance, pHash outcomes, and optional field-null/failure details.

Turning consent off stops future capture and leaves existing traces unchanged. There is no automatic retention limit. DELETE /api/settings/scan-diagnostics is the explicit per-user deletion action; deleting an account revokes in-flight writes and removes its trace subtree as well. Trace directories use mode 0700 and JSON/JPEG files use 0600. Diagnostics are not included in SQL backups because they are filesystem analysis data.

Bulk removal

Authenticated owners can remove complete entries in one atomic request:

  • POST /api/collection/bulk-delete
  • POST /api/binders/{binder_id}/entries/bulk-delete
  • POST /api/decks/{deck_id}/entries/bulk-delete

The JSON body is {"item_ids": [12, 34]}. IDs are collection-entry IDs for the collection route and list-entry IDs for binder/deck routes, never catalogue card IDs. Deck requests accept plan-entry IDs, not internal allocation-row IDs. The list must contain 1–500 unique, strictly positive integer IDs; malformed, duplicate or out-of-range inputs return 422.

Success returns {"removed_ids": [12, 34], "removed_entries": 2, "removed_copies": 5}. Missing, foreign or wrong-list entries return 404 for the whole request. Collection entries with physical-list allocations or active product links return 409. All validation, deletes and cleanup share one commit; an error rolls everything back.

Collection removal uses the same protections as single-entry removal. Other condition/variant entries for the same card remain, and an owner/card photo is removed only when the owner's last matching entry disappears. Catalogue data, other users and sold-product history remain intact. Binder/deck removal does not delete owned collection entries or photos. Real-deck removal releases only the selected plans' allocations; planned lists do not change owned quantities. Each route retains the established per-user allocation lock and owner-scoped row locks. Releasing a list and later removing collection entries are two separately confirmed transactions, not one cross-view operation.

Tests: python -m unittest discover -s tests -p test_bulk_removal.py from backend/. Set BULK_REMOVAL_POSTGRES_URL to an approved test database and run test_bulk_removal_postgres.py to exercise the same contract plus a real lock race. That suite creates and removes only a randomly named test schema.

Bulk Collection Add

POST /api/collection/bulk-add accepts BulkCollectionAddRequest with multiple CollectionItemCreate items and returns BulkCollectionAddResponse:

  • added: new collection rows created
  • updated: existing matching rows whose quantity was incremented
  • failed: items that could not be added
  • errors: per-card error details

Each item is committed independently, so one invalid or unavailable card does not roll back the rest of the batch. Existing rows are matched by card, variant, language, and current user.

Notifications

backend/services/telegram.py reads Telegram credentials and currency from that user's UserSetting rows. Price alerts convert both the EUR market price and threshold to the selected currency using the shared rate cache; if no rate has ever been synced, both values are marked unavailable in the selected currency.

Migrations

  • Migrations are raw SQL statements in backend/database.py
  • They are idempotent and run on startup
  • Automatic pre-upgrade backups run before init_db() migrations on existing installs when the app version changes
  • Legacy migration comments still mention older columns like grade or removed integrations, but the current runtime model and routers do not include eBay functionality

POST /collection/release-assignments is a separate owner-scoped action. Send item_ids plus the exact assignments snapshot from the collection 409 response (assignment_id, list_id, item_id, quantity). A changed snapshot returns 409 without mutation. The release is atomic, preserves collection ownership and product links, and reduces Real Deck plans only by the released copies. The collection 409 response lists all affected cards, binder/deck assignments and active product sources.

Bulk-removal API workflow

Every inventory-changing action in this workflow is callable independently of the UI. Use the installation's normal authentication (for example an Authorization: Bearer <token> header). IDs below are database entry IDs, not TCGdex card IDs.

  • POST /api/collection/bulk-delete: remove whole owned entries, with { "item_ids": [123, 124] }.
  • POST /api/binders/{binder_id}/entries/bulk-delete: remove entries from that binder only, with the same payload shape and binder-entry IDs.
  • POST /api/decks/{deck_id}/entries/bulk-delete: remove entries from that deck only, with the same payload shape and deck-entry IDs.
  • POST /api/collection/release-assignments: release explicitly confirmed binder/real-deck assignments without deleting ownership. Send the selected item_ids and the exact assignments snapshot returned by a collection-removal 409. Each snapshot row contains assignment_id, list_id, item_id, and quantity. Changed assignments return 409 without mutation.
  • DELETE /api/products/{product_id}/cards/{product_card_id}: unlink one product source without deleting the product or collection inventory and without recording a sale. Sale/trade history prevents unlinking.

For product unlinking from a displayed confirmation, pass both expected_collection_item_id and expected_active_quantity as query parameters. They are optional for backward compatibility, but pairing them protects against unlinking a link that changed after it was displayed. A mismatched entry/quantity returns 409; an incomplete pair or invalid value returns 422; missing/foreign links return 404. A successful response is the updated product, not a collection deletion.

Example product unlink (substitute the IDs and quantity from the 409 response's conflicts[].products[]):

curl --fail-with-body --request DELETE \
  --header "Authorization: Bearer $TOKEN" \
  "$BASE_URL/api/products/7/cards/42?expected_collection_item_id=123&expected_active_quantity=2"

Example explicit assignment release:

curl --fail-with-body --request POST \
  --header "Authorization: Bearer $TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"item_ids":[123],"assignments":[{"assignment_id":91,"list_id":8,"item_id":123,"quantity":2}]}' \
  "$BASE_URL/api/collection/release-assignments"

Neither action automatically retries collection deletion. After resolving every blocker, make a separate POST /api/collection/bulk-delete request. Each request is atomic; the complete multi-step workflow is not one transaction. The endpoints and optional unlink preconditions are also described by the backend's OpenAPI schema.

Atomic failures and follow-up requests

Bulk collection/list removal, assignment release and product unlinking remain all-or-nothing per request. Business-rule failures return a structured detail object with:

  • code and message: machine-readable category and readable explanation.
  • changed: false: the failed request committed no inventory changes.
  • blockers: all blockers found for that request, rather than only the first blocked entry. Each includes a reason code, resource, relevant submitted IDs and an action such as release_assignments, unlink_products, refresh_confirmation, remove_from_request, or cannot_unlink_history.
  • conflicts: collection card names/printing details, assigned lists and exact copy counts, and product sources. Existing collection-conflict fields remain compatible.
  • resolutions, where applicable: proposed API method, path and exact snapshot body for explicitly confirmed follow-up calls. These are suggestions, never actions automatically executed by the server. Product resolutions mark can_execute: false and list blocked_by_history when sale/trade provenance prevents unlinking.

A mixed request containing unavailable IDs and owned blocked entries reports both categories. Missing, wrong-list and foreign-owned IDs use the same not_found_or_not_accessible reason and disclose only IDs already supplied by the caller, not another owner's card names, product names or list contents. Such requests return 404; relationship/stale-confirmation conflicts return 409. Invalid request bodies still use standard 422 validation errors. Authentication/authorization failures and unexpected server errors are not card-by-card business-rule results.

A client can inspect a collection failure, explicitly approve and submit the returned assignment-release and product-unlink resolution bodies, then separately retry the original collection deletion. Removing invalid IDs changes the requested batch and must be an explicit client decision, not an automatic best-effort retry. Each follow-up has its own transaction and may itself reject a stale snapshot.

Bulk product unlink

POST /api/products/cards/bulk-unlink accepts 1–5000 unique product-link snapshots:

{
  "links": [
    {"product_id": 7, "product_card_id": 42, "collection_item_id": 123, "active_quantity": 2},
    {"product_id": 8, "product_card_id": 43, "collection_item_id": 124, "active_quantity": 1}
  ]
}

Every field is a strict positive integer; active_quantity is at most 999. Duplicate product_card_id values are rejected. A success returns unlinked_ids, unlinked_links and unlinked_copies. Purchases, owned quantities, private photos and catalogue cards are preserved, and no sale/trade entry is created.

If any link is unavailable, changed or protected by sale/trade history, no links are removed. The structured failure reports every discovered not_found_or_not_accessible, product_link_changed and product_history_protected blocker. current_links includes owner-safe current snapshots and names so clients can refresh their confirmation; multiple reasons can apply to the same link. Missing/foreign links are not included in current_links.

The popup exposes this API as Unlink listed products, with a separate confirmation for the full displayed link/copy totals. Individual unlink remains available and backward compatible. Collections remain limited to 500 entries and assignment release to 5000 assignments per request. If a resolution exceeds these documented bounds, explicitly choose a smaller request; do not assume multiple requests form one atomic transaction.