[Feature]: Cache domain reputation lookups to remove repeated WHOIS/DNSBL/threat-intel calls on the /predict hot path

[pavsoss]

Problem Statement

backend/domain_checker.py::analyze_domain runs, per domain and per /predict
call, a WHOIS lookup + 4 sequential DNSBL DNS queries (each up to a 5s timeout) +
3 threat-intel HTTP calls for up to 5 domains. There is no caching anywhere
in the module. Consequences:

• Duplicate/repeated domains (extremely common in spam corpora and bulk scans)
  re-incur multi-second latency every single time.
• The 4 sequential DNSBL lookups alone can block ~20s worst case on the request
  path.
• Uncached duplicate queries hammer Spamhaus/SpamCop/URLhaus, risking
  provider-side rate-limiting or bans.
• extract_domains over-matches plain text (e.g. file.txt, e.g), so WHOIS
  even runs on non-domains.

This is distinct from the already-merged concurrency work and the open
expensive-endpoint rate-limiting issue: this is about not repeating the work,
not parallelizing or throttling it.

Proposed Solution

Add a TTL cache keyed by domain with separate positive/negative TTLs (bad
verdicts cached longer than transient failures), per-domain locking to collapse
concurrent duplicate lookups (thundering-herd protection), env-configurable
TTLs/size limits, and cache hit/miss counters surfaced for the analytics
dashboard. Tighten extract_domains with real hostname/TLD validation so false
positives never reach WHOIS.

Location: backend/domain_checker.py (+ tests/test_domain_cache.py).

Alternatives Considered

• Per-request memoization only (helps within one message, not across requests).
• A distributed cache (Redis) - heavier dependency than needed; an in-process TTL
  cache with locking covers the single-worker Flask deployment and can be swapped
  later.

Additional Information

Test coverage: hit/miss, TTL expiry, negative caching, dedup-under-concurrency,
and extractor precision.

[github-actions]

👋 Welcome back! Thanks for your continued contribution. Your issue/PR is received and waiting for maintainer review.

[github-actions]

The issue highlights inefficiencies in the backend/domain_checker.py::analyze_domain function, which performs WHOIS lookups, DNSBL queries, and threat-intel calls without caching, leading to repeated redundant operations, increased request latency, and risk of rate-limiting from third-party services when processing duplicate domain queries. The proposed solution is to implement a TTL-based cache with domain-level locking to avoid duplicate lookups, configurable settings for caching, and enhanced domain extraction validation to reduce false positives, thereby improving performance and scalability.

[Userunknown84]

You have a good knowledge of backend.