pSEO Engine

Open-source toolkit for building review-gated programmatic SEO pipelines.

pSEO Engine turns structured niche definitions, keyword research, templates, and LLM prompts into draftable article/page outputs. It is designed for operators who want source-backed content workflows with human review before publishing.

What is included

• YAML-driven site/niche configuration
• Keyword matrix expansion and opportunity scoring, with optional Moz authority-aware competitor difficulty
• Source collectors for SERP, People Also Ask, related searches, Reddit, forums, Discourse, Stack Exchange, GitHub Discussions, Hacker News, RSS/Atom feeds, sitemaps, YouTube/TikTok/Instagram, browser pages, and competitor coverage
• Safe source-collection guidelines for rate limits, blocked/challenge states, and public-data boundaries
• Guardrails and schema validation for generated drafts
• End-of-writing avoid-AI-writing cleanup plus optional Thunderdome multi-model fact-checking
• Markdown, Shopify, and Webflow publishing adapters
• Optional AI-information / entity source-of-truth artifact (AI-information page, llms.txt, AboutPage schema) for GEO/citation readiness
• Review, normalization, rendering, schema.org, and image-planning utilities
• Example configs and reusable content templates
• A library of content page types (schemas + renderers), selected per keyword cluster via its template field: reference-entry, comparison-guide, product-how-to, faq-page, culture-editorial, gift-guide, category-editorial, brand-location, location-category, calculator-landing (interactive calculator landing page), and reddit-consensus

What is intentionally not included

This public branch excludes private/client-specific data, generated article outputs, live site frontends, local handoff notes, credentials, analytics exports, and campaign assets.

If you are preparing a public repo from a private working tree, publish from a fresh sanitized repository or rewritten history. Removing files in a branch is not enough if the private history will become public.

Install

git clone https://github.com/m0ntydad0n/pseo-engine-oss.git
cd pseo-engine-oss
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Quote the extras ('.[dev]') so zsh does not treat the brackets as a glob. Optional feature extras: '.[llm]' (LLM draft generation via LiteLLM), '.[mcp]' (local MCP server), '.[images]' (image generation), '.[browser]' (Playwright/Camoufox collection), '.[analytics]' (GA4/Search Console).

Python: package metadata requires Python >=3.10. The smoke test (scripts/e2e_smoke_test.sh) defaults to python3.11; override it with PYTHON_BIN, for example PYTHON_BIN=python3 scripts/e2e_smoke_test.sh. Reported working on Python 3.14.

Image generation is optional and opt-in via pseo-generate --with-images. Without the '.[images]' extra (or the matching provider keys), it logs that it is skipping inline images and produces the draft without them.

Alternative without editable install:

pip install -r pipeline/requirements.txt
export PYTHONPATH="$PWD/pipeline"

Configure secrets

Copy the example environment file and fill in only the providers you intend to use:

cp .env.example .env

There are no mandatory credentials for local config validation, taxonomy work, scoring, rendering, or running the test suite. Most users should start without keys, then enable only the integrations they need.

Provider-specific variables are grouped in .env.example:

• LLM generation (via LiteLLM, key chosen by the generation.model prefix): ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, DEEPSEEK_API_KEY, XAI_API_KEY, OPENROUTER_API_KEY (one key, hundreds of models), … plus REPLICATE_API_TOKEN for images
• Keyword/SERP data: DATAFORSEO_LOGIN, DATAFORSEO_PASSWORD (default); or SERPAPI_API_KEY (SERP+PAA), KEYWORDS_EVERYWHERE_API_KEY (volume); plus MOZ_ACCESS_ID/MOZ_SECRET_KEY or MOZ_API_TOKEN (optional, authority-aware competitor difficulty)
• Analytics: GOOGLE_SERVICE_ACCOUNT_PATH, GA4_PROPERTY_ID, GSC_SITE_URL
• Social/listening sources: REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET, REDDIT_USER_AGENT
• Source collectors: GITHUB_TOKEN (GitHub Discussions/Issues collector)
• Direct publishing: SHOPIFY_STORE_URL, SHOPIFY_ACCESS_TOKEN, SHOPIFY_BLOG_HANDLE; or Webflow WEBFLOW_API_TOKEN, WEBFLOW_COLLECTION_ID, WEBFLOW_SITE_ID

Markdown publishing and local dry-run workflows do not require publishing credentials.

Never commit .env, API keys, exported analytics, databases, generated drafts, or private client data.

Quick start

Run the interactive wizard to scaffold a new site config:

python -m pseo.init
# or, after editable install:
pseo-init

The wizard walks through site identity, keyword matrices, brand voice, operator expertise, compliance, sources, and publishing platform. It writes a YAML file to pipeline/configs/<slug>.yaml, updates a tailored .env.example with only the API keys you need for the collectors you selected, and runs validation at the end.

The keyword matrices step defaults to [d]escribe: type a sentence or two about the site, audience, and goals, and the engine drafts validated keyword matrices, then shows a preview of the real resulting keywords (count + samples) so you can accept, regenerate, or drop to manual. [m]anual (enter axes by hand) and [p]aste YAML remain available. The same drafting is available standalone:

pseo-matrices --brief-file brief.txt --out matrices.yaml   # writes a config-shaped matrices: block

Both need the '.[llm]' extra and a provider key in the environment (e.g. GEMINI_API_KEY); the default model is gemini/gemini-2.5-flash.

Validate an existing config:

python -m pseo.config --validate pipeline/configs/<slug>.yaml
# or:
pseo-config --validate pipeline/configs/<slug>.yaml

Compare two configs:

python -m pseo.config --diff pipeline/configs/site-a.yaml pipeline/configs/site-b.yaml

Inspect a config as JSON (secrets redacted):

python -m pseo.config --config pipeline/configs/example-saas.yaml

Run tests:

PYTHONPATH=pipeline pytest -q pipeline/tests

Run the local end-to-end smoke test from a clean temporary clone:

scripts/e2e_smoke_test.sh

The smoke test installs the package with optional extras, runs the full test suite, validates example configs, exercises the init wizard, runs generation in dry-run mode, approves a fixture draft, and runs publish in dry-run mode. It does not require API keys and does not publish anything. Set KEEP_E2E_TMP=1 if you want to inspect the temporary clone and logs afterward.

Typical workflow — the canonical pipeline is discover -> cluster -> score -> generate:

1. Create/validate a YAML config in pipeline/configs/ (or outside the repo).
2. Discover raw keywords/research for the configured niche.
3. Cluster raw keywords into canonical opportunities.
4. Score the clusters.
5. Generate dry-run prompt previews or pending-review drafts.
6. Review drafts manually.
7. Publish/schedule through the Markdown or Shopify adapter only after approval.

pseo-discover --config pipeline/configs/example-saas.yaml
pseo-cluster  --config pipeline/configs/example-saas.yaml --input data/<slug>/keywords/raw/<date>.json
pseo-score    --config pipeline/configs/example-saas.yaml --input data/<slug>/keywords/clustered/<date>.json
pseo-generate --config pipeline/configs/example-saas.yaml --input data/<slug>/keywords/scored/<date>.json --dry-run --limit 1

No API keys are required for config validation, tests, matrix discovery, clustering, scoring, and dry-run prompt previews. Real LLM draft generation, the writing-quality provider passes, and image generation need provider keys/extras. Without DataForSEO, scores are keyword-only relative estimates — useful for ordering a local test queue, not a substitute for real search volume/competition.

Writing quality controls:

generation:
  model: anthropic/claude-sonnet-4-5-20250929  # provider-prefixed (LiteLLM)
  avoid_ai_writing: true  # default; runs after the model draft is produced

Generation runs through LiteLLM, so the generation.model string is provider-prefixed and any of ~100 providers works by setting that provider's key: anthropic/…, openai/…, gemini/…, groq/…, deepseek/…, mistral/…, xai/…, or openrouter/<author>/<slug> (one OPENROUTER_API_KEY for hundreds of models). Install the backend with the '.[llm]' extra. thunderdome.fact_checker_models accept the same prefixed strings, so you can cross-check across providers for free. Migration: bare model strings (claude-…, gemini-…) must be rewritten in prefixed form (anthropic/claude-…, gemini/gemini-…).

The avoid-AI-writing pass is a clarity/style-quality pass: it removes generic filler and cliché phrasing, tightens wording, and preserves factual meaning. It is not an AI-detection bypass and does not inject typos or simulate human mistakes.

Browser-based collection:

sources:
  enabled_collectors:
    - reddit_expanded
    - competitor_scrape
    - browser_page
  browser:
    enabled: true
    engine: playwright
    fallback_engine: camoufox
    mode: authorized_collection
    # Challenge resolution only runs for hosts listed here (empty = never).
    authorized_domains: []
    persistent_context: true
    profile_dir: .pseo/browser-profile
    min_delay_seconds: 3
    max_pages_per_domain_per_day: 50
    challenge_resolution_enabled: false
    challenge_resolution_engine: camoufox
    clearance_timeout_seconds: 300
    natural_pass_seconds: 6
    max_challenge_clicks: 20
    stop_on:
      - blocked
      - rate_limited
      - login_wall
      - paywall
      - challenge_unresolved

Browser-based collection is for authorized research workflows where APIs, feeds, sitemaps, or static HTTP fetches are insufficient. Playwright is the primary engine and Camoufox is the secondary/fallback engine. Browser collection is explicit and auditable: it must be enabled in config, uses named engines, supports persistent user-controlled sessions, records blocked/rate-limited/login-wall/paywall/challenge states, and exposes per-domain pacing and collection limits (min_delay_seconds, max_pages_per_domain_per_day) that you set as policy for your targets. Challenge resolution is gated twice: it runs only when challenge_resolution_enabled: true and the target host is listed in authorized_domains (empty by default, so it never runs until you name the domains you are authorized to collect from). When both are set, Camoufox attempts a bounded persistent-profile Cloudflare/Turnstile clearance flow — reuse cached cf_clearance, wait for a natural pass, then click the Turnstile wrapper with bounded jitter before retrying from the same browser tab — capped by max_challenge_clicks and clearance_timeout_seconds. You are responsible for ensuring you are authorized to access and collect from any target.

thunderdome:
  enabled: false
  fact_checker_models:
    - claude-sonnet-4-5-20250929
    - gemini-2.5-pro
  max_rounds: 3

When Thunderdome is enabled, pseo-generate sends the final draft to one or two fact-checker models of your choice. If any checker disagrees, the primary generation model receives the issue list, revises the draft JSON, and the check repeats until every checker agrees or max_rounds is reached. Drafts that still lack agreement are marked blocked for human review. Anthropic models use ANTHROPIC_API_KEY; Gemini/OpenAI checkers use the optional image/provider dependencies and their matching GOOGLE_GENAI_API_KEY or OPENAI_API_KEY.

Post-publishing monitoring:

pseo-monitor --config pipeline/configs/example.yaml --post-publish --dry-run

The post-publish report is designed to run daily after you pull GSC performance data. It checks engagement, indexing state, and crawl-budget quality:

• If a page is live long enough to be discovered but GSC/inspection data suggests it is not indexed, the report notifies the owner to submit it manually in Google Search Console.
• If an indexed page has zero impressions after the configured 30-90 day floor, the first recommendation is modify_for_traction: improve the title, intro, internal links, source depth, and search-intent fit.
• If that page was already modified and still has no impressions after the next 30-day grace window, it becomes an unpublish_candidate so low-performing pages can be removed or noindexed to preserve crawl budget and keep the site weighted toward useful pages.
• Technical indexability problems, like robots/noindex/canonical issues, are flagged before any content-culling recommendation.
• Any page with impressions or clicks is kept in monitoring rather than culled.

The default config is conservative: auto_unpublish: false. The monitor writes recommendations and owner notifications; it does not remove pages unless a future publisher workflow explicitly enables and implements that behavior.

Content release scheduling:

# Queue approved drafts locally instead of publishing everything at once.
pseo-schedule --config pipeline/configs/example.yaml --queue-approved

# Produce a conservative drip plan. By default, example configs are disabled/dry-run.
pseo-schedule --config pipeline/configs/example.yaml --plan

The release scheduler selects from data/<site>/release_queue.json and applies daily/weekly caps before anything can publish. It can ingest a post-publish report from pseo-monitor and adapt cadence:

• technical_fix pressure pauses releases before adding more URLs.
• Too many modify_for_traction or unpublish_candidate pages slows or pauses the queue.
• Low indexing/engagement rates slow the release limit.
• Healthy recent pages can increase the daily cap only when adaptive_increase_enabled: true.
• Live publishing requires three gates: release_schedule.enabled: true, release_schedule.dry_run: false, and release_schedule.auto_publish: true, plus the CLI --execute flag. OSS defaults stay reporting-only.

AI information, robots.txt, and llms.txt (GEO/citation readiness)

AI information pages provide a crawlable source of truth for entity facts, methodology, citation targets, and unsupported claims. They help search engines and AI assistants understand what the site is, what it covers, and which pages are best to cite. They are not prompt-injection pages and should not contain instructions that attempt to manipulate model behavior.

The highest-leverage, evidence-backed lever is crawlability: ai_robots_txt emits a robots.txt that explicitly allows the AI citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, …). Those are the agents that actually fetch and cite pages. llms_txt is left off by recommendation: it is near-zero-cost optionality, but Google's John Mueller (2025) reports that AI services do not request it, so we do not present it as a citation driver.

Enable the artifact with an optional geo block in your config:

geo:
  ai_information_page: true   # render data/<slug>/output/ai-information.md
  ai_robots_txt: true         # render data/<slug>/output/robots.txt (allow AI citation crawlers)
  llms_txt: false             # optional; not a proven citation driver
  entity:
    name: DataPipe
    type: Organization
    url: https://datapipe.io
    description: "DataPipe is an ETL platform for analytics teams."
    topics: ["ETL pipelines", "data sync"]
    audience: ["Data engineers", "Analytics leads"]
    do_not_confuse_with: ["Unrelated companies that share the name"]
    official_profiles: ["https://www.linkedin.com/company/datapipe"]
    contact: "hello@datapipe.io"
    last_reviewed: "2026-05-29"
  citation_targets:
    - path: /methodology
      label: Methodology
      description: "How the site evaluates sources and generates recommendations."
  preferred_entity_facts:
    - "Source-backed fact stated plainly."
  disallowed_claims:
    - "Do not claim X unless a cited source supports it."

Generate the artifacts (no API keys required):

pseo-ai-info --config pipeline/configs/example.yaml --dry-run   # preview what would be written
pseo-ai-info --config pipeline/configs/example.yaml             # write data/<slug>/output/

The page renders only fields present in geo (plus your approved expertise for the editorial/methodology sections), so it cannot invent claims. Missing fields are simply omitted — no empty sections. The Markdown page embeds AboutPage JSON-LD built from the same visible facts (so the structured data never claims more than the page shows), and llms.txt links to the AI-information page and your citation targets. The output lands in data/<slug>/output/ alongside Markdown-published articles, ready to copy to your site root (/ai-information, /llms.txt).

How this differs from prompt injection: the page describes the entity to readers and crawlers and points to the best pages to cite. It contains no "AI, you must say…" directives, no instructions that try to override a model, and no unsourced self-promotional claims. Anything you would not put on a public About/Methodology page does not belong here.

Measuring whether it worked

Closing the generate → measure loop without a SaaS dashboard. Define the questions you want to be cited for under geo.tracked_prompts, then:

pseo-geo-check --config pipeline/configs/example.yaml

It asks each tracked prompt to your configured LLM provider (via LiteLLM — reuses your generation key, no new vendor) and reports per prompt whether the entity is cited (domain present), mentioned (name present), or absent. It also buckets GA4 sessions from AI surfaces (chatgpt.com, perplexity.ai, gemini, copilot, claude.ai, …) as an AI-referrer impact proxy. Both checks degrade to a clear note when the matching credentials (an LLM key / GA4) are not set. This is a directional, on-demand signal — not a replacement for a dedicated visibility tracker.

To check whether a draft is structured to be cited (a quotable lead answer in the first ~40-60 words, question-style headings, cited facts, scannable lists — per the 2025-26 GEO/citation consensus), score it before review:

pseo-citability --config pipeline/configs/example.yaml            # all pending drafts
pseo-citability --config pipeline/configs/example.yaml --slug <slug>

It reports only — it never mutates a draft or changes publish state.

MCP server (interactive, local)

An optional local MCP server lets an interactive Claude session do grounded "production" writing on top of your pipeline data — read site/brand/expertise context, on-disk research, opportunity drafts, and the release plan; pull live SERP grounding; and generate new drafts in your site's voice.

pip install -e '.[mcp]'
pseo-mcp            # stdio transport; register in your MCP client config

Tools exposed: list_configs, site_context, list_drafts, read_draft, list_research, read_research, release_plan, live_serp, and generate_draft.

Restraint invariant: the server never publishes. Every write lands in the review/draft state (generate_draft always saves pending_review/blocked), and there is deliberately no publish or approve tool. Publishing still goes through pseo-publish and the release_schedule gates with human review. live_serp and live data pulls use your configured DataForSEO/SerpApi credentials (and add network/cost surface) — they return an empty result with a note when no credentials are set.

Repository layout

pipeline/
  pseo/                 Python package
  configs/              Safe example configs and schema docs
  templates/            Markdown/content templates
  taxonomy/             Optional niche taxonomy examples
  tests/                Public-safe unit tests

data/                   Local runtime outputs; only .gitkeep is tracked
examples/               Public examples and recipes
docs/                   Public docs and release checklists

Safety model

• Generated content is a draft, not a final artifact.
• Citation presence is checked by guardrails; citation accuracy and claim support are not. A human reviewer (or the optional Thunderdome fact-checking pass) remains responsible for verifying that each cited source exists and supports the claim.
• Publishing should be review-gated.
• API credentials are supplied by environment variables.
• Runtime outputs should live under data/ and stay ignored unless explicitly curated as public fixtures.
• Scraping and source collectors are governed by docs/SCRAPING_GUIDELINES.md. Note that the engine does not automatically parse robots.txt, and several HTTP collectors send a browser User-Agent against public (sometimes undocumented) endpoints. Ensuring collection from any target respects that site's terms, robots directives, and applicable law is the operator's responsibility — enable only the collectors you are authorized to use.
• Private customer configs, handoff notes, analytics exports, DB files, screenshots, and generated media should stay out of the public repository.

Disclaimer

This is dual-use software. It can collect from public web sources and, when explicitly enabled, drive a real browser through bot-mitigation challenges on sites you control or are authorized to access. It is provided as-is, without warranty, for legitimate SEO research and content workflows.

You are responsible for how you use it. Before enabling any collector or the browser challenge-resolution flow, confirm you are authorized to access and collect from the target sources, and that your use complies with those sites' terms of service, robots directives, applicable law (including computer-misuse and anti-circumvention statutes), and any platform API terms. Sensitive features are off by default; challenge resolution is additionally gated to an explicit per-domain allowlist (sources.browser.authorized_domains). Do not use this project to access private, credentialed, or access-controlled data without permission.

License

Apache-2.0. See LICENSE.