Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
127 lines (94 loc) · 8.52 KB

ARCHITECTURE.md

File metadata and controls

127 lines (94 loc) · 8.52 KB
Raw

Architecture

English · 中文

Pipeline overview

  1. Graph build — Extracts entities and relationships from your document into a Neo4j knowledge graph. NER uses few-shot examples and rejection rules to filter garbage entities. Chunk processing is parallelized with batched Neo4j writes (UNWIND).

  2. Agent setup — Generates personas grounded in the knowledge graph. Each entity gets 5 layers of context: graph attributes, relationships, semantic search, related nodes, and LLM-powered web research (auto-triggers for public figures or when graph context is thin). Individual vs. institutional personas are detected automatically via keyword matching.

  3. Simulation — All three platforms (Twitter, Reddit, Polymarket) run simultaneously via asyncio.gather. A single LLM-generated prediction market with non-50/50 starting price drives Polymarket trading. Agents see cross-platform context: traders read Twitter/Reddit posts, social media agents see market prices. A sliding-window round memory compacts old rounds via background LLM calls. Belief states track stance, confidence, and trust per agent with heuristic updates each round.

  4. Report — A ReACT agent writes analytical reports using simulation_feed (actual posts/comments/trades), market_state (prices/P&L), graph search, belief trajectory, and Nash equilibrium tools. Reports cite what agents actually said and how markets moved.

  5. Interaction — Chat directly with any agent via persona chat, send questions to groups, or branch the simulation with a counterfactual event at any round to explore "what if" scenarios side-by-side. Click any agent to view their full profile and simulation history.

Cross-platform simulation engine

All three platforms execute simultaneously each round. Data flows between them:

                    ┌─────────────────────────────────────────┐
                    │         Round Memory (sliding window)    │
                    │  Old rounds: LLM-compacted summaries     │
                    │  Previous round: full action detail       │
                    │  Current round: live (partial)            │
                    └──────┬──────────┬──────────┬────────────┘
                           │          │          │
                    ┌──────▼───┐ ┌────▼─────┐ ┌─▼────────────┐
                    │ Twitter  │ │  Reddit  │ │  Polymarket   │
                    │          │ │          │ │               │
                    │ Posts    │ │ Comments │ │ Trades (AMM)  │
                    │ Likes    │ │ Upvotes  │ │ Single market │
                    │ Reposts  │ │ Threads  │ │ Buy/Sell/Wait │
                    └──────┬───┘ └────┬─────┘ └─┬────────────┘
                           │          │          │
                    ┌──────▼──────────▼──────────▼────────────┐
                    │         Market-Media Bridge              │
                    │  Social sentiment → trader prompts       │
                    │  Market prices → social media prompts    │
                    │  Social posts → trader observation       │
                    └──────┬──────────┬──────────┬────────────┘
                           │          │          │
                    ┌──────▼──────────▼──────────▼────────────┐
                    │         Belief State (per agent)         │
                    │  Positions: topic → stance (-1 to +1)    │
                    │  Confidence: topic → certainty (0 to 1)  │
                    │  Trust: agent → trust level (0 to 1)     │
                    └─────────────────────────────────────────┘

Polymarket integration

A single prediction market is generated by the LLM during config creation, tailored to the simulation's core question. Market-title generation routes through the Smart slot (see Models) so phrasing is sharp, time-bound, and resolvable — this is the prompt that frames the entire simulation, so it's worth the stronger model. The AMM uses constant-product pricing with non-50/50 initial prices based on the LLM's probability estimate. Traders see actual Twitter/Reddit posts in their observation prompt alongside portfolio and market data.

Performance

Optimization Before After
Neo4j writes 1 transaction per entity Batched UNWIND (10x faster)
Chunk processing Sequential Parallel ThreadPoolExecutor (3x faster)
Config generation Sequential batches Parallel batches (3x faster)
Platform execution Twitter+Reddit parallel, Polymarket sequential All 3 parallel
Memory compaction Blocking Background thread

Web enrichment

When generating personas for public figures (politicians, CEOs, founders) or when graph context is thin (<150 chars), the system makes an LLM research call to enrich the profile with real-world data. Set WEB_SEARCH_MODEL=perplexity/sonar-pro in .env for grounded web search via OpenRouter.

Per-round frame API

GET /api/simulation/<id>/frame/<round> returns a compact snapshot of a single round — actions, active-agent count, market prices at that round, and belief state — for scrubbing UIs on large simulations. Alternative to loading all N × M actions upfront via /run-status/detail. Query params: platforms=twitter,reddit,polymarket, include_belief, include_market. Used by ReplayView for timeline scrubbing and by the CLI (miroshark-cli frame <id> <round>).

Memory & retrieval pipeline

Beyond the simulation engine, MiroShark ships a research-grade graph memory stack inspired by Hindsight, Graphiti, Letta, and HippoRAG. Every ingested document and simulation action flows through:

Ingestion

text → NER (with ontology)
     → batch embed (OpenRouter text-embedding-3-large or local Ollama)
     → Entity resolution (fuzzy + vector + LLM reflection — dedups "NeuralCoin"/"Neural Coin"/"NC")
     → MERGE entities into Neo4j with canonical UUIDs
     → Contradiction detection (LLM adjudicates same-endpoint pairs → invalidate old)
     → CREATE RELATION edges with {valid_at, invalid_at, kind, source_type, source_id}

Retrieval (storage.search(...))

query
  ├─ vector edge search (Neo4j HNSW)   ─┐
  ├─ BM25 edge search (Neo4j fulltext) ─┼─ temporal + kind filters → fused candidates (top 30)
  └─ BFS traversal from seed entities  ─┘
                                        ↓
                           BGE-reranker-v2-m3 cross-encoder (Apple MPS / CUDA / CPU)
                                        ↓
                         top `limit` with _sources tag ("v" / "k" / "g" / combos)

Zoom-out layer (storage.build_communities(...))

  • Leiden community detection on the entity graph (via igraph)
  • LLM-generated title + 2-sentence summary per cluster
  • Persisted as :Community nodes with MEMBER_OF edges
  • Semantic search over cluster summaries via the browse_clusters agent tool

Reasoning memory

Every report generation persists a full ReACT trace as a traversable subgraph:

(:Report)-[:HAS_SECTION]->(:ReportSection)-[:HAS_STEP]->(:ReasoningStep)

Step kinds are thought | tool_call | observation | conclusion. Query past reports' reasoning with storage.get_reasoning_trace(section_uuid).

What it buys you

  • Multi-hop queries work (graph traversal catches facts where only the connection matches)
  • Temporal queries work (as_of="2026-04-10T14:00Z" returns the world as known at that moment)
  • Epistemic filtering (kinds=["belief"] returns only agent opinions, not ground-truth facts)
  • Reports are re-queryable ("why did the agent conclude X?")
  • First-call recall is high enough that the report agent's 5-call budget goes further

All 11 features are on by default and can be individually disabled via .env flags — see Configuration.