🥈 Silver Medal — ICCSE 2026 Agentic AI Innovation Competition
A voice-first English speaking-practice companion. Pick a real scenario, talk to it, and get phoneme-level pronunciation scoring + grammar/expression correction in the flow, a post-session report, and a quantifiable "are you improving?" trend over time. It remembers you (name, goal, interests, weak sounds), proactively reaches out to drill what's due, and you can even keep practising inside WeChat with native voice messages.
- 🌐 Live: https://uiniq.zymonody.cloud
- Scene practice — interview / ordering / meeting / IELTS / daily, each a real task with a goal, a role, key expressions, and pass criteria (custom scenes too).
- Real-time voice — streaming ASR (live captions) + streaming TTS; tap the orb to speak, tap to interrupt; the coach replies in ~2–3 s.
- Pronunciation assessment (phoneme-level) — Tencent 智聆 SOE: per-word and per-phoneme scores + stress, shown coloured inside the sentence you said, with a detail drawer.
- Grammar / expression correction — Doubao corrects in passing and in a structured side panel, without interrupting you.
- Post-session report — pass/needs-practice, ability radar, key expressions used/missing, one-tap replay of the weakest line.
- Quantifiable improvement — Progress page: recent-vs-earlier deltas (pronunciation / fluency / grammar), radar, fluency curve, scene mastery; weak sounds + frequent errors feed an SRS loop.
- Resume & review — the live conversation is cached in Redis (refresh/exit and continue), and the full transcript is persisted to Postgres so any past session can be read back like a chat log in History.
- WeChat native voice (iLink direct) — bind by QR, then send voice messages to the coach in WeChat: it transcribes, replies, returns structured feedback, and shows a typing indicator.
- Proactive companion — daily nudges, streak, and a researched feed of practice suggestions.
Passive POST /api/chat (SSE) ─┐
├─▶ app/session.run_agent_turn ─▶ agent harness (LLM + tools)
Active proactive engine ─────┘ │
(background ticker) ├─ coach tools write to the LearnerModel
▼
Passive out: SSE to the request Active out: WebSocket /ws push to the browser
Both the HTTP chat and the background engine call the same run_agent_turn;
only the prompt source and the output transport differ. Pronunciation + grammar
assessment runs out of band in an arq worker and is pushed over /ws.
- Backend — FastAPI, layered
api → services → repositories → db/models. Postgres + pgvector (learner model, sessions, recordings, memory embeddings), Redis (nudge bus · conversation resume cache · sample-reply warmup · arq jobs), an arq worker (assessment · proactive nudge · feed · sample-reply warmup), Alembic migrations applied on startup. - Frontend — Next.js 14 (App Router, TypeScript, Tailwind).
- AI — ByteDance Volcano (火山引擎): chat on Ark Doubao (OpenAI-compatible, streaming function-calling, reasoning disabled for real-time latency), voice on Doubao Seed-TTS 2.0 + streaming/file ASR. Pronunciation on Tencent 智聆 SOE (WebSocket).
- Storage — audio on Qiniu Kodo (S3-compatible) via a pluggable storage
layer (
STORAGE_BACKEND=local|s3).
backend/ FastAPI service
app/
main.py app factory + lifespan (proactive engine, weixin poller, migrations)
session.py shared agent entrypoint (passive + active)
api/ chat (SSE) · voice (WS ASR + TTS) · recordings · sessions ·
conversation (resume) · profile · live (/ws bus) · weixin
agent/ reusable harness: runner, tools, llm/ (OpenAI-compatible)
skills/coach/ the coaching skill (prompt + tools)
services/ assessment · recording · learner · conversation_cache · weixin · sample_cache
integrations/ asr · tts · pronunciation (SOE) · weixin (iLink + SILK) · storage (local|s3)
jobs/ arq worker (run_assessment, generate_proactive_nudge, generate_feed, generate_sample_reply)
learner/ proactive/ db/ (models + Alembic)
scripts/ migrate · seed_demo · repl
frontend/ Next.js (App Router)
app/(home)/ / · /progress · /history · /settings + /practice/[topicId]
components/uiniq/ Talk (orb) · AssessmentBubble · SessionPanel · Onboarding · ...
hooks/ useChat · useRecorder · useAssessment · useWs
deploy/ nginx snippet (note the exact-match /ws block)
Full stack with Docker (Postgres + Redis + backend + worker + frontend):
cp backend/.env.example backend/.env # fill in the credentials below
docker compose up -d --build # backend waits for DB/Redis healthy
docker compose exec backend python -m scripts.seed_demo # ~2-week history so Progress isn't empty
# frontend http://localhost:3001 · API http://localhost:8800The worker must be running for assessment / nudges / feed (
docker compose up -dstarts it;make upwith noSVC=starts everything).
Text-only harness (no mic): cd backend && python -m scripts.repl.
Non-secret model/voice selection is fixed in config.py; env carries only credentials:
LLM_API_KEY, VOLCANO_APP_ID/ACCESS_KEY, TENCENT_APP_ID/SECRET_ID/SECRET_KEY,
WEIXIN_ENABLED/WEIXIN_ENCRYPTION_KEY, and for Kodo STORAGE_BACKEND=s3 +
S3_ENDPOINT/REGION/BUCKET/ACCESS_KEY/SECRET_KEY. See backend/.env.example
(committed with placeholders only — real keys live in .env / CI secrets).
- Resilience — timeouts + bounded backoff retries on every LLM/ASR/TTS call;
per-tool timeout + per-turn token budget; the streaming LLM client retries
connection setup. WSS to Volcano/SOE forces a direct connection (
proxy=None). - Observability — structured logs with an
X-Request-IDthreaded HTTP → agent → proactive (LOG_JSON), run telemetry, Prometheus/metrics. - Health —
/healthz(liveness) +/readyz(DB-checked); DockerHEALTHCHECK. - Scale-out — Redis nudge bus + arq jobs let the API, worker, and proactive
engine run as separate processes; WS clients filter by
learner_id. - Config safety —
ENV=prodfails fast on missing keys / unsafe CORS. - CI/CD — GitHub Actions: ruff + Alembic + pytest (backend), eslint + tsc +
vitest + build (frontend); on push to
main, deploy to the server via Docker Compose.
cd backend && pip install -r requirements.txt && pytest -q
ruff check app tests scripts
cd ../frontend && npm test