[!IMPORTANT] Status on main (2026-09-23).

• «LLM» mode — the product. The web app sends every utterance with its dialog context through a same-origin Next.js proxy to the Python router core-llm/ on OpenRouter (google/gemini-2.5-flash-lite; its author reports 104/104 on the dev set at ~457 ms p50). The client's voice is recognized by ElevenLabs Scribe v2 and every reply is spoken by ElevenLabs (Flash v2.5 in Russian, v3 conversational in Kazakh) through the Go voice service voice/. Needs an OpenRouter key and an ElevenLabs key: NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build.
• «Мок» mode — keyless. The same UI, dialog executor, 31 mock backend actions, supervisor trace and quality metrics, with a transparent keyword baseline instead of the LLM and browser speech instead of ElevenLabs: docker compose up --build and nothing else.
• In progress 🚧: the Go backend («Бэкенд» mode), streaming voice in the web app (the voice service's /ws/voice gateway), persistence and a public deployment. Every capability below is marked ✅ / 🚧 / 📋.

Кратко по-русски. Bagyt — веб-симулятор голосового робота контакт-центра страховой компании (вымышленная Saqta Insurance из стартового кита Halyk Bank). Клиент говорит по-русски, по-казахски или вперемешку; робот выбирает один из 40 сценариев с учётом всего диалога, отвечает голосом, переспрашивает, когда не уверен, передаёт разговор оператору с контекстом и не выполняет необратимых действий без явного «да». Супервизор после каждой реплики видит сценарий, уверенность, обоснование, альтернативы и задержку по этапам. Проверка без ключей: docker compose up --build → http://localhost:3000/admin, режим «Мок» (прозрачный детерминированный baseline вместо LLM). С ключом OpenRouter в core-llm/.env: NEXT_PUBLIC_API_MODE=core docker compose --profile llm up --build — режим «LLM» и продуктовый роутер core-llm/ (Gemini 2.5 Flash-Lite через OpenRouter, 104/104 на dev-наборе по отчёту автора, ~457 мс p50). Голос в режиме «LLM» — ElevenLabs: распознавание Scribe v2 (language_code=kk понимает русский, казахский и смешанную речь) и озвучка Flash v2.5 (RU) / v3 conversational (KZ); нужен ELEVENLABS_API_KEY в корневом .env и NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build. Живая версия: https://bagyt.plus.

From the hackathon floor

Photos from the day: the entrance, the hall, the team at work, picking the case

The morning queue at the entrance	The hall	Swag
Building Bagyt		qairuhub
Picking the case: Track 09 «Коммуникации» → Voice Router

Jury quick path

The organizers ask	Short answer	Details
What does it solve?	Voice robots mis-route live speech — topic switches, requests between two scenarios, Russian↔Kazakh switching inside a phrase — because the scenario is picked by an encoder intent classifier. Bagyt picks it with an LLM layer over the dialog context, asks or hands off when unsure, and shows the supervisor why.	§1
How do I run it?	Live: bagyt.plus — just open it: nothing to install, no login, no keys (they live on the server). Locally without keys: docker compose up --build → http://localhost:3000/admin (engine «демо»). Locally with the LLM router and ElevenLabs voice: the .env files from our Google Drive folder (or your own keys) in core-llm/.env (OpenRouter) and .env (ElevenLabs), then NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build. Without Docker: cd frontend && npm ci && npm run dev.	§7
Which technologies?	Next.js 16 · React 19 · TypeScript · Tailwind 4 · Python LLM core on OpenRouter (Gemini 2.5 Flash-Lite) · Go 1.26 voice service on ElevenLabs (Scribe v2 STT, Flash v2.5 / v3 conversational TTS) · browser Web Speech API as the keyless fallback	§5
How do I verify it?	A 10-step walkthrough in the web app (RU, KZ, mixed, multi-intent, topic return, confirmation, clarify → handoff), the same phrases in «LLM» mode, the official evaluate.py and the unit tests	§8

Try it online

🌐 Live app	bagyt.plus
🔑 Jury access	Open access — no login, no password. The call screen (/call) and the supervisor console («Аналитика», /admin) are open to everyone
🎬 Demo video (3 min)	Google Drive folder
🔐 Keys for a local run	The .env files (OpenRouter, ElevenLabs) are in the same Google Drive folder, shared for the jury — never in this repo. Put them in place (§7.6) and run NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build
🧪 Test clients	Say or type a phone when the robot asks: +77010000007 Sergey Popov (claim under review) · +77010000001 Arman Tulegenov (OGPO + CASCO) · +77010000003 Yerlan Omarov (policy to renew) — all synthetic, §8.3

About API keys. None are in this repository, on purpose. bagyt.plus runs with our OpenRouter and ElevenLabs keys in private env files on the server (§7.5), so nothing is needed on your side. For a full local run, take the .env files from the Google Drive folder next to the demo video; without them, «Мок» works with no keys at all.

Test it in 60 seconds:

1. Open bagyt.plus → «Аналитика» (the conversation on the left, the robot's reasoning on the right). No login needed.
2. The live app keeps the OpenRouter and ElevenLabs keys on the server — you need no keys of your own. Turn the replies' sound on with the speaker icon next to the text field.
3. Click the mic, say «Хочу продлить ОГПО и добавить сына» and click again to send — or type it. Then try «Кеше аварияға түстім, но я не виноват, виновник у вас застрахован».
4. After each phrase, read the right column: the scenarios with confidence, why, the rejected alternatives, what the robot did and how many milliseconds each stage took.
5. More: the 10-step walkthrough in §8 · no key? docker compose up --build gives the keyless «Мок» build (§7.1).

Contents

1. Problem and users
2. What is implemented
3. How it works
4. Architecture
5. Technologies
6. Requirements
7. Install and run
8. How to verify
9. Results
10. Data and integrations
11. Design
12. Decision log
13. Limitations and known issues
14. Roadmap
15. Team
16. Documentation map
17. Maintaining this README

---

1. Problem and users

The problem (from the case). Voice robots have learned to speak and listen well, but the layer that chooses the conversation scenario is usually still an encoder intent classifier trained on fixed phrasings. It breaks exactly where live speech starts: the client changes topic mid-dialog, asks something on the border of two scenarios, or switches from Russian to Kazakh inside one sentence. Every wrong choice is a transfer to an operator or a lost client — and a pause longer than a second already sounds like a dropped call.

Who it is for.

User	What they need	What Bagyt gives them
Contact-center client	Explain the problem in their own words — in Russian, Kazakh or both; change topic; get it solved without an operator	Scenario chosen from the whole dialog, not from keywords; a topic stack that returns to interrupted questions; answers grounded in the client's own data; a voice reply in the client's language
Supervisor / operator	See which scenario was chosen, why, where the robot hesitated or failed, and how fast it was	A trace after every turn — scenario, confidence, reason, alternatives, policy verdict, actions, per-stage latency — plus the official quality metrics on the dev set

Domain. The organizer's synthetic starter kit: Saqta Insurance, a fictional Kazakh non-life insurer (OGPO motor liability, CASCO, DMS health, travel, property, accident) — 40 scenarios + 3 system intents, 43 slots, 31 backend actions, a knowledge base and 11 test clients. "Today" in the data is 2026-10-01.

Our answer in one line. Replace the classifier with an LLM routing layer that reads a compact index of every scenario — its meaning, Russian and Kazakh cues, and not_this_if boundaries — together with the dialog context, and answers with scenario IDs and confidence percentages only. A deterministic policy, not the model, then decides whether to act, ask, or hand the call to a human, and assembles the explanation the supervisor sees.

---

2. What is implemented

Legend: ✅ works on main · 🚧 in progress, not on main yet · 📋 planned · ❌ cut · ⏳ to be filled in

Client conversation

Capability	Status	Where
ElevenLabs speech recognition, hands-free: click the mic and speak — it stops by itself after a pause (VAD) or on a second click, with live partial transcripts and a mic-level ring → Next.js POST /api/stt → voice service POST /api/voice/stt → Scribe v2 with language_code=kk (Russian, Kazakh and mixed speech). The recognizer in «LLM» mode; in «Мок» the browser's Web Speech recognizes, and its network errors switch to ElevenLabs automatically	✅ needs ELEVENLABS_API_KEY + --profile voice	api/stt/route.ts, voice/ — §5.2
ElevenLabs voice replies in «LLM» mode: Next.js POST /api/tts → voice service → Flash v2.5 for Russian, v3 conversational for Kazakh, the Russian voice for English greetings. Each reply keeps its MP3 for the conversation: play/pause, seekable waveform, elapsed/duration, download	✅ needs ELEVENLABS_API_KEY + --profile voice	api/tts/route.ts, audio-message.tsx
Browser voice (keyless): Web Speech API in, speechSynthesis out — «Мок» mode, and the fallback when the voice service is unavailable	✅	frontend/src/lib/voice.ts
Text input as the fallback channel	✅	conversation-panel.tsx
One RU / ҚАЗ switch in the header for the interface, speech recognition and the reply, saved in the browser; the header also shows the engine («демо» or «LLM») and has «Новый диалог»	✅	lib/ui-language.ts, console-shell.tsx
Language detection ru / kk / mixed; answer in the client's dominant language, keep the session language on code-switched turns	✅	mock/router.ts, mock/engine.ts
Client identification by phone / IIN against mock_backend.json; spoken numbers → digits in RU and KZ («плюс жеті жеті жүз бір…» → +7701…)	✅	mock/router.ts (wordsToDigits, extractSlots)

Routing and dialog logic

Capability	Status	Where
LLM router — the product: system prompt + a compact index of the catalog (scenarios.index.txt) + dialog context → ID:PERCENT pairs → decision policy. CLI, dev-set evaluation, dialog replay with context, model benchmark	✅	core-llm/router.py, core-llm/prompt.md, core-llm/scenarios.index.txt
«LLM» mode in the web app: browser → same-origin proxy POST /api/core-route → core-llm/server.py (POST /api/route, port 8090, private) → OpenRouter. The core's verdict (route / clarify / handoff / out_of_scope / goodbye) is authoritative; the browser executor then collects slots and runs the demo actions. The key never reaches the browser, and there is no silent fallback to the mock	✅ needs OPENROUTER_API_KEY	api/core-route/route.ts, lib/core-router.ts, core-llm/server.py
The same routing inside the Go turn API (SSE)	🚧	integration notes in core-llm/README.md
Router output contract used by the UI: ranked scenarios[] with confidence + reason, alternatives[], language, slots, is_continuation, model, tier	✅	frontend/src/lib/contract.ts
Keyless mock router: token overlap with the catalog's ru/kk examples + hand-written cues that mirror not_this_if; labelled «Мок» in the UI	✅	mock/router.ts
Multi-intent: split the utterance; urgent scenarios first (SC11, SC15, SC38), then in spoken order	✅	core-llm/prompt.md, mock/router.ts, mock/engine.ts
Decision policy: ≥ 0.75 run · 0.45–0.75 clarify · < 0.45 twice → service menu (operator on request) · out-of-scope · goodbye (UI); route ≥ 60 % · extra intents ≥ 20 % (core-llm)	✅	mock/engine.ts (decide), core-llm/router.py (Router.decide)
Topic stack: park the interrupted scenario, come back to it when the new one is done	✅	mock/engine.ts
Continuation: an answer to a pending slot or a yes/no question is not re-routed	✅	mock/engine.ts, core-llm/prompt.md (CONTEXT rules)
Greetings without a request — SYS_GREETING (status: greeting, policy action greeting), an application system intent; the official 40 scenarios are unchanged. Replies «Здравствуйте, чем могу помочь?» / «Сәлеметсіз бе, қалай көмектесе аламын?» / “Hello, how can I help you?” (for “Hello”, “Hi”, “Good morning”) through the same TTS path; keeps the active scenario, the pending slot or confirmation and the queue, and does not count as an unclear turn. «Здравствуйте, хочу продлить полис» still routes to SC27	✅	core-llm/prompt.md, core-llm/router.py, mock/engine.ts
Slot extraction: phone, IIN, policy / claim number, plate, city, dates relative to 2026-10-01, doctor, country, …	✅	mock/router.ts
Scenario executor: identify → one question per missing slot → actions → read back → explicit «да» → execute → close	✅	mock/engine.ts (execute, confirmFlow)
All 31 backend actions of actions.json as mocks over mock_backend.json + knowledge_base.json	✅	mock/actions.ts
The 9 irreversible actions run only after a preview and an explicit confirmation	✅	mock/actions.ts, mock/engine.ts
Handoff to one of 6 operator queues with a context summary	✅	mock/engine.ts
Hybrid fast path with a measured latency gain	📋	—
Speculative routing on partial transcripts	📋	—

Supervisor console (/admin)

Capability	Status	Where
Conversation next to «what the robot understood»: scenario, confidence, reason, live candidates, the catalog's boundary rules	✅	admin/page.tsx, components/app/admin/decision.tsx
Dialog state: client, awaited slot or confirmation, known data, actions (read / preview / executed)	✅	components/app/admin/dialog.tsx
Per-stage latency waterfall against the 1.5 s target	✅ (mock timings are simulated)	components/app/admin/speed.tsx
Session metrics: turns, average confidence, clarifications, handoffs, median response time	✅ (in memory)	lib/api.ts (computeStats)
Routing quality card (the official evaluate.py metrics on the dev set in one click)	❌ removed from the console in bc402c6 — run the metrics from the CLI (§8.4)	components/app/admin/quality.tsx (unused)
Trace JSON of the last turn in the dataset README format	✅	components/app/admin/shared.tsx (readmeTrace)
Turn log	✅	components/app/admin/turn-log.tsx
History across sessions and error statistics over time (PostgreSQL)	📋	—
Catalog editing without developers	📋	—

Backend and infrastructure

Capability	Status	Where
Go HTTP service with GET /health	✅	backend/cmd/server/main.go
Go turn API behind «Бэкенд» mode: POST /api/session, POST /api/turn (SSE), trace, supervisor stats, scenarios, POST /api/eval/run	🚧 contract fixed	docs/API_CONTRACT.md
LLM access: OpenRouter in core-llm ✅; OpenAI / NVIDIA / any OpenAI-compatible provider behind one Go interface 🚧	✅ · 🚧	core-llm/router.py
Voice service voice/ (Go): ElevenLabs Scribe v2 (streaming and batch) + per-language TTS; HTTP /api/voice/stt and /api/voice/tts used by the web app; a real-time WebSocket gateway /ws/voice with barge-in, speculative replies and cached fillers; per-call logs and p50/p95 latency stats	✅ HTTP in the web app · 🚧 /ws/voice not wired to the web app yet	voice/, voice/README.md — §5.2
geko.sh Seta STT / Tokay TTS (KZ-first)	📋	—
PostgreSQL persistence of turns and traces	📋	—
One command: docker compose up --build (web app, keyless); --profile llm adds the core-llm service, --profile voice the ElevenLabs voice service	✅	docker-compose.yml, frontend/Dockerfile, core-llm/Dockerfile, voice/Dockerfile
Tests: the core-llm HTTP adapter and greetings; the frontend's core adapter and dialog engine; the voice service against fake ElevenLabs servers (no credits)	✅	core-llm/test_server.py, frontend/scripts/test-core.cjs (npm test), voice/ (go test ./...)
Deployment: bagyt.plus on a VPS — Nginx with Let's Encrypt HTTPS in front of the full Compose stack (--profile server); keys in private server env files	✅	deploy/, §7.5
Phone channel: Asterisk AudioSocket (a Kazakh SIP number) and Twilio Media Streams transports in the voice service, with a setup guide	✅ code · ⏳ live number	voice/transport/, voice/deploy/README.md
Emotion detection and tone adaptation	❌ cut	—

---

3. How it works

3.1 One turn, from voice to answer

1. Hear. The client clicks the mic and speaks. In «LLM» mode the recording goes through /api/stt and the voice service to ElevenLabs Scribe v2 with language_code=kk, which handles Russian, Kazakh and mixed speech — a second click sends it. In «Мок» the browser's Web Speech API transcribes (ru-RU or kk-KZ) and stops on a pause. The moment the client stops speaking is client_t0 — latency is measured from it.
2. Triage. Language ru / kk / mixed, normalization (spoken numbers → digits), urgency, and the split of a multi-intent utterance into parts.
3. Route. In «LLM» mode the utterance and its dialog context — previous turns with their scenarios, the active scenario, the bot's last reply — go through the same-origin proxy /api/core-route to core-llm. The model answers with nothing but ID:PERCENT pairs (about 6 tokens, which keeps routing near 450 ms); the core applies its policy, and the explanation shown to the supervisor is assembled deterministically from the scenario names, the percentages and the verdict. In «Мок» mode the keyword baseline returns the same shape. The console shows the candidates and their confidence.
4. Decide. Deterministic policy code — not the model — turns the ranking into an action: run, continue, clarify, handoff, out_of_scope or goodbye. Urgent scenarios go first; the rest wait on the topic stack.
5. Act. A per-scenario state machine driven by scenarios.json: identify the client, ask for the next missing slot, call mock actions, read back and wait for an explicit «да» before anything irreversible, close, then resume the interrupted topic.
6. Answer. One or two sentences built from the scenario's response templates and real action results — never invented facts — streamed sentence by sentence.
7. Speak. TTS in the reply language: in «LLM» mode ElevenLabs Flash v2.5 for Russian and v3 conversational for Kazakh — the only realtime ElevenLabs model that speaks it — with a replayable, downloadable recording; browser speech in «Мок». End-to-end latency = end of the client's speech → first audio of the answer.
8. Explain. The turn ends with a trace in the dataset README format; /admin renders it next to the conversation.

3.2 The LLM router in one call

$ python3 core-llm/router.py "Хочу продлить ОГПО и заодно добавить в него сына"
SC27:95 SC04:95   (primary SC27, intents [SC27, SC04], status route, ~450 ms)

One line out of the model: two requested scenarios (renewal SC27, add a driver SC04), both at 95 %, in spoken order. The policy turns it into status=route with two intents; an unsure answer (< 60 %) becomes clarify, and SYS_OUT_OF_SCOPE / SYS_GOODBYE / SC37 become out_of_scope / goodbye / handoff. Example from core-llm/README.md.

3.3 A real trace from the web app

The case's own example, typed into the web app in mock mode on main (produced by the same engine the browser runs):

{
  "turn": 1,
  "transcript": "Здравствуйте, я вчера оплатил, деньги списались, а заказ не подтвердился… а, и ещё, адрес доставки поменять надо",
  "language": "ru",
  "scenarios": [
    { "scenario_id": "SC30", "confidence": 0.94, "reason": "Charged but policy not issued — в реплике есть признак «деньги списал»" },
    { "scenario_id": "SC29", "confidence": 0.82, "reason": "Update contact details — похоже на примеры этого сценария из каталога" }
  ],
  "alternatives": [
    { "scenario_id": "SC33", "confidence": 0.34 },
    { "scenario_id": "SC31", "confidence": 0.17 },
    { "scenario_id": "SC12", "confidence": 0.15 }
  ],
  "reason": "Charged but policy not issued — в реплике есть признак «деньги списал»",
  "slots": { "incident_date": "2026-09-30", "contact_field": "address" },
  "actions": [],
  "latency_ms": { "stt": 43, "triage": 30, "router": 327, "policy": 16, "executor": 0, "response": 140, "tts_first_audio": 0, "total": 556 }
}

How to read it:

• Two requests in one breath → two scenarios, in spoken order: SC30 "Charged but policy not issued" (0.94) and SC29 "Update contact details" (0.82). The policy verdict is run SC30 with SC29 parked on the stack ("stack": ["SC29"]).
• Alternatives show what was considered and rejected: SC33 office addresses (0.34), SC31 payment methods (0.17), SC12 victim claim (0.15).
• Slots are already extracted: «вчера» → incident_date: 2026-09-30 (relative to the dataset's "today", 2026-10-01); «адрес … поменять» → contact_field: address.
• The answer acknowledges the second request and asks for the phone number, because SC30 requires identification: «Поняла, второй вопрос (обновление контактных данных) тоже разберём. Разберёмся. Подскажите, пожалуйста, ваш номер телефона.»
• latency_ms: in mock mode the stage timings are simulated delays; in the browser tts_first_audio and total are replaced by real measurements (end of speech → first audio).

---

4. Architecture

4.1 System overview

flowchart LR
  subgraph B["Browser · Next.js 16 app (frontend/)"]
    direction TB
    CALL["Client screen /call<br/>mic + text + audio player"]
    ADMIN["Supervisor console /admin<br/>decision · dialog · speed · quality"]
    STORE["Conversation store + dialog executor<br/>lib/store.ts · lib/mock/engine.ts"]
    VOICE["Browser speech (keyless)<br/>Web Speech API · speechSynthesis"]
    MOCK["Keyword baseline (keyless)<br/>lib/mock/router.ts"]
  end

  subgraph N["Next.js server routes"]
    direction TB
    PROXY["Route /api/core-route<br/>LLM proxy"]
    STTR["Route /api/stt<br/>recording → text"]
    TTSR["Route /api/tts<br/>reply → MP3"]
  end

  CORE["core-llm/server.py · Python<br/>prompt.md + scenarios.index.txt<br/>→ ID:PERCENT pairs → policy"]
  VSVC["voice/ · Go service, profile voice<br/>/api/voice/stt · /api/voice/tts<br/>+ /ws/voice gateway · phone transports"]

  subgraph G["Go backend (backend/) · in progress"]
    API["POST /api/turn → SSE events"]
  end

  subgraph X["External providers · keys only in server-side env files"]
    direction TB
    LLM["OpenRouter<br/>google/gemini-2.5-flash-lite"]
    EL["ElevenLabs<br/>Scribe v2 · Flash v2.5 · v3 conversational"]
  end

  KIT[("data/ · official starter kit<br/>40 scenarios · 43 slots · 31 actions<br/>knowledge base · 11 test clients")]

  CALL --> STORE
  ADMIN --> STORE
  STORE <--> VOICE
  STORE -->|mode Мок| MOCK
  STORE -->|mode LLM| PROXY
  STORE -->|voice in| STTR
  STORE -->|voice out| TTSR
  STORE -.->|mode Бэкенд| API
  PROXY --> CORE
  CORE --> LLM
  STTR --> VSVC
  TTSR --> VSVC
  VSVC --> EL
  MOCK --> KIT
  STORE -->|slots, actions, replies| KIT

One conversation store and one dialog executor drive both screens. The build-time mode NEXT_PUBLIC_API_MODE (mock / core / real) picks the router: the keyword baseline in the browser, the Python LLM core behind a same-origin proxy, or the Go backend (in progress). Every mode produces the same typed event stream, so the UI does not know the difference. Voice goes through two more same-origin routes, /api/stt and /api/tts, to the Go voice service, which alone holds the ElevenLabs key; in Compose it runs with VOICE_BRAIN=echo, so routing stays in core-llm. API keys stay on the server side (core-llm/.env, the root .env) and never reach the browser. The Go backend will reuse the core's two text files and its policy, as described in core-llm/README.md.

4.2 One turn as an event stream

sequenceDiagram
  autonumber
  actor C as Client
  participant UI as Browser (/call or /admin)
  participant E as Dialog engine (browser today, Go backend later)
  participant S as STT / TTS
  participant L as LLM router (core-llm via /api/core-route)
  C->>UI: clicks the mic, speaks RU / KK / mixed, pauses
  UI->>E: turn with text (or audio_base64) and client_t0
  E-->>UI: turn.start
  E->>S: audio (real mode)
  S-->>E: transcript + language
  E-->>UI: stt.final (text, language, ms)
  E-->>UI: triage (language, urgent, parts, ms)
  E->>L: utterance + dialog context + catalog index
  L-->>E: scenario IDs with confidence percentages
  E-->>UI: router.candidates (live), then router.decision
  Note over E: policy (thresholds, urgent first, topic stack), then executor (identify, slots, mock actions, confirmation)
  E-->>UI: policy, action (read / preview / execute), state
  E-->>UI: response.delta (streamed), then response.final
  E->>S: reply text (real mode)
  E-->>UI: tts.audio (base64 audio or browser_tts)
  E-->>UI: turn.done (trace, latency_ms)
  UI->>C: voice reply + trace on /admin

Event	Payload	What it drives in the UI
turn.start	turn, session_id, t0	a new row in the turn log
stt.partial / stt.final	text, language, ms	live transcript, language badge
triage	language, urgent, parts[], normalized, ms	parts of a multi-intent utterance
router.candidates (0..n)	candidates[], partial	live confidence bars
router.decision	RouterDecision, ms	scenario, reason, alternatives
policy	PolicyVerdict, ms	run / continue / clarify / handoff / out_of_scope / goodbye
action (0..n)	ActionCall — mode read / preview / execute	the actions list
state	DialogState	dialog card: client, awaited slot, stack
response.delta / response.final	text, language, ms	the streamed reply bubble
tts.audio	audio_base64 + mime, or browser_tts: true	playback
turn.done	Trace, latency_ms	trace JSON, speed waterfall, metrics
error	message	error banner; ends the turn

In «Мок» and «LLM» modes these events are produced in the browser; the Go backend will stream the same events over POST /api/turn → text/event-stream, one JSON event per data: line. Contract: docs/API_CONTRACT.md; types: frontend/src/lib/contract.ts.

4.3 Decision policy

flowchart TD
  IN(["Router decision:<br/>ranked scenarios + confidence"]) --> Q0{"Client says goodbye?"}
  Q0 -- yes --> GB["goodbye<br/>SYS_GOODBYE"]
  Q0 -- no --> Q1{"Yes / no to a<br/>pending confirmation?"}
  Q1 -- yes --> CF["continue<br/>execute or cancel the preview"]
  Q1 -- no --> Q2{"Out of scope?<br/>loans, life insurance, weather"}
  Q2 -- yes --> OOS["out_of_scope<br/>polite refusal + what we can do"]
  Q2 -- no --> Q3{"Answers the<br/>pending slot?"}
  Q3 -- yes --> SL["continue<br/>fill the slot, no re-routing"]
  Q3 -- no --> Q4{"Top confidence ≥ 0.75?"}
  Q4 -- yes --> RUN["run<br/>urgent first: SC11, SC15, SC38<br/>other intents → topic stack"]
  Q4 -- no --> Q5{"≥ 0.45 and not<br/>SYS_UNCLEAR?"}
  Q5 -- yes --> CLR["clarify<br/>one question, top-2 options"]
  Q5 -- no --> Q6{"Second low-confidence<br/>turn in a row?"}
  Q6 -- no --> CLR
  Q6 -- yes --> HO["service menu<br/>operator on request (SC37)"]

In «LLM» mode the core's verdict is authoritative (route_status: route / clarify / handoff / out_of_scope / goodbye; a scenario counts as requested from ROUTER_THRESHOLD=60 %, extra intents from 20 %). The thresholds in the diagram (CONFIDENCE_RUN = 0.75, CONFIDENCE_CLARIFY = 0.45 in contract.ts, from the dataset README) apply to the keyword baseline. An explicit request for a human (SC37) goes straight to handoff. The model proposes, code disposes: the router never executes anything by itself.

4.4 Scenario executor

stateDiagram-v2
  [*] --> Identify: scenario requires identification
  [*] --> Slots: no identification needed
  Identify --> Identify: phone not found, ask again once
  Identify --> Slots: find_client ok
  Slots --> Slots: ask the next missing slot, one question per turn
  Slots --> Actions: all required slots known
  Actions --> Preview: scenario has an irreversible action
  Actions --> Close: read-only scenario
  Actions --> Handoff: scenario handoff rule
  Preview --> Execute: client says yes
  Preview --> Close: client says no, nothing is done
  Execute --> Close
  Close --> Slots: resume the interrupted topic from the stack
  Close --> [*]: stack empty
  Handoff --> [*]

Slots already known from the identified client (phone, IIN, e-mail, city, the matching policy or claim) are filled automatically, so the robot does not ask for what it already knows.

4.5 Dialog state

Kept per session and sent to the UI after every turn (state event):

{
  "session_id": "s_ab12", "language": "ru",
  "client_id": "C007", "client_name": "Sergey Popov",
  "active_scenario": "SC17", "stack": ["SC27"],
  "slots": { "phone": "+77010000007" },
  "awaiting": { "kind": "slot", "slot": "claim_number" },
  "low_conf_streak": 0, "turn": 3, "ended": false
}

awaiting is null, { "kind": "slot", "slot": … } or { "kind": "confirmation", "action": … }.

4.6 Three modes, one contract

	«Мок» (default)	«LLM» — the product	«Бэкенд»
Chosen at build time	NEXT_PUBLIC_API_MODE=mock (default)	NEXT_PUBLIC_API_MODE=core	NEXT_PUBLIC_API_MODE=real
Router	keyword / cue baseline in the browser (mock-lexical)	core-llm via /api/core-route → OpenRouter (google/gemini-2.5-flash-lite)	Go backend, POST /api/turn (SSE)
Policy, slots, actions, replies	browser dialog executor over the starter kit (frontend/src/lib/mock/)	the core's verdict + the same browser executor	Go backend
STT	Web Speech API (Chrome / Edge)	ElevenLabs Scribe v2 via /api/stt (default) or Web Speech	ElevenLabs Scribe v2 Realtime, streamed
TTS	browser speechSynthesis	ElevenLabs via /api/tts: Flash v2.5 (RU) · v3 conversational (KZ); browser speech as the fallback	ElevenLabs over WebSockets
Keys	none	OPENROUTER_API_KEY in core-llm/.env; ELEVENLABS_API_KEY in .env for voice	provider keys in .env
Run	docker compose up --build	NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build	—
Status	✅	✅	🚧

4.7 Why this is not an intent classifier — and not hardcoded

• No encoder, no training. Nothing in this repo is a model trained to map utterances to intents. The product router is a generative LLM that reads a compact index of each scenario — its meaning, Russian and Kazakh cues, and not_this_if boundaries (e.g. "approved, but too little" → SC19 dispute, not SC17 status) — together with the dialog context, and answers with scenario IDs and confidence percentages (core-llm/).
• The model proposes, code decides. Thresholds, clarification, handoff and confirmation are deterministic policy, so a wrong or unsure model turns into a question or a human — not into a wrong action. The supervisor's explanation is assembled from the catalog, the percentages and the thresholds, so it always matches what the policy actually did.
• The catalog is data. Scenarios, slots, actions and reply templates are read from the starter-kit JSON; the LLM reads a hand-made index of the same catalog. A new scenario means a JSON entry and an index line — no code change and no retraining.
• The jury's utterances are not in the repo. dev_utterances.json is read only by the evaluators (lib/eval.ts, scripts/eval-mock.ts, core-llm/router.py --eval), never by a router at runtime.
• Honest about tuning. Both the mock's cue lists and the LLM prompt were refined against the dev set, so both dev-set scores are reported as in-sample (§9.1). The mock is labelled «Мок» everywhere and is not our answer to the case.

4.8 Repository map

.
├── frontend/                    Next.js 16 app — the product UI; runs end-to-end in mock mode
│   ├── src/app/                 routes: / (landing) · /call (client) · /admin (supervisor) · api/core-route (LLM) · api/stt + api/tts (voice)
│   ├── src/lib/contract.ts      frontend ⇄ backend event and trace contract (source of truth)
│   ├── src/lib/core-router.ts   adapter: the core-llm result → the UI's RouterDecision
│   ├── src/lib/store.ts         conversation store shared by /call and /admin
│   ├── src/lib/api.ts           mock ⇄ real switch, SSE reader, session stats
│   ├── src/lib/voice.ts         browser STT / TTS / audio recorder
│   ├── src/lib/eval.ts          port of data/evaluate.py for the /admin quality card
│   ├── src/lib/mock/            router.ts · engine.ts (policy + executor) · actions.ts (31 mock actions)
│   ├── src/components/          app/ (console) · landing/ · ui/ (coss ui) · block/ (ObsidianUI)
│   ├── src/data/                copies of ../data/*.json used by the browser
│   └── scripts/                 eval-mock.ts (dev set) · replay-dialogs.ts (sample dialogs) · test-core.cjs (npm test)
├── core-llm/                    LLM router core (Python, standard library only)
│   ├── prompt.md                system prompt: output format, multi-intent, system intents, context rules
│   ├── scenarios.index.txt      compact catalog index: meaning · ru cues · kk cues · ≠ boundaries
│   ├── router.py                Router class · CLI · --eval · --dialogs · --bench
│   ├── server.py                private HTTP adapter: GET /healthz, POST /api/route (port 8090)
│   └── test_server.py           unit tests for the adapter
├── voice/                       ElevenLabs voice service (Go): STT / TTS, /ws/voice gateway, phone, logs
│   ├── elevenlabs/              client: Scribe v2 Realtime + batch STT, TTS over HTTP and WebSockets
│   ├── speech/                  per-language model routing, on-disk phrase cache, LLM-delta chunker
│   ├── agent/ · brain/          real-time conversation engine · reply brains (backend / OpenRouter / echo)
│   ├── transport/ · deploy/     browser WebSocket, Asterisk AudioSocket, Twilio · phone setup
│   └── server/ · cmd/           HTTP routes (/api/voice/stt, /api/voice/tts, /api/voice/stats …) · binaries
├── backend/                     Go service (net/http): GET /healthz + /health, Dockerfile; turn API in progress
├── docker-compose.yml           web app, Go backend, PostgreSQL; profiles "llm" (core-llm), "voice" (ElevenLabs), "server" (both)
├── data/                        official starter kit (read-only) + our predictions_mock.json
├── docs/                        case, analysis, PRD, SPEC, API contract, tasks, pitch, progress, research, design
├── DESIGN.md                    design-system reference
├── THIRD_PARTY.md               every third-party component, model, dataset and API we use
├── .env.example                 backend variables, keyless defaults
└── AGENTS.md · CLAUDE.md        rules for AI coding agents working in this repo

---

5. Technologies

Layer	Technology	Version	Used for	Status
Frontend	Next.js (App Router), React, TypeScript	16.3.6 · 19.2.8 · 5	Landing, client screen, supervisor console; server routes /api/core-route, /api/stt, /api/tts	✅
UI	Tailwind CSS · coss ui on Base UI (@base-ui/react) · ObsidianUI blocks · lucide-react · GSAP	4 · 1.8 · — · — · 3.15	Design system, accessible primitives, icons, landing text stream	✅
Typography	Geist, Geist Mono via next/font (latin + cyrillic)	—	Text; mono only for numbers and IDs	✅
LLM router	Python 3 (standard library only) · OpenRouter · google/gemini-2.5-flash-lite by default (any model via ROUTER_MODEL)	—	Scenario choice: ID:PERCENT pairs, temperature=0, reasoning off, provider sorted by latency; a private HTTP adapter for the web app — §5.1	✅
LLM, Go backend	One OpenAI-compatible client: OpenAI (gpt-4.1-mini), NVIDIA Build (meta/llama-3.3-70b-instruct), OpenRouter / Groq / Gemini / Ollama via LLM_BASE_URL; mock	—	The same routing inside the turn API	🚧
Browser speech	Web Speech API (SpeechRecognition), speechSynthesis, MediaRecorder	built in	Keyless STT / TTS; audio capture	✅
Speech	ElevenLabs Scribe v2 (scribe_v2_realtime, language_code=kk — also transcribes Russian and mixed speech) · eleven_flash_v2_5 (RU TTS) · eleven_v3_conversational (KZ TTS)	—	Recognition and voice replies — §5.2	✅ web app («LLM») · streaming gateway in voice/
Voice service	Go · github.com/coder/websocket	1.26	ElevenLabs STT / TTS behind /api/voice/*, the /ws/voice gateway, phone transports, latency logs	✅
Speech, alternative	geko.sh Seta seta-kk-ru-v2 (STT) · Tokay tokay-kk-v1 (TTS)	—	KZ-first recognition with code-switching	📋
Backend	Go, standard net/http	1.26.5	API gateway and the turn pipeline	✅ health · 🚧 API
Storage	PostgreSQL	—	Turns, traces, statistics	📋
Evaluation	Python 3 — official data/evaluate.py; TypeScript scripts run with tsx	—	Dev-set accuracy, sample-dialog replay	✅
Tests	node:test + the TypeScript compiler (npm test) · Python unittest · go test with fake ElevenLabs servers	—	Core adapter, dialog engine, the core's HTTP adapter, the voice service	✅
Infrastructure	Docker Compose (images node:22-bookworm-slim, python:3.13-slim, Go for voice/) · a VPS with Nginx + Let's Encrypt for bagyt.plus	—	One-command run · hosting	✅ · ✅

Every third-party component, model, dataset and API with its license: THIRD_PARTY.md.

5.1 OpenRouter — the routing LLM

OpenRouter is the one gateway to the LLM, used in two places: the scenario router core-llm/ (behind «LLM» mode and the CLI) and the voice service's own brain (voice/brain/openrouter.go). One key, any model, provider routing and a cost report for every call.

Feature	How Bagyt uses it	Where
Any model behind one API	google/gemini-2.5-flash-lite by default; any OpenRouter model id through ROUTER_MODEL (router) or OPENROUTER_MODEL (voice service). 15+ models were compared on the same dev set with router.py --bench (§9.1)	core-llm/router.py
Provider routing	provider: {sort: "latency", allow_fallbacks: true} — the fastest healthy provider serves each call; ROUTER_PROVIDER_SORT=latency | throughput | price	core-llm/router.py
Model fallbacks	The voice brain sends models: [primary, …fallbacks] (OPENROUTER_FALLBACK_MODELS=openai/gpt-4o-mini,openai/gpt-4.1-nano), so a failing model does not drop the call	voice/brain/openrouter.go
Prompt caching	The system prompt (prompt.md + scenarios.index.txt, ~5,800 tokens) is byte-identical on every call, so ~75 % of it is served from the provider's cache; warmup() primes the connection and the cache at start	core-llm/router.py
Deterministic, tiny answers	temperature=0, max_tokens=64, reasoning off (reasoning: {enabled: false}, or ROUTER_REASONING=minimal | low); the model answers with ID:PERCENT pairs, about 6 tokens, so decoding takes tens of milliseconds	core-llm/router.py, core-llm/prompt.md
Cost and usage per call	usage: {include: true} returns prompt, completion and cached tokens and the cost in USD; --eval and --bench print them — about $0.05 for the whole 104-utterance dev set	core-llm/router.py
Robust transport	One keep-alive HTTPS connection per thread (no TLS handshake per call), one retry on a dropped socket, an X-Title header; if a provider rejects temperature or reasoning (HTTP 400), the router drops that parameter and retries	core-llm/router.py
Streaming replies	The voice brain streams over server-sent events: the model first writes a [[SC12|0.88]] header (scenario and confidence), then the spoken reply, which goes straight into TTS (temperature 0.3 for natural speech)	voice/brain/openrouter.go
Keys stay on the server	OPENROUTER_API_KEY lives in core-llm/.env (or the voice service's environment); the browser only talks to the same-origin /api/core-route	frontend/src/app/api/core-route/route.ts

Measured: scenario choice 457 ms p50 · 617 ms p95 end to end (reported by the router's author, §9.3); first token 369 ms for gemini-2.5-flash-lite against 705 ms for gpt-4o-mini, 794 ms for claude-haiku-4.5 and 1,238 ms for gpt-4.1-mini (§9.7).

5.2 ElevenLabs — speech in and out

ElevenLabs handles both ends of the conversation. The Go voice service voice/ (Compose profile voice) holds the key and the models; the web app reaches it through two same-origin routes, /api/stt and /api/tts, so the key never reaches the browser.

In the web app («LLM» mode, --profile voice):

Feature	Details
Speech recognition	Click the mic and speak: recording stops by itself after a pause (VAD) or on a second click, with live partial transcripts and a mic-level ring while you talk. The recording (webm / opus, up to 20 MiB) goes POST /api/stt → POST /api/voice/stt → Scribe v2 with language_code=kk; the answer is {text, language, ms, provider: "elevenlabs"} and the transcript appears in the conversation. ElevenLabs is the recognizer in «LLM» mode; in «Мок» the browser's Web Speech is, and its network errors switch to ElevenLabs by themselves
Voice replies	With sound on (the speaker icon), every reply is spoken by ElevenLabs: POST /api/tts → POST /api/voice/tts → mp3_22050_32. Russian → eleven_flash_v2_5, Kazakh → eleven_v3_conversational, English greetings → the Russian voice with English pronunciation (lang: en). Voice: Sarah (EXAVITQu4vr4xnSDxMaL), configurable per language
Audio player	Each reply keeps its MP3 for the conversation — play / pause, a seekable waveform computed from the recording, elapsed / duration and download — with the transcript below the player. Muting turns autoplay off but still creates the recording
Fallbacks	A service error shows a notice and falls back to browser speech; «Мок» always uses browser speech, so it needs no key (and has no downloadable recording)

In the voice service (voice/README.md):

Feature	Details
Streaming recognition	Scribe v2 Realtime over WebSocket: 20 ms audio frames while the caller talks, partial captions every ~100–200 ms, word timestamps. SpeechEnd() maps the last word back to wall-clock time, so latency is measured from the real end of speech. Batch Scribe v2 handles whole recordings — what /api/stt uses today
Languages	language_code=kk for Russian, Kazakh and mixed speech: auto-detect heard Kazakh as Turkish, and ru as a secondary language bent mixed phrases towards Russian spelling. Domain keyterms bias recognition: Saqta, ОГПО, КАСКО, ДМС, полис, сақтандыру
End of turn	Push-to-talk commit on the web — the final transcript 263–325 ms after release; server VAD with a 0.4 s silence window on the phone (VOICE_VAD_SILENCE_SECS)
Voice per language	Flash v2.5 (~75 ms model latency) over the stream-input WebSocket for Russian; v3 conversational over the text-to-dialogue WebSocket for Kazakh — the only realtime ElevenLabs model that speaks it, and as fast (first audio 208–221 ms)
Output formats	pcm_16000 (web), pcm_8000 (Asterisk / a Kazakh SIP number), ulaw_8000 (Twilio), mp3_22050_32 (the web player) — no resampling on our side
Latency tricks	The TTS socket opens on the first partial; a chunker sends the first clause with flush, so audio starts early; a keep-alive HTTP/2 transport with Warm(); fixed phrases (greeting, the «Секунду.» filler played at 1.5 s) cached on disk — no latency and no credits after the first run; speculative replies; barge-in (tts.clear)
Channels	Browser WebSocket /ws/voice (push-to-talk or VAD), Asterisk AudioSocket, Twilio Media Streams, and a voice test page at /
Brains	backend (the team's POST /api/turn), openrouter (a built-in Saqta agent: catalog, knowledge base and mock clients in the prompt), echo (keyless — what Compose runs, because the web app routes with core-llm)
Observability	A JSONL log per call with t_ms for every stage, optional WAV recording, /api/voice/events (live SSE), /api/voice/stats (p50 / p95 of end of speech → first audio, STT, LLM first text, TTS first audio), /api/voice/sessions
Tooling	go test ./... against fake ElevenLabs servers (no credits); go run ./cmd/voicedemo quota (characters left on the plan), voicedemo tts / stt demos that write their latencies to voice/demos/RESULTS.md; live smoke tests with VOICE_LIVE=1

Measured with real providers (§9.7): final transcript 263–325 ms after push-to-talk release; first audio 208–221 ms in Kazakh and 211–245 ms in Russian; 1.70–1.95 s from the end of speech to the reply's first audio live in phone-like VAD mode, ~1.1 s estimated for web push-to-talk. Smoke check through /api/stt on bundled synthetic recordings: RU payment → SC30 ✅, KK renewal → SC27 ✅, mixed → SC11 instead of SC12 ❌ because «кеше» (yesterday) was misrecognized (§9.4). The team key is on the ElevenLabs Creator plan (~128k characters a month).

---

6. Requirements

For	You need
Recommended — everything	Docker with Compose v2. Nothing else for «Мок»; an OpenRouter API key for «LLM»; an ElevenLabs API key for ElevenLabs voice
Without Docker	Node.js 22+ with npm (the version in the Dockerfile; Next.js 16 itself needs ≥ 20.9), Python 3.10+ for core-llm (standard library only) and Go 1.26+ for the voice service
Voice	A microphone; browsers allow it on localhost or HTTPS only. ElevenLabs recognition works in any modern browser; the keyless Web Speech recognizer needs Chrome or Edge on desktop. Text works everywhere
Evaluation scripts	Python 3 for data/evaluate.py; npx tsx (npx downloads tsx on first run)
Go backend	Go ≥ 1.26.5 (see backend/go.mod)
API keys	None for «Мок». «LLM» needs OpenRouter, plus ElevenLabs for voice; «Бэкенд» will need provider keys (§7.6)

---

7. Install and run

Current platform header contains only Call/Console navigation and the RU/ҚАЗ interface language switch. Interface language is saved locally and does not change speech recognition or reply language. Runtime mode/STT selectors and the evaluation card are no longer shown in the platform. Choose NEXT_PUBLIC_API_MODE=mock|core before building (for example, NEXT_PUBLIC_API_MODE=core docker compose --profile server up --build -d). Older screenshots/instructions mentioning header mode controls or the quality card describe the earlier UI; CLI evaluation commands remain available.

Server stack in Docker

Requires Docker Compose v2.24+. All server components now have images: backend (Go API skeleton), db (PostgreSQL 17), core-llm (Python router), and voice (ElevenLabs STT/TTS). The Go service currently exposes /healthz and the compatibility alias /health; its turn API and database persistence are not implemented yet. The working dialog uses frontend core mode.

For a fresh clone, copy .env.example to .env and core-llm/.env.example to core-llm/.env. Put ELEVENLABS_API_KEY in the root .env and OPENROUTER_API_KEY in core-llm/.env. Existing local env files should be edited, not overwritten.

# All server components, without starting frontend:
docker compose --profile server up --build -d --wait db backend core-llm voice

# Full application, including frontend in LLM mode:
NEXT_PUBLIC_API_MODE=core docker compose --profile server up --build -d --wait

# Status and logs:
docker compose --profile server ps
docker compose --profile server logs --tail=100 backend core-llm voice
curl --fail http://localhost:8080/healthz

# Stop the complete stack, preserving database files:
docker compose --profile server down

Backend is bound to 127.0.0.1:8080 (BACKEND_PORT overrides it). PostgreSQL, core-llm and voice are only reachable inside the Compose network; frontend proxies speech and routing requests. Health checks cover all four server services. Backend starts after PostgreSQL is ready. Database files live in the named postgres-data volume; down preserves them. POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB configure a new volume; if changed, update DATABASE_URL too. Existing volumes keep their initialized users/passwords.

Without keys, use the default docker compose up --build: frontend mock mode + Go health service + PostgreSQL. The server profile (or the separate llm and voice profiles) enables integrations requiring provider keys. Docker health checks verify running processes, not provider quota or credentials.

7.1 Quick start — keyless, about two minutes

git clone https://github.com/BAITC-Hacks/hack-9dd7b5f3-plus.git
cd hack-9dd7b5f3-plus
docker compose up --build

Open http://localhost:3000/admin (supervisor console, with the conversation on the left) or http://localhost:3000/call (the client's view). It is built in «Мок» mode — no keys, no accounts, no login. Port 3000 busy? FRONTEND_PORT=3200 docker compose up --build.

Without Docker:

cd hack-9dd7b5f3-plus/frontend
npm ci
npm run dev        # starts in «Мок» when there is no frontend/.env.local

Voice in «Мок» is the browser's own (Web Speech in Chrome / Edge, speechSynthesis). For ElevenLabs voice — also the way around browsers that can't reach Google's speech service (Arc, Brave, Yandex, restricted networks) — run the voice profile from §7.2. The recognizer follows the mode: the browser's Web Speech in «Мок», ElevenLabs in «LLM».

7.2 LLM mode — the real router in the web app

The ready .env files are in our Google Drive folder next to the demo video — copy core-llm/.env and the root .env from there, or create them from the examples below.

With Docker — the router and ElevenLabs voice:

cp core-llm/.env.example core-llm/.env      # put OPENROUTER_API_KEY there
cp .env.example .env                        # put ELEVENLABS_API_KEY there
NEXT_PUBLIC_API_MODE=core docker compose --profile llm --profile voice up --build
# PowerShell: $env:NEXT_PUBLIC_API_MODE='core'; docker compose --profile llm --profile voice up --build

The web app is built in «LLM» mode: ElevenLabs recognizes your speech — click the mic, speak, click again — and, with sound on (the speaker icon), speaks the replies. Without --profile voice, «LLM» mode still routes with the LLM and falls back to browser speech. The Python and voice services are reachable only inside the Docker network, and the keys never reach the browser. The mode is fixed at build time — rebuild to switch between «Мок» and «LLM».

Without Docker — three terminals:

# terminal 1 — repository root: the LLM router
cp core-llm/.env.example core-llm/.env      # put OPENROUTER_API_KEY there
python3 core-llm/server.py                  # 127.0.0.1:8090 · GET /healthz · POST /api/route

# terminal 2 — the ElevenLabs voice service (reads ELEVENLABS_API_KEY from the repo-root .env)
cd voice
VOICE_HTTP_ADDR=127.0.0.1:8091 VOICE_BRAIN=echo VOICE_GREETING=false VOICE_FILLER_AFTER_MS=0 go run ./cmd/voice

# terminal 3 — the web app
cd frontend
cp .env.local.example .env.local            # NEXT_PUBLIC_API_MODE=core, CORE_LLM_URL=…:8090, VOICE_TTS_URL / VOICE_STT_URL=…:8091
npm ci
npm run dev

If core-llm is down or the key is wrong, «LLM» mode shows an error — there is no silent fallback to the mock. If the voice service is down, replies fall back to browser speech with a notice.

7.3 Router CLI, tests and the Go backend

# the LLM router from the command line (needs core-llm/.env)
python3 core-llm/router.py "Кеше аварияға түстім, но я не виноват, виновник у вас застрахован"
python3 core-llm/router.py --eval           # dev set → evaluate.py metrics + predictions
python3 core-llm/router.py --dialogs        # dialogs_sample.json turn by turn, with context
python3 core-llm/router.py --bench google/gemini-2.5-flash-lite,openai/gpt-4.1-mini   # compare models

# tests — no keys needed
python3 -m unittest discover -s core-llm -p 'test_*.py'
(cd frontend && npm test && npm run lint && npm run build)
(cd voice && go test ./...)                 # fake ElevenLabs servers, no credits

# ElevenLabs tools (need ELEVENLABS_API_KEY)
(cd voice && go run ./cmd/voicedemo quota)  # characters left on the plan
(cd voice && go run ./cmd/voicedemo tts && go run ./cmd/voicedemo stt)   # RU / KZ / mixed demos + latency → voice/demos/RESULTS.md

# the Go backend
(cd backend && go run ./cmd/server)         # :8080 (override with PORT); GET /health → {"status":"ok"}

More router options (dialog context flags, the Python API): core-llm/README.md.

7.4 Real mode (frontend → Go backend) 🚧

Build with NEXT_PUBLIC_API_MODE=real and NEXT_PUBLIC_API_URL=http://localhost:8080. The backend has to implement docs/API_CONTRACT.md; ⏳ until the turn API is on main, this mode cannot create a session.

7.5 Deployed version

The VPS deployment uses https://bagyt.plus (37.151.92.185). Nginx terminates Let's Encrypt HTTPS and forwards to the frontend on 127.0.0.1:3080. The frontend calls core-llm:8090 and voice:8090 over the private Compose network; provider keys never go to the browser. Go backend health is on 127.0.0.1:8085/healthz. The Go service is still a skeleton; the working dialog routes through Python core + voice.

Server files:

• /opt/bagyt/current: active release (symlink into /opt/bagyt/releases/).
• /opt/bagyt/shared/app.env: root runtime settings and ElevenLabs key, mode 600.
• /opt/bagyt/shared/core.env: OpenRouter key, mode 600.
• Each release links .env and core-llm/.env to these private files.

Production settings in app.env are NEXT_PUBLIC_API_MODE=core, FRONTEND_PORT=127.0.0.1:3080, BACKEND_PORT=8085, CORS_ORIGINS=https://bagyt.plus, a generated POSTGRES_PASSWORD and the matching DATABASE_URL. Keep the existing DB password when updating a release. Runtime keys follow the env examples; do not commit the actual env files.

cd /opt/bagyt/current
docker compose -p bagyt -f docker-compose.yml -f deploy/compose.production.yml --profile server up --build -d --wait
docker compose -p bagyt -f docker-compose.yml -f deploy/compose.production.yml --profile server ps

deploy/compose.production.yml adds frontend restart and health checks. deploy/bagyt.plus.nginx.conf is the domain configuration. Install it into /etc/nginx/sites-available/bagyt.plus and link into sites-enabled after issuing the certificate with Certbot webroot /var/www/bagyt-acme. Certbot renews automatically; the server's deploy hook reloads Nginx after renewal. All five containers restart automatically.

Verification: open /call and /admin, allow microphone access, send a Russian or Kazakh request, then check the selected scenario and audio response. No login is required.

bagyt.plus — the live version. Locally: Docker Compose (§7.1), no login required.

7.6 Environment variables

One env file per component; each ships as a keyless example. Secrets live only in your local copies (all gitignored) and in the hosting provider's variables — never in the repo.

File	Copy from	Read by
.env	.env.example	Docker Compose variable substitution (FRONTEND_PORT, NEXT_PUBLIC_API_MODE), the ElevenLabs voice service (ELEVENLABS_* — voice/ also finds this file on its own) and the Go backend
core-llm/.env	core-llm/.env.example	core-llm/router.py and server.py; in Compose, the core-llm service's env_file
frontend/.env.local	frontend/.env.local.example	npm run dev (Next.js reads env files from frontend/, not from the repo root); the example starts in «LLM» mode
voice/.env (optional)	voice/.env.example	The voice service run on its own: every ElevenLabs, OpenRouter, VAD, filler, log and transport knob with its default

Variable	File	Default	Read on main today?	Purpose
NEXT_PUBLIC_API_MODE	.env (Compose build arg) or frontend/.env.local	mock (core in .env.local.example)	✅ lib/api.ts	Mode, fixed at build time: mock | core («LLM») | real (Go)
CORE_LLM_URL	frontend/.env.local	http://127.0.0.1:8090 (Compose: http://core-llm:8090)	✅ app/api/core-route (server only)	Where the Next.js proxy finds core-llm. Never prefix it — or any key — with NEXT_PUBLIC_
FRONTEND_PORT	.env	3000	✅ Compose	Host port of the web app
NEXT_PUBLIC_API_URL	frontend/.env.local	http://localhost:8080	✅ lib/api.ts	Go backend base URL for «Бэкенд» mode
VOICE_TTS_URL / VOICE_STT_URL	frontend/.env.local	http://127.0.0.1:8091 (Compose: http://voice:8090)	✅ app/api/tts, app/api/stt (server only)	Where the Next.js routes find the voice service; VOICE_STT_URL falls back to VOICE_TTS_URL
CORE_LLM_HOST / CORE_LLM_PORT	core-llm/.env	127.0.0.1 / 8090 (Compose: 0.0.0.0)	✅ core-llm/server.py	Bind address of the router's HTTP adapter
OPENROUTER_API_KEY	core-llm/.env	—	✅ core-llm/router.py	LLM router via OpenRouter
ROUTER_MODEL	core-llm/.env	google/gemini-2.5-flash-lite	✅	Model id on OpenRouter
ROUTER_PROVIDER_SORT	core-llm/.env	latency	✅	OpenRouter provider routing: latency | throughput | price
ROUTER_REASONING	core-llm/.env	off	✅	off | skip | minimal | low
ROUTER_THRESHOLD	core-llm/.env	60	✅	Confidence (%) from which a scenario counts as requested; below it → clarify
ROUTER_TIMEOUT	core-llm/.env	8	✅	Per-request HTTP timeout, seconds
ELEVENLABS_API_KEY	.env	—	✅ voice/	ElevenLabs STT and TTS — read only by the voice service
ELEVENLABS_VOICE_ID / ELEVENLABS_VOICE_ID_KK	.env	EXAVITQu4vr4xnSDxMaL (Sarah) / same as RU	✅ voice/	The voice for Russian / Kazakh replies
ELEVENLABS_TTS_MODEL_RU / ELEVENLABS_TTS_MODEL_KK	.env	eleven_flash_v2_5 / eleven_v3_conversational	✅ voice/	TTS model per reply language
ELEVENLABS_STT_LANGUAGE	.env (Compose sets kk)	kk	✅ voice/	Scribe language — kk handles Russian, Kazakh and mixed speech
ELEVENLABS_STT_KEYTERMS	voice/.env	Saqta,ОГПО,КАСКО,ДМС,полис,сақтандыру	✅ voice/	Domain words that bias recognition
VOICE_BRAIN	voice service	auto (Compose: echo)	✅ voice/	auto | backend | openrouter | echo — who writes replies when the voice service drives the call
OPENROUTER_MODEL / OPENROUTER_FALLBACK_MODELS	voice/.env	google/gemini-2.5-flash-lite / openai/gpt-4o-mini,openai/gpt-4.1-nano	✅ voice/brain	Model and fallbacks of the voice service's own OpenRouter brain
PORT	.env	8080	✅ cmd/server	HTTP port
LLM_PROVIDER	.env	mock	🚧	mock | openai | nvidia | openai_compatible
OPENAI_API_KEY / OPENAI_MODEL	.env	— / gpt-4.1-mini	🚧	OpenAI provider for the Go backend
NVIDIA_API_KEY / NVIDIA_MODEL	.env	— / meta/llama-3.3-70b-instruct	🚧	NVIDIA Build (NIM) provider
LLM_BASE_URL / LLM_API_KEY / LLM_MODEL	.env	—	🚧	Any OpenAI-compatible API (OpenRouter, Groq, Gemini, Ollama, …)
GEKO_API_KEY	.env	—	📋	geko.sh STT / TTS for KZ / RU
SPEKO_API_KEY	.env	—	📋 optional	Voice-provider router; not on the KZ / RU path
DATABASE_URL	.env	postgres://plus:plus@db:5432/plus?sslmode=disable	📋	PostgreSQL connection
CORS_ORIGINS	.env	http://localhost:3000	🚧	Origins allowed to call the API

7.7 Troubleshooting

Symptom	Fix
«Распознавание речи работает в Chrome или Edge…»	Use desktop Chrome or Edge — or just type: text runs the same pipeline
«Нет связи с сервисом распознавания…»	The browser can't reach Google's speech service; the app switches to ElevenLabs recognition. Run the voice profile with ELEVENLABS_API_KEY (§7.2) — or type
«ElevenLabs не смог озвучить ответ…» / «…распознать речь…»	Check ELEVENLABS_API_KEY, the voice ID and the quota left: cd voice && go run ./cmd/voicedemo quota
«Сервис озвучки недоступен…» / «Сервис распознавания недоступен…»	Start the voice service (--profile voice, or go run ./cmd/voice) and check VOICE_TTS_URL / VOICE_STT_URL
«Нет доступа к микрофону…»	Allow the microphone in the address bar; the page must be on localhost or HTTPS
«Ничего не расслышала…»	Click the mic, start speaking right away, pause to send
Kazakh speech comes out as Russian words	Switch the header language to ҚАЗ — it sets the recognizer too. Mixed phrases work best in «LLM» mode, where ElevenLabs Scribe with kk handles both (§13)
Kazakh answers are read with a Russian voice	In «Мок», browsers rarely ship a Kazakh voice, so it falls back to a Russian one. In «LLM» with the voice profile, ElevenLabs eleven_v3_conversational speaks Kazakh
«core-llm не отвечает…» in «LLM» mode	Start the router — docker compose --profile llm up --build, or python3 core-llm/server.py — and check CORE_LLM_URL
«Ошибка core-llm…» / «LLM недоступна…»	Check OPENROUTER_API_KEY and ROUTER_MODEL in core-llm/.env
Port 3000 is busy	FRONTEND_PORT=3200 docker compose up --build, or npm run dev -- -p 3001
The previous dialog interferes	Click «Новый диалог» — within one dialog the client stays identified and slots are remembered

---

8. How to verify

Prefer to watch? The demo video (3 min) is in our Google Drive folder. Prefer to click? bagyt.plus — open access, no login. Everything below also works locally.

8.1 Setup — 30 seconds

1. Run §7.1 and open http://localhost:3000/admin in Chrome.
2. The default build is «Мок» (the header shows «демо»). Turn sound on with the speaker icon next to the text field.
3. Click «Новый диалог» before each scenario below. Every step works by typing or by voice (click 🎤, speak, pause); the header's language switch sets the interface, speech recognition and reply language together.

8.2 Walkthrough — 10 scenarios

The expected results below were produced by running these exact inputs through the mock engine on main.

#	Say or type	You should see (mock mode)	Proves
1	«Хочу узнать статус моего заявления» → then the phone: «плюс семь семьсот один ноль ноль ноль ноль ноль ноль семь» (or +77010000007)	SC17 "Claim status" 0.96 → the robot asks for the phone → Sergey Popov identified (find_client, get_claim) → «По заявлению CL-500330: на рассмотрении…»	Voice in and out, identification, spoken numbers, a data-grounded answer
2	«Здравствуйте, я вчера оплатил, деньги списались, а заказ не подтвердился… а, и ещё, адрес доставки поменять надо» — the case's own example	Two scenarios: SC30 0.94 + SC29 0.82; policy run, SC29 waits on the stack; slots incident_date: 2026-09-30, contact_field: address; the robot promises to handle the second question too	Multi-intent, topic stack, slot extraction
3	Switch the language to ҚАЗ, then say or type «Сәлеметсіз бе, полисімнің мерзімі қашан бітеді?» → then the phone in Kazakh words: «плюс жеті жеті жүз бір нөл нөл нөл нөл нөл нөл бір»	Language KK; SC25 "Check policy validity" 0.90; the answer is in Kazakh: «Тексерейін. Телефон нөміріңізді айтып жіберіңізші.» → Arman Tulegenov → «SQ-OGPO-104501 полисі 2027-03-14 дейін жарамды.»	Kazakh in and out
4	«Кеше аварияға түстім, но я не виноват, виновник у вас застрахован»	Language RU+KK (mixed inside one phrase); SC12 "Claim as victim under culprit's OGPO" 0.90, while SC11 "Road accident just happened" is only an alternative (0.32); the robot asks for the culprit's plate	Code-switching, a scenario boundary
5	«Проверьте, действует ли мой полис, и ещё мне звонили якобы из страховой и просили код из смс»	SC38 "Fraud report" (urgent) is handled first although it was said second; SC25 is queued; anti-fraud advice	Urgent-first priority
6	«Хочу записаться к терапевту по ДМС» → +77010000002 → «Подождите, а какие клиники есть в Астане?»	SC21 → Aigerim Bekova → switch to SC23 (list_clinics: Saulet Medical, Nur Med Astana) → the robot returns on its own: «Теперь по вопросу «Запись к врачу по ДМС»…»	Topic switch and return
7	«Хочу продлить ОГПО» → +77010000003 → «Да»	renew_policy:preview + read-back «Проверю: полис SQ-OGPO-102850, стоимость 31 200 тенге… Всё верно? Это действие нельзя будет отменить.» → only after «Да»: renew_policy:execute, send_sms	No irreversible action without an explicit «да»
8	«Алло, я по поводу страховки» → «Ну там вопрос»	SYS_UNCLEAR → one clarifying question with two options → a second vague turn → a menu of the services the robot can help with (no automatic transfer); «Соедините с оператором» (SC37) → transfer_to_operator with a context summary	Asks instead of guessing; hands off to a human on request
9	«Можно у вас взять кредит на машину?»	SYS_OUT_OF_SCOPE → a polite refusal plus what the robot can help with	Out-of-scope detection
10	On /admin, after any turn: «JSON последней реплики»	The trace in the dataset README format — transcript, language, scenarios with confidence, alternatives, reason, slots, actions, latency_ms; «Скорость» next to it shows the per-stage waterfall against the 1.5 s line	The trace the case asks for

The official dev-set metrics — 97.1% for the mock, in-sample (§9.1) — are one command away: §8.4.

In «LLM» mode (NEXT_PUBLIC_API_MODE=core) — the product router; needs a running core-llm with a key (§7.2) or the deployed version. Expected results as stated by the router's author; ⏳ re-check at freeze:

#	Say or type	You should see
L1	«Хочу продлить ОГПО и добавить сына»	Two scenarios, SC27 renewal + SC04 add a driver; the robot acknowledges the second request and asks for identification; «Консоль» shows the model, the percentages, the alternatives, the core's verdict and the timings
L2	«А какие документы нужны при ДТП?»	A topic switch; the console shows the new scenario
L3	«Полисімнің мерзімін ұзартқым келеді»	Kazakh, routed by the same core
L4	python3 core-llm/router.py --eval	The dev set through the real router — 104 paid requests, metrics as in evaluate.py
CLI	python3 core-llm/router.py "Хочу продлить ОГПО и заодно добавить в него сына"	SC27:95 SC04:95 (core-llm/README.md)

ElevenLabs voice — «LLM» mode with --profile voice and ELEVENLABS_API_KEY (§7.2). Expected results as stated by the voice integration's author; ⏳ re-check at freeze:

#	Do	You should see
V1	Turn sound on (the speaker icon), click 🎤, say «Хочу продлить ОГПО и добавить сына», click 🎤 again	The transcript appears in the conversation, the LLM routes it (SC27 + SC04), and ElevenLabs speaks the reply; an audio player with a waveform, elapsed / duration and a download button sits above the reply's text
V2	Say «Полисімнің мерзімін ұзартқым келеді»	A Kazakh transcript (Scribe v2 with kk), and a Kazakh reply spoken by eleven_v3_conversational
V3	Mute with the speaker icon, send another phrase	No autoplay, but the recording is still created and playable

Greetings — any mode (verified in «Мок» on main):

#	Say or type	You should see
G1	«Здравствуйте» · «Сәлеметсіз бе» · “Hello”	SYS_GREETING: «Здравствуйте, чем могу помочь?» · «Сәлеметсіз бе, қалай көмектесе аламын?» · “Hello, how can I help you?” — the dialog state is untouched
G2	«Здравствуйте, хочу продлить полис»	SC27 (renewal), not a greeting — the robot asks for the phone

8.3 Test clients

Synthetic clients from data/mock_backend.json — say or type the phone when the robot asks.

Phone	Client	City	What they have
+77010000001	Arman Tulegenov	Almaty	OGPO SQ-OGPO-104501 and CASCO SQ-CASCO-204118 (to 2027-03-14); claim CL-500198 paid
+77010000002	Aigerim Bekova	Astana	DMS SQ-DMS-604220 (to 2026-12-31)
+77010000003	Yerlan Omarov	Shymkent	OGPO SQ-OGPO-102850 (to 2026-09-29); payment P-3001 charged, policy not issued
+77010000004	Natalia Smirnova	Almaty	Property SQ-PROP-404077; claim CL-500311 waiting for documents
+77010000007	Sergey Popov	Pavlodar	CASCO SQ-CASCO-204300 (to 2026-10-20); claim CL-500330 under review
+77010000010	Kamila Utepova	Atyrau	CASCO SQ-CASCO-204350 (to 2027-04-30)

All 11 clients: data/mock_backend.json or Appendix D of docs/CASE_ANALYSIS.md.

8.4 Reproduce the numbers from a terminal

# 1) official scorer on the committed mock predictions — needs only Python
python data/evaluate.py data/predictions_mock.json data/dev_utterances.json

# 2) regenerate the predictions from the current mock router, then score them
cd frontend
npx tsx scripts/eval-mock.ts
python ../data/evaluate.py ../data/predictions_mock.json ../data/dev_utterances.json

# 3) replay the 10 annotated sample dialogs through the mock engine (§9.2)
npx tsx scripts/replay-dialogs.ts

# 4) the LLM router on the dev set and on the dialogs (needs OPENROUTER_API_KEY in core-llm/.env)
cd ..
python3 core-llm/router.py --eval
python3 core-llm/router.py --dialogs

# 5) unit tests — no keys needed
python3 -m unittest discover -s core-llm -p 'test_*.py'
(cd frontend && npm test)

Use python3 where python is not on the path.

8.5 Case requirements → where to check them

Case requirement	Where you see it	Status
Must-have: voice interaction in the web — the jury speaks, the robot answers by voice	/call or /admin: mic → spoken reply (steps 1, 3, V1–V2)	✅ ElevenLabs STT + TTS in «LLM» · ✅ browser speech in «Мок»
Must-have: scenario selection on an LLM layer, not an encoder classifier	«LLM» mode in the web app → /api/core-route → core-llm/: prompt + catalog index → LLM → policy (§4.7)	✅ needs an OpenRouter key — ⏳ open on the deployed version
Must-have: correct selection on the jury's 10 utterances	Dev set: 104/104 for the LLM router (reported, in-sample), 97.1% for the mock (§9.1)	⏳ the jury's set
Must-have: trace panel after every utterance — scenario, reason, alternatives, per-stage timing	/admin: decision card, «Скорость», «JSON последней реплики»	✅
Must-have: Russian and Kazakh, including mixing inside a phrase	Steps 3, 4 and V2; ElevenLabs Scribe with kk for RU, KZ and mixed speech, Kazakh replies by eleven_v3_conversational	✅ (time markers in mixed speech can be misheard, §13)
Optional: context retention and return to an interrupted topic	Steps 2 and 6; core-llm --dialogs	✅
Optional: ask instead of guessing; hand off with context	Step 8	✅
Optional: parameter extraction from speech	Steps 1–3 (spoken phone, dates, address)	✅
Optional: the 500 ms target for scenario choice	LLM router p50 457 ms, 75 % of calls ≤ 500 ms (reported, §9.3)	✅ p50 · ⏳ in the web app
Optional: supervisor panel with error statistics	/admin session metrics; the dev-set error list via evaluate.py (§8.4)	✅ partly (per session)
Optional: streaming processing	Typed event stream in the web app; streaming STT and TTS over WebSockets in the v