English · 한국어
Haan is a Korean full-duplex speech dialogue model: the Moshi architecture with its Helium backbone replaced by Qwen3-8B, built to test one claim —
multi-turn duplex behaviour can emerge without a single hour of multi-turn Korean audio.
Every other route to a full-duplex model in a new language ends at the same bottleneck: acquire multi-turn dialogue audio in the target language. J-Moshi did it with roughly 67,000 hours of Japanese. Haan has zero hours of Korean multi-turn audio, and that absence is not a limitation being worked around — it is the experiment.
The wager is that a duplex conversation decomposes into three abilities that can be sourced separately, and that the composition is what emerges:
| Ability | Where it comes from | Cost |
|---|---|---|
| What to say — Korean reasoning and dialogue | Qwen3-8B already has it, at the text level | free; only has to be preserved |
| How to say it — Korean articulation through an audio interface | single-utterance corpora (Zeroth-Korean, LibriTTS-R) | cheap and abundant |
| When to say it — turn-taking, yielding, overlap, backchannel | an authored timing corpus (Nunchi) that carries structure and nothing else | authored, 217 dialogues |
None of the three requires a recorded Korean multi-turn conversation. If duplex behaviour falls
out of their composition, the contribution is a pipeline stage — Base → Instruction Tuning → RL
→ full-duplex transfer — that can turn any instruction-tuned LLM into a speech agent
without collecting dialogue audio in its language. If it does not, the negative result is worth
reporting too, and the repository is deliberately written so that "the hypothesis failed" and
"the pipeline was broken" can be told apart (see
RISKS_AND_DIAGNOSTICS.md §9, an archaeology of the
defects that a 450-step pilot passed without complaint).
Haan (romanization; pronounced /haːn/ — Korean 한 held long)
- Long-vowel spelling of Korean 한. Visually and phonetically distinct from "하나 (Hana)".
- A one-syllable shape a non-Korean speaker can pronounce unaided: H + long-a + n.
- The lengthened spelling avoids the domestic ambiguity where "한 모델" is heard as "one model" rather than a model named 한.
One sound carries three characters and three ideas at once.
repeated misunderstandings, conversations cut short, resentment accumulating
What actually happened after Korean banks began routing consultation calls through AI. In Hana Bank's own 2024 customer survey, 52% of callers who reached the call-bot abandoned the consultation before finishing their business, and 53% were dissatisfied.1 Across AI contact centres, 60.8% of agents report that their working conditions got worse, daily complaint volume is up 17%, and the most common reason customers give for dissatisfaction is that the AI does not understand what they are asking (73.6%).2 The anger a short-memoried machine produces does not disappear; it is transferred to the human who takes the next call — one Hana Bank agent describes callers who, primed by voice-phishing warnings, panic and raise their voice the moment a synthetic voice asks for their name and date of birth.1 The pattern is familiar enough in Korea to have become comedy.3
Haan is the technology that undoes this 恨. A conversation where a customer does not explain the same thing three or four times; a structure where an agent does not apologise for a machine's forgetting.
the structural limit known as the "three-minute wall"
Today's commercial assistants — GPT-4o, Gemini, Qwen3-Omni — hold roughly three minutes of conversation. A complex financial consultation averages an hour. A Transformer's KV cache grows with the dialogue until it evicts the opening context.
Haan removes this 限 in the end — but that is a goal of the later series (V2+V3). TITANS MAE (Memory as Embedding) compacts past information into a fixed-size embedding space, so context across a session of hours is recalled at the same strength regardless of distance.
Where V1 stands: what runs today inherits Moshi's full-duplex structure, so its context memory is still Moshi's — minutes. Crossing the three-minute wall belongs to the TITANS transition in V2+V3; this stage's contribution is Korean turn-taking transfer.
an independent alternative to the US- and China-led ecosystems
TITANS is not yet a settled academic standard — which is exactly what makes an independent line still possible. Haan aims to be a Korean-optimized full-duplex consultation engine of its own, raising the technical self-sufficiency of the domestic consultation and call-centre industry rather than deepening a dependency on someone else's stack.
| Half | What it means |
|---|---|
| Hearing Always | The model listens continuously instead of gating the microphone shut while it speaks, so a change of intent reaches it the moment it happens rather than at the end of a turn. |
| Acting Naturally | Listening, speaking and — eventually — acting happen at once rather than in turns. Acting as real tool execution belongs to V4; what is built here is the listening-and-speaking half. |
"working conditions got worse after AI agents were introduced" — 66%4 "stress from customer complaints caused by AI errors increased" — 53%4
The customer anger produced by an incomplete turn-based AI, deployed to cut labour costs, has been passed down to human agents as emotional labour. Haan aims to break that cycle with technology.
- A psychological safety net for agents: lowering the share of angry customers, so agents can work with respect and concentrate on the work that needs a person.
- Business actually finished: beyond repeating an answer, the AI completes the task itself through real-time tool calling (V4's goal).
- Care for the digitally vulnerable: an interface where elderly and disabled callers, who struggle to adapt to the mechanical rhythm of turn-taking, can speak as comfortably as they would with a person.
This is not simply an efficiency gain. It aims to be a demonstration of warm technology — breaking the technical cycle that charged the cost of degraded service quality to the agent.
theRiverLethe — named for the river of forgetting in Greek myth — is the sibling library that holds the TITANS memory implementation. The two projects meet on purpose: Haan builds the full-duplex half here, and then imports theRiverLethe to integrate TITANS, which is how 限 actually gets crossed. Neither repository is the other's predecessor; they are the two halves of one system, and this one is written first because timing has to work before memory is worth having.
Hence the package name, project_amnesty, and the line the whole idea rests on: cross Lethe,
the river of forgetting, and arrive at Haan — the memory that undoes the grievance. Amnesty
reads two ways at once, pardon and amnesia, and Haan is the pardon reached through memory
rather than through forgetting.
Haan — a Korean full-duplex voice consultation AI that remembers past the three-minute wall (限), and uses that memory to undo the grievance (恨) accumulated between callers and the agents who answer them, built independently (韓).
That sentence is the destination, not the present tense. Unlimited memory belongs to V2+V3; what exists in this repository today is the Moshi-based Korean full-duplex transfer described above — a distinction this README keeps everywhere, because collapsing it is how a research plan starts reading as a product claim.
flowchart TB
UIN["user audio in — 24 kHz"] --> MIMI
SPREV["Haan's own audio, previous frames"] --> MIMI
MIMI["Mimi codec (frozen)<br/>split-RVQ · 12.5 Hz · 1 semantic + 7 acoustic"]
MIMI -->|"8 codes x 2 streams"| EMB
TXT["text channel — inner monologue<br/>Qwen3 ChatML + PAD/EPAD stream tokens"] --> EMB
EMB["shared audio embedding tables (8)<br/>+ FiLM role modulation, self / user"] --> TEMP
TEMP["Temporal Transformer<br/>Qwen3-8B backbone"] --> Z["frame context z_s"]
Z --> THEAD["text head → next text token"]
Z --> DEPTH["Depth Transformer (shared, warm-started from Moshi)<br/>batch = 2 · self and user in parallel · 8 steps each"]
DEPTH --> CODES["8 audio codes per stream"]
CODES --> DEC["Mimi decoder → speech out"]
- The codec is frozen. Speech enters and leaves as Mimi's split-RVQ tokens at 12.5 Hz — one WavLM-distilled semantic codebook alongside seven acoustic ones. Teacher and student, past and future versions all share the same codebook space; nothing about Mimi is trained here.
- Qwen3-8B replaces Helium as the Temporal Transformer. Moshi's RQ-Transformer keeps its shape; only the large backbone is swapped. Because Qwen3 already reads and writes Korean, Japanese and English, the remaining problem shrinks from teach it a language to teach it to realize a language it already knows through an audio interface.
- Shared audio embeddings + FiLM role tokens. Original Moshi keeps 16 fully separate
embedding tables (8 self, 8 user); train only the speaking direction in Korean and the
user-side tables never see a Korean gradient. Haan shares one set of 8 tables and marks the
role by modulation instead:
h = γ[role] ⊙ h + β[role]. A plain additive role vector was ruled out by measurement — on the released Moshi weights a constant offset explains only 1.76% (semantic) and 7.82% (acoustic) of the self/user difference; the rest is per-code. - One shared Depth Transformer, run at batch 2. Self and user codebooks are predicted in parallel, 8 steps each, rather than PersonaPlex's sequential 16 — roughly twice as fast, and closer to Moshi's own factorization. Live inference drops the user branch entirely.
- The text channel keeps Qwen3's instruct format. The inner monologue stays native ChatML
with
<think>and<tool_call>intact and rendered non-vocal in post-processing, so the backbone's agent behaviour survives the graft. Turn boundaries are carried by Moshi-native stream tokens (PAD/EPAD), never by chat turn markers. - Warm start. The Depth body, its output heads and the codebook embeddings are lifted from Moshi unchanged (same frozen Mimi, same cardinality 2048). The audio input embeddings are copied from Moshi's user-side tables, because only those were trained on a multi-speaker distribution and carry no single-actor voice bias.
- Voice is supplied in context, not in weights. Timbre comes from a reference clip prefixed onto the agent audio channel — an independent implementation of the hybrid-system-prompt technique, using none of PersonaPlex's weights or data.
Full detail, including the discarded alternatives and why they were discarded, is in
ARCHITECTURE.md.
| Source | What it teaches | Notes |
|---|---|---|
Zeroth-Korean (kresnik/zeroth_korean) |
Korean articulation, both directions | used as ko_tts (speak) and ko_asr (listen) — the same rows recycled |
LibriTTS-R (blabble-io/libritts_r) |
English multi-speaker realization | en_tts; keeps the audio interface broad while Korean content is thin |
Nunchi (data/raw/nunchi_dialogues.json) |
timing only — when to start, yield, overlap | 217 authored dialogues (202 train / 15 probe). Both audio channels are silence-bank codes and loss is on the stream tokens alone, so no words are learned from it. Word durations are sampled from measured speech-rate lognormals; inter-turn gaps from the Stivers et al. (2009) cross-language turn-gap prior — mode ≈ 200 ms, with genuine mass below zero for barge-in |
| Text anchors (ko / en / ja) | retention of the backbone's multilingual text ability | a constant 5% of every batch, a counterweight to catastrophic forgetting rather than an objective |
Nunchi (눈치) is the Korean sense of reading a room — knowing when to speak and when to hold a silence. That is precisely and exclusively what the corpus trains.
What is not here matters as much:
- No multi-turn Korean dialogue audio. Zero hours. This is the hypothesis.
- No knowledge-distillation teacher. The design documents describe an English multi-turn KD path with the original Moshi as teacher; the stage-1 run as it actually stands uses no teacher at all — turn structure comes from the authored Nunchi corpus instead. The docs still carry the KD design; treat it as designed, not as running.
- No non-commercial corpora or weights. The weights are intended for an Apache-2.0 release, so NC-licensed material was excluded on purpose — including the J-Moshi family (CC BY-NC 4.0) and PersonaPlex's weights and data.
Detail in DATA_STRATEGY.md.
Stage 1 is a single continuous mix schedule keyed on the cumulative optimizer step, so it
survives --resume and holds its final weights past the last anchor. Realization comes first —
the audio interface has to exist before timing has anything to land on — and then the timing
share climbs:
| step | ko_tts |
en_tts |
ko_asr |
nunchi |
+ text_anchor |
|---|---|---|---|---|---|
| 0 | 0.45 | 0.35 | 0.10 | 0.10 | constant 0.05 |
| 5,000 | 0.40 | 0.25 | 0.10 | 0.25 | constant 0.05 |
| 20,000 → 45,000 | 0.35 | 0.20 | 0.10 | 0.35 | constant 0.05 |
The timing share is constrained never to fall below 0.05: it is the hypothesis under test, and it must not silently drop out of a schedule.
This is pipeline 1 of a two-pipeline plan. Because Nunchi audio is silence and the content corpora are single utterances, the model is never shown a sample where real audio and real turn structure co-occur. Pipeline 2 (stage 2, planned) closes that gap by rollout: mid-training, prompts cut from Nunchi-style dialogues are continued in simulation mode and the model's own audio, laid under a real turn structure, is trained on as ordinary teacher-forced data. Its data mix is deliberately left unconfigured until the stage-1 probes report — a guessed schedule that "looks configured but isn't" is exactly the failure the trainer refuses to allow.
Beyond that the curriculum plans an optional acoustic-prosody graft, a Japanese data-efficiency
sweep (0h / 1h / 10h / 100h against J-Moshi's ~67,000h) and a full ablation program. All of it is
design; none of it has run. See TRAINING_CURRICULUM.md.
The name and the persona hold across the whole series; the technical substance splits in two, and this README follows that split everywhere.
| What runs here today (V1) | What the programme ends at | |
|---|---|---|
| Architecture | Moshi-derived full duplex — frozen Mimi codec, Temporal + Depth Transformers | TITANS family, MAE (Memory as Embedding) |
| Backbone | Qwen3-8B in place of Helium | decided with the final design |
| Core problem | make Korean full duplex emerge by transferring turn-taking, without Korean multi-turn data | unlimited context (crossing the three-minute wall) and real-time tool calling |
| Context and memory | inherits Moshi's horizon — minutes | O(1) memory, long sessions held internally |
| Written up in | ARCHITECTURE, TRAINING_CURRICULUM, DATA_STRATEGY, RISKS_AND_DIAGNOSTICS |
RESEARCH_ROADMAP, and a final design document still to be written |
What V1 is actually reaching for, axis by axis:
| Axis | V1 |
|---|---|
| Listening and speaking at once | Half duplex cannot start inferring until the user stops. Moshi's paired self/user streams are predicted in parallel, so the model speaks while it is still listening. |
| Korean turn-taking, emergent | Silence, interruption and backchannel timing are transferred rather than recorded: an authored timing corpus supplies structure, single-utterance corpora supply articulation. |
| The cost of teaching a language, removed | Qwen3-8B already knows Korean; only the audio modality is grafted on, so "teach it the language" is skipped by swapping the backbone. |
| Context reach | Moshi's, which is minutes. Long-session memory is V2+V3's problem, not this one's. |
| Agent format preserved | The inner monologue keeps Qwen3's ChatML with <think> and <tool_call> intact and renders those spans silently, so the backbone's agent behaviour survives the graft. This preserves the format and the judgement; executing a tool in real time is V4. |
The research programme is planned as a series, each version removing one structural limit of full-duplex dialogue models. Nothing in the table below has shipped. V1 is the version this repository implements, and its stage-1 training run is in progress; every other row is a design on paper.
| Version | Meant to add | Still cannot | Status |
|---|---|---|---|
| V1 — Haan | Korean full-duplex speech: a model that speaks Korean at all in a duplex loop, with turn-taking learned structurally rather than gated by a VAD | more than one simultaneous speaker; context beyond Moshi's horizon (minutes); real-time tool execution; RVQ quantization artefacts and the codebook's language bias remain | training in progress — stage 1 live, no working duplex model yet |
| V1.5 — MultiHaan | several speakers at once, and telling them apart: a role axis added to RoPE and FiLM extended from 2 channels to N, initialized from V1's weights | still requires explicit per-speaker channel separation, so it still needs clean separated audio; still RVQ; still bounded context | planned |
| V2+V3 — NaturalHaan + RetentiveHaan | human-sounding speech and long sessions: remove RVQ entirely (continuous 1-D autoencoder predicting waveform amplitudes) to drop quantization artefacts and the English-phoneme bias that the roadmap identifies as the real blocker for multilingual conversational speech; replace explicit channel separation with waveform summation, which finally makes naturally-mixed audio usable as training data; absorb the resulting sequence-length explosion with TITANS MAC memory | tool calling remains impossible — channel interaction is implicit in the summation, so text cannot steer the audio channels | planned |
| V4 — AgenticHaan | acting while speaking: per-channel MAG memory gated by the text channel, so a <tool_call> flips the audio channel into filler mode instantly, plus inference-time sharding that runs the reasoning and audio shards as genuinely parallel processes synchronised through shared memory |
— endpoint of the current plan | planned |
Read end to end, the series converges on one target: a TITANS-based any-to-any full-duplex multimodal system, whose five axes are what the versions above are dismantling one at a time — listening and speaking at once through channel separation; unlimited context through Memory-as-Embedding, holding attention at O(1) with roughly three times the memory efficiency of a KV cache and learning during inference via test-time training; reacting to interruptions and changes of intent through a surprise metric and a forget gate that commit what matters to memory immediately; generating action while speaking, an independent channel calling a tool so the answer and the API call finish together; and doing all of it inside 100 ms end to end, which is where a reply stops feeling like a wait. All five are what this repository is being built toward; none of them is claimed by V1, the version training today.
Two things worth stating plainly. First, the memory ambition in the name (限, the "three-minute wall") lives in V2+V3 and V4, not in V1: what runs today inherits Moshi's context horizon and makes no claim to have crossed it. Second, the roadmap assigns each version a target venue between September 2026 and May 2027; none has been submitted.
Source: RESEARCH_ROADMAP.md.
- Stage 1 is running on 3× RTX PRO 5000 Blackwell 48GB under a custom FSDP2 loop — 90k tokens/step (token budget 6000 × grad-accum 5 × world size 3), lr 2e-5, bf16 reduce, ZeRO-3 on the backbone blocks, torchao 8-bit AdamW. Measured 19–20 s/step; 45,000 steps is about 10 days. As of 2026-08-02 the run is past step 1,250.
- A 450-step pilot ran earlier on 4×A100-80GB and is written up in
REPORT.md. Its numbers are pilot numbers at 1% of the trajectory, not results: the timing stream CE fell 17.9 → 1.38, content audio CE 8.6–10.5 → 4.6, grad norm 875 → 13, and role separation began to move off zero. The audio at that point was pre-linguistic; what the pilot established is that the pipeline runs and does not collapse. - The pilot also surfaced a real warning — Korean and Japanese retention perplexity drifted up sharply as the learning rate approached peak. In stage 1 the retention gate is deliberately advisory: it records the drift curve but does not stop the run, because stage 1 asks one question (does Korean duplex speech emerge?) and text retention is a property of the final artefact, not a precondition for learning to speak.
- No duplex model has shipped. There are no released weights and no benchmark results.
Dependencies are managed with uv. Use uv add / uv sync,
not uv pip install, so that uv.lock stays the source of truth.
git clone https://github.com/latentforge/haan && cd haan
uv sync --extra personaplex --extra dev| Extra | Contents |
|---|---|
| (base) | torch, torchaudio, transformers, datasets, bitsandbytes, accelerate, torchao, kernels, soundfile, scipy, matplotlib, wandb |
personaplex |
sentencepiece, protobuf, pyloudnorm — tokenizer conversion and voice-prompt loudness normalization |
dev |
notebook tooling (ipykernel, jupyter-client, ipywidgets, ipywebrtc, nbconvert, nbformat) |
transformers is pinned to a fork carrying the Moshi patches (feat/personaplex), and
torch/torchaudio come from the PyTorch cu128 index rather than PyPI — the host GPUs are
sm_120.
Note
No system FFmpeg is required. Audio rows are pulled from the hub undecoded and read through
soundfile, whose wheel bundles its own libsndfile; the torchcodec decode path that
datasets would otherwise use is bypassed on purpose.
Verify the install:
uv run python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count())"Everything runs through one dispatcher. The recipe — batch shape, precision, NCCL settings — is defined once inside it, so a manual launch, a milestone restart and a supervisor restart all go through the same definition and an intervention cannot apply to only some paths.
bash scripts/haan.sh train # launch stage-1 training on 3 GPUs
bash scripts/haan.sh gate step_3000 # CPU-only ko/ja/en retention gate; advisory, never stops the run
bash scripts/haan.sh milestone 500 # cooperative stop -> consolidate -> probes + listening + TTS -> relaunch
bash scripts/haan.sh tts <ckpt-dir> [step] # TTS-conditioned listening probe on a consolidated bf16 checkpoint
bash scripts/haan.sh supervise # stall watchdog: frozen log + live ranks -> forensics -> restart
bash scripts/haan.sh usage # the above, from the sourceExtra arguments to train land after the recipe and therefore override it. HAAN_DRY_RUN=1
prints the command instead of running it, and HAAN_RUN_ID (default stage1) selects the wandb
run every subcommand joins. Logs go to logs/stage1.log, checkpoints to trainer_output/stage1/.
The tts probe is worth calling out: self-play listening asks the model to invent content,
timing and both speakers at once, so it sounds like babble long after articulation works. The TTS
probe instead hands it the training task — voice prompt plus a frame-aligned transcript — and
asks only for audio codes. If that is unintelligible, the failure is articulation and nothing
else.
| Document | Contents |
|---|---|
PROJECT.md |
overview, expected contributions, related work (J-Moshi lineage and the wider full-duplex field), compute budget |
docs/ARCHITECTURE.md |
backbone swap, embeddings and role tokens, Depth Transformer sharing, voice prompts, instruction template, role-separation roadmap |
docs/TRAINING_CURRICULUM.md |
phase curriculum, joint loss, LoRA decisions, kernel notes, and the measured stage-1 infrastructure optimization gate |
docs/DATA_STRATEGY.md |
what "single-turn" is allowed to mean, synthesis pipeline, bidirectional reuse, voice prompts |
docs/RISKS_AND_DIAGNOSTICS.md |
failure mechanisms and how each is diagnosed, plus §9's archaeology of defects found during bring-up |
docs/RESEARCH_ROADMAP.md |
the version series above, in full |
docs/references/ |
the papers this design is built on or measured against |
REPORT.md |
the stage-1 pilot report (450 steps) |
Those documents are written in Korean, and they live on the main branch — the release
branch, which is what you get by default, ships the code and this README pair only. The links
above therefore point at main explicitly. This README and its Korean counterpart are the
bilingual entry point to both.
Code in this repository is released under the Apache License 2.0. Model weights are intended for release under the same terms, which is why non-commercial corpora and weights were excluded from the data strategy from the start.
Footnotes
-
경향신문, "[단독] AI 도입 콜센터, 월급도 깎았다···대체 1순위? '자르기 쉬워 잘랐을 뿐'", 2026-04-15 — figures from Hana Bank's 2024 customer survey. ↩ ↩2
-
한국노총 중앙연구원 survey of AICC workplaces, reported in 쿠키뉴스, 2026-06-19; see also 연합뉴스, 2026-03-29, and 소비자가만드는신문's case study of ARS/chatbot gatekeeping, 2026-04-21. ↩
-
Survey of call-centre agents, reported by 연합뉴스, 2026-03-29. The Korean labour federation's research institute found the same shape at AICC workplaces: 60.8% of agents report worse working conditions and daily complaint volume up 17% (쿠키뉴스, 2026-06-19). ↩ ↩2