Skip to content

feat(assemblyai): add streaming inactivity timeout option #1888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chasef07 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(assemblyai): add streaming inactivity timeout option #1888

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/assemblyai-inactivity-timeout.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-assemblyai': patch
---

Add AssemblyAI streaming `inactivityTimeout` support.
50 changes: 50 additions & 0 deletions plugins/assemblyai/src/stt.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,33 @@
// SPDX-License-Identifier: Apache-2.0
import { VAD } from '@livekit/agents-plugin-silero';
import { stt } from '@livekit/agents-plugins-test';
import { once } from 'node:events';
import type { AddressInfo } from 'node:net';
import { describe, expect, it } from 'vitest';
import { WebSocketServer } from 'ws';
import { STT } from './stt.js';

async function startWebSocketServer() {
const wss = new WebSocketServer({ host: '127.0.0.1', port: 0 });
await once(wss, 'listening');
const address = wss.address() as AddressInfo;
return { wss, baseUrl: `ws://127.0.0.1:${address.port}` };
}

async function closeWebSocketServer(wss: WebSocketServer): Promise<void> {
for (const client of wss.clients) client.close();
await new Promise<void>((resolve) => wss.close(() => resolve()));
}

async function waitUntil(predicate: () => boolean, timeoutMs = 1000): Promise<void> {
const startedAt = Date.now();
while (Date.now() - startedAt < timeoutMs) {
if (predicate()) return;
await new Promise((resolve) => setTimeout(resolve, 10));
}
throw new Error('timed out waiting for condition');
}

describe('AssemblyAI options', () => {
it('accepts u3-rt-pro-beta-1', () => {
const stt = new STT({ apiKey: 'test-key', speechModel: 'u3-rt-pro-beta-1' });
Expand Down Expand Up @@ -47,6 +71,32 @@ describe('AssemblyAI options', () => {
}),
).toThrow(/previousContextNTurns/);
});

it('forwards inactivity timeout to the streaming query', async () => {
const { wss, baseUrl } = await startWebSocketServer();
let requestUrl = '';

wss.on('connection', (_ws, req) => {
requestUrl = req.url ?? '';
});

try {
const stream = new STT({
apiKey: 'test-key',
baseUrl,
inactivityTimeout: 45,
}).stream();

await waitUntil(() => requestUrl !== '');
stream.close();

const url = new URL(`ws://127.0.0.1${requestUrl}`);
expect(url.pathname).toBe('/v3/ws');
expect(url.searchParams.get('inactivity_timeout')).toBe('45');
} finally {
await closeWebSocketServer(wss);
}
});
});

const hasAssemblyAIApiKey = Boolean(process.env.ASSEMBLYAI_API_KEY);
Expand Down
6 changes: 6 additions & 0 deletions plugins/assemblyai/src/stt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@ export interface STTOptions {
encoding: STTEncoding;
speechModel: STTModels;
languageDetection?: boolean;
/**
* Session inactivity timeout in seconds. AssemblyAI accepts integer values
* from 5 to 3600; when unset, no inactivity timeout is applied.
*/
inactivityTimeout?: number;
Comment on lines +69 to +72

@devin-ai-integration devin-ai-integration Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Inactivity timeout uses seconds but the plain-form name implies milliseconds per repo convention

The timeout field is named without the InS suffix (inactivityTimeout at plugins/assemblyai/src/stt.ts:72) yet documented as accepting seconds, so users familiar with the codebase convention will misinterpret the expected unit.

Impact: Users may pass a value like 45000 (thinking milliseconds) when the API actually expects 45 (seconds), causing the session to never time out or to receive an API validation error.

Repository time-unit convention and within-file inconsistency

CLAUDE.md line 155 states: "Time units: Use milliseconds for all time-based values by default. Only use seconds when the name explicitly ends with InS."

Line 182 reiterates: "Only use seconds as the unit if the variable name explicitly ends with InS (e.g. delayInS)"

Within the same STTOptions interface, other time fields follow the convention correctly:

  • bufferSizeMs at plugins/assemblyai/src/stt.ts:64 — explicit Ms suffix
  • minTurnSilence at plugins/assemblyai/src/stt.ts:75 — plain name, documented as "(ms)"
  • maxTurnSilence at plugins/assemblyai/src/stt.ts:77 — plain name, documented as "(ms)"
  • #speechDurationInS at plugins/assemblyai/src/stt.ts:209 — explicit InS suffix for seconds

The new inactivityTimeout at line 72 uses a plain name (implying ms) but the JSDoc at lines 69–71 says "in seconds" and the value is passed directly to the API at line 330 without conversion. To comply with the convention, the field should either be renamed to inactivityTimeoutInS or accept milliseconds and convert before sending.

Prompt for agents 
The field `inactivityTimeout` in `STTOptions` (plugins/assemblyai/src/stt.ts:72) is documented as accepting seconds but uses a plain name which, per CLAUDE.md time-unit convention, implies milliseconds. Two options to fix:

1. Rename the field to `inactivityTimeoutInS` throughout the codebase (stt.ts and stt.test.ts), keeping the value in seconds as-is.

2. Keep the plain name `inactivityTimeout`, change the JSDoc to say milliseconds, and divide by 1000 at the point it's sent to the API (line 330: `inactivity_timeout: this.#opts.inactivityTimeout !== undefined ? this.#opts.inactivityTimeout / 1000 : undefined`). Update the test value from 45 to 45000.

Option 1 is simpler and less error-prone since the AssemblyAI API natively accepts seconds. Files affected: plugins/assemblyai/src/stt.ts (interface + usage at line 330) and plugins/assemblyai/src/stt.test.ts (line 87, 95).
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@devin-ai-integration devin-ai-integration Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 ElevenLabs uses same naming pattern, suggesting an established (if incorrect) precedent

The ElevenLabs plugin (plugins/elevenlabs/src/tts.ts:76) also uses inactivityTimeout in plain form for a seconds value (default 180 at line 30). This suggests the convention violation is an established pattern in the codebase for API-passthrough timeout values. A reviewer may want to consider whether to fix both at once, or accept this as an intentional exception for external API parameters that natively use seconds.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

endOfTurnConfidenceThreshold?: number;
/** Minimum silence (ms) before a confident end-of-turn is finalized. */
minTurnSilence?: number;
Expand Down Expand Up @@ -322,6 +327,7 @@ export class SpeechStream extends stt.SpeechStream {
? JSON.stringify(this.#opts.keytermsPrompt)
: undefined,
language_detection: languageDetection,
inactivity_timeout: this.#opts.inactivityTimeout,

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop reconnecting after inactivity timeout closes

When inactivityTimeout is set and the stream has no audio/messages for that period, AssemblyAI closes the socket for the configured inactivity timeout; however run() treats any server-initiated close while the input is still open as a retryable error, so this parameter makes an idle long-lived stream immediately open another idle session with the same timeout instead of actually cleaning up. Please special-case the inactivity close (or defer reconnect until new audio / send KeepAlive, depending on the intended behavior).

Useful? React with 👍 / 👎.

prompt: this.#opts.prompt,
agent_context: this.#opts.agentContext,
previous_context_n_turns: this.#opts.previousContextNTurns,
Expand Down