Voice Input and Calls

Loop uses local recognition and speech on the computer running its backend. Voice controls add text to the existing conversation and retain its permission checks, tool results, queue and history. The browser never uses cloud speech recognition or synthesis.

Usage

Open Settings → Voice and follow the three setup steps:

Choose voice features: select editable voice input or input with local spoken replies.
Install voice components: use the Download icon button beside the title to install all components. Any missing component keeps this action visible; only a complete installation changes it to the Uninstall icon button. The same row provides cancellation; download progress appears only in each package ring. Four evenly spaced horizontal package markers show waiting circles with package names below and detached separator lines, inner rings filling clockwise from the top with centered percentages for active downloads (a dash when the total is unknown), with a highlight repeatedly tracing the filled arc (static under reduced-motion preferences), green checks for installed packages and a red cross for the failed package; hover or focus the cross to read that package’s error; hovering identifies the package and its state. Hover or focus the info icon beside each package name to inspect that package’s version, sizes and license. Package names stay on one line with ellipsis and show the full name on hover. Loading starts when the command is sent; completion comes from backend resource checks. Errors stay in this step with manual retry, and status refresh recovers background work after a lost command response.
Test voice: select a microphone and test recognition after the selected resources are ready. A nonempty final transcript marks this step complete; partial speech and voice previews do not. Stopping a successful test retains its completion, while changing microphone or resource readiness resets it. The successful check is saved in this browser for the selected feature profile, microphone and resource versions, so closing Settings or reloading the page retains completion. Changing the profile or microphone, managing components, or observing unavailable or changed resources requires a fresh check.

Management is available only on the host computer; paired phones can select and test components already installed there. Missing published engine assets or unsupported platforms remain unavailable rather than silently falling back to a different speech service. Downloads support cancellation, without a separate pause control.

Component sizes distinguish the download archive, the installed file size required even before installation, and bytes currently installed. Temporary download and extraction space is additional.

Use Verify and repair for an installed profile to recheck file hashes and replace damaged downloadable components. A locally imported engine may require another local import when its files are damaged. Updates become available when the installed activation differs from the available catalog.

Uninstall also clears managed partial downloads and damaged installations. Resources are stored under the backend's Loop data directory (~/.loop/voice by default), never inside the PWA. The current prepared ASR/VAD models require about 167.3 MiB of download; adding bilingual spoken replies adds about 123.8 MiB. The platform engine and temporary installation space are additional, and Settings reports reviewed sizes when published assets are available.

The locally validated macOS ARM64 engine brings the complete profile to about 299.7 MiB of download and 384.5 MiB expanded. These are measurements of prepared local artifacts; published versions may differ.

Choose Test microphone to inspect recognition in session-independent test mode without requiring a chat session or sending an Agent prompt. Its separate sampling row hides the capsule until microphone preparation completes and listening begins. Starting disables the microphone button; connecting and finalizing show a spinner around it. During preparation only the microphone spinner is visible. The capsule contains Discard, a waveform derived from live PCM energy that shifts left as samples arrive, and Stop; it has no Send action or visible status caption. A countdown outside the capsule on its right appears at 30S only after recording starts and decreases while sampling; its fixed-width slot remains reserved while its text is hidden before recording, while connecting and after recording ends. Each sample automatically stops 30 seconds after listening first starts; connecting time is excluded and status changes do not reset this limit. Intermediate phrase transcription keeps recording, waveform feedback and Stop available. During recording, the microphone spinner follows local audio energy and stays active through quiet speech and brief pauses using a lower continuation threshold, and clears after 1.2 seconds without detected speech; connection and finalization also retain their spinner. Stop drains queued microphone frames and the last partial PCM frame, then waits for backend transcription completion before showing all collected final phrases. While waiting, the sampling controls remain visible and Stop stays disabled. Completion keeps the collected final text visible in the same row with a Retry action, marking the test complete only when nonempty. Retry clears the previous result and starts a fresh recording and transcription on the same prepared connection, without restarting the microphone or engine. During recording, the leftmost X cancels capture, clears the sample and hides the capsule; Retry appears only beside the completed result. The input row keeps the same height before, during and after recording; long results scroll within the fixed-height text area. Clicking the microphone again also starts a fresh sampling cycle. Retry waits for the previous recognition boundary before clearing buffered phrases so old text cannot enter the new sample. The speech-preview control remains on its own row and is disabled during sampling. Test speech plays a short local sample when speech resources are installed, mutes microphone input during playback, then closes the connection and releases capture. A spinner surrounds the preview icon during connection and synthesis. Once connected, the sample text appears to its right; its highlight advances using the actual audio playback clock, approximating position within the text without word timestamps. Preview loading, completion and failures do not change microphone-test state or Step 3 completion. Completed microphone tests retain their prepared connection with capture paused for Retry; disabling the test, leaving Settings or starting speech preview closes it. The controls prevent overlapping use of the shared local engine; preview does not verify recognition. Both require an explicit browser microphone permission and a secure origin.

Select a microphone to use it for the next call. Device names refresh after microphone permission is granted; opening settings alone never requests recording access. The selection is stored in this browser, and an unavailable device falls back to the system default.

Both session entry buttons become available once the required components are ready for the selected profile. Step 3 is an optional microphone and speech test; its verification result does not gate entry. Hovering over a disabled entry points to Settings → Voice for details. Loading or failed resource checks also keep entry disabled. An active recording can still be finished, and a call can still be ended.

The composer toolbar sizes voice controls to their contents with a 4px gap between Voice input and Call, an 8px gap before Send and matching button footprints and gives Send and the orb the same footprint without extra horizontal padding. Audio animation stays inside the orb without outward expansion. It places two icon-only voice controls immediately before Send, in this order: Voice input → Call → Send.

Voice input, the microphone icon to the left of Call and Send, remains visible whether the draft is empty, typed or transcribed, allowing another recording after Stop. New dictation appends to the existing draft. Draft whitespace is preserved. While recording, sampling controls replace the entire bottom toolbar with the full-width waveform, while an audio-reactive orb occupies the rightmost Send position, leaving the editable draft above in place. The toolbar shows a dense live waveform with smaller dots, Discard, Stop and reserved countdown. Discard and Stop use red icons in Light mode and softer coral-red icons in Dark mode. Interim text stays buffered. Choose Stop or wait for the 60-second limit to end capture; pending transcription continues until the backend confirms that queued audio and the last phrase have finished, then fills the editable draft. Intermediate final phrases do not close the connection. A finalization failure or timeout reports an error instead of inserting an incomplete result. Use the usual Send button afterward. Discard drops this recording and preserves the existing draft and attachments. Manual typing pauses voice input. Attachments and selected Skills remain in the draft.
Call, the waveform icon immediately to the left of Send, replaces the entire bottom toolbar with full-width waveform controls and a Stop button that ends the call; an animated audio-reactive orb appears at the rightmost Send position; the left side contains icon controls for microphone mute, spoken-reply pause and steering the next utterance, with hover titles and pressed states. The lower call panel and duplicate End call button are removed. A paused call uses a Resume icon. Correction is selected by default when starting or resuming a call and turns off after the next submitted utterance. During initial connection, a locally synthesized melody loops until readiness, cancellation or failure; the looping wave animation also covers reconnection. Reconnection after readiness stays silent. Tone nodes disconnect immediately when stopped. The connection melody uses direct Web Audio output, isolated from the speech media stream. The melody starts once microphone audio preparation provides a running audio context; browser audio permissions still apply. The animation respects reduced-motion preferences. The call keeps listening and submits final utterances automatically. The typed draft, attachments and selected Skills stay separate. Ordinary requests use the existing task queue. Choose Use next utterance to correct direction to submit the next final utterance as steering at the current tool batch boundary.

Call controls occupy the full composer toolbar before the rightmost orb during a continuous call; there is no separate lower call panel or submission-status text row such as “Sent to Agent”. Both toolbar actions share one voice connection. Switching sessions closes that connection. During a call, microphone mute and spoken-reply pause have independent toggles; the red Stop button ends the call. Task cancellation remains available through explicit voice commands. Speaking stops playback immediately and keeps listening. The explicit phrases stop task, cancel task, 停止任务, 停止执行 and 取消任务 cancel the current Agent task. They do not undo completed changes. The screen remains available for messages, code, tool output and approval cards; speech does not approve tool requests.

Say resume task, 继续任务, 恢复任务 or 继续执行 to resume an available stopped task. Say 调整要求：…, 纠正方向：… or correct direction: … to submit the nonempty remainder as steering. Only final transcripts are interpreted as commands.

Editing an existing queued message pauses automatic sending. It cannot withdraw an utterance that the Agent has already accepted. Unmute after editing to continue the call. New utterances never replace a queue item being edited. Existing request IDs and stream IDs protect voice submissions against uncertain delivery; the ordinary Check and retry action applies if acceptance is unknown.

Lifecycle and Recovery

Microphone capture and playback start from an explicit click or tap. The browser requests mono audio with echo cancellation and noise suppression, resamples its actual capture rate to 16 kHz PCM, and shows call status independently from Agent task status. Browser audio processing and physical echo performance vary by device.

The browser also finalizes active speech after 1.2 seconds of silence. Finishing voice input waits for the backend to acknowledge completion of queued audio and final transcription before closing. A 65-second deadline reports failure rather than inserting incomplete text. Spoken replies begin at completed sentences in the streaming draft and retain the unfinished tail until completion. Narration plays one segment while synthesizing at most one segment ahead, retaining remaining text in order; user interruption or pausing reading clears that backlog.

Transient socket interruptions reconnect to the same live call with a fresh one-time ticket, retrying transient failures up to five times within 30 seconds. Previous audio and partial utterances are discarded. Completed speech is not replayed, and commands are not automatically resent. If the backend or call lease expires, choose Continue voice to start again; server restarts do not restore queued tasks.

Backgrounding, locking the phone, losing the network or a system microphone interruption pauses the foreground call. Navigating away also closes microphone capture and clears the live transcript; restoring the page from browser history keeps the call paused until Continue voice is chosen. Mute and spoken-reply pause requested during connection also apply when startup finishes. Switching chats ends the previous foreground call; accepted Agent tasks continue in their original session. Hanging up does not stop a task.

Privacy and Limits

Concurrent status checks share Release discovery rather than returning an unpublished engine while the manifest is still loading. Settings retries unavailable engine discovery every five seconds; the backend bounds remote retry frequency to once per minute. A valid installed activation remains usable when remote discovery fails.

Microphone capture and backend initialization prepare concurrently. Chat dictation and microphone tests request recognition only, so installed TTS files are neither read for integrity verification nor loaded for these calls. Continuous calls and Test speech retain synthesis. Every selected engine/model file still receives full integrity verification before use; each recording creates a fresh engine process. Failed or cancelled microphone preparation closes any backend call already created.

Audio remains in bounded processing buffers and is not saved as a recording. Final text is stored through the usual conversation history and sent to the configured Agent, which may be cloud hosted. A phone sends audio to the computer through its paired connection; the current tunnel service terminates transport TLS, so local inference does not mean audio stays on the phone.

Wake words, continuous background listening and operating-system voice integration are not provided. Mobile microphone permissions, audio-route changes, speaker echo and foreground recovery require real iOS/Android browser and installed-PWA checks. Model and engine availability is reported by the backend; the UI does not imply that release assets have been published.

Both surfaces map microphone RMS levels continuously to waveform height, suppressing levels below 0.002 and reaching the unchanged 28-unit peak at 0.3 RMS instead of 0.05, so ordinary and louder speech remain visually distinct. Both surfaces share recording controls, countdown and the input-session lifecycle for waveform sampling, speech feedback, transcript buffering, finalization and cancellation. Settings hides the waveform capsule, Discard, Stop and countdown until microphone preparation completes and listening begins; only the microphone spinner is shown during preparation. The capsule stays visible during finalization, then persistent completed text and Retry replace it without changing the row height, including for an empty result. Settings accepts final transcripts with a 30-second limit; chat retains editable dictation with a 60-second limit.

Source and Tests

Submit actions, call entry, voice controls, call toolbar, call actions and their colocated tests.
Composer integration, submission queue and their colocated tests.
Settings, profile step, resource step, test step, microphone sampler, shared sampling controls, composer sampling, live waveform, step layout, resource actions, package progress and their colocated tests.
PCM resampling, PCM utilities, playback, narration, narration tracking and socket transport, each with colocated tests.
Backend contract, chat input, approvals and mobile/PWA.

Component downloads and installation run concurrently. A failed package does not stop other packages; the operation waits for all workers and only activates after every required package and the engine probe succeed. Retry verifies and reuses completed packages. Cancellation stops all downloads and waits for cleanup before allowing another operation. Progress sums bytes across packages.

The feature dropdown shows each profile name with its description underneath inside the option. The closed dropdown shows only the selected name. Arrow keys navigate options, Enter selects and Escape closes.

Microphone and feature choices share the same dropdown option layout, selected checkmark and keyboard navigation.

Stopping installation enters a paused state without an installation error. Download stays disabled and a Resume icon continues retained partial downloads after cleanup.

Speaker echo handling

Microphone capture requests browser echo cancellation, noise suppression and automatic gain control. Local speech uses a MediaStream-backed audio element rather than direct Web Audio speaker output, allowing the browser media playback path to provide an echo reference. Capture remains active during speech for interruption; no transcript similarity filter drops repeated user words. Playback is started from the initiating gesture, and its tracks and media element are released on cancellation, failure or hangup. Browser support and acoustic conditions still affect cancellation: verify speaker-only playback causes no submissions, then verify user interruption during playback at several speaker volumes. Automated tests cover routing and cleanup, not acoustic effectiveness. No recordings are saved by this change.

Call narration reads assistant text as streaming sentences complete, including intermediate execution messages, in ordered segments of at most 110 UTF-16 code units. Long sentences are split rather than skipped, and queued segments are retained until played or explicitly interrupted. Code blocks and URLs remain omitted. Tool transitions retain brief factual announcements. Connection or reading pauses do not mark unseen updates as already narrated.