Can AI clean up your dictation without sending it to the cloud?

Yes — cleaning up dictated text (stripping filler words, fixing punctuation, breaking a rambling paragraph into readable sentences) can run entirely on your own machine. It doesn't take a frontier model. A quantized large language model with a few billion parameters, running through the same local-inference stack as your speech-to-text engine, is enough to turn rough spoken output into clean prose.

One indie developer proved the concrete shape of this: a macOS dictation app pairing a 0.6B Parakeet transcription model with a 3B Llama model for cleanup, both running on-device through Apple's MLX framework, with the two steps combined finishing in well under a second on a stock M-series Mac (Hacker News, "Show HN: local speech-to-text is shockingly fast on Apple Silicon"). Local rewrite is a solved engineering problem at small scale. So why does almost every dictation app that markets "AI formatting" still send your words to a server to get it?

Why does "local" dictation still send your text to the cloud?

Transcription and cleanup are two different jobs, and most apps only localize the first one. Wispr Flow is the clearest example of not localizing either: by its own description of how the product works, your audio is captured, sent to Wispr's cloud servers, processed by AI models — including third-party providers like OpenAI and Meta — and the result is sent back (wisprflow.ai; independently confirmed by a third-party review noting "voice processing happens in the cloud... including providers like OpenAI and Meta"). That's not a side effect — cloud processing is what makes its context-aware AI formatting possible in the first place.

The more interesting case is apps that market themselves as local-first but still go to the cloud at the cleanup step. Spokenly and OpenWhispr both run transcription on-device by default, then hand the rewrite step to a cloud API through "bring your own key" (BYOK): OpenWhispr's own docs list "GPT for AI agent" under its OpenAI BYOK option, and its comparison page is explicit that "the AI cleanup step can optionally use a cloud LLM API if you configure it, while the underlying speech-to-text always runs locally" (docs.openwhispr.com; openwhispr.com/compare/wisprflow). A side-by-side review of Spokenly puts it just as plainly: "Local by default, but the AI text cleanup features route through OpenAI/Anthropic APIs. If you use BYOK, your audio stays local but your text leaves your Mac for processing" — and notes that Spokenly's "fully-local mode has no AI cleanup" at all (getseam.app).

Which points at the actual gap: the apps that stay fully local today mostly do it by skipping cleanup, not by solving it locally. VoiceInk is described in that same comparison as "100% local, always... No cloud features at all" — but also as offering "no post-processing beyond basic filler word removal." A companion review of Seam vs. Spokenly lines the tradeoff up in one table: Seam is 100% local with no AI text cleanup; Spokenly has AI text cleanup (GPT-4, Claude) that sends text — not audio — to OpenAI or Anthropic's servers to get it (getseam.app/blog/seam-vs-spokenly). Local dictation and AI cleanup have been sold as a tradeoff you have to pick one side of. They aren't, technically — they're just rarely built together.

What can a local LLM actually do with dictated text?

The same Show HN dictation project lays out a concrete, working task list for its on-device cleanup model: removing filler words ("um," "uh") and stutters/repeats, converting spoken numbers, converting special characters, expanding acronyms ("A P I" → "API"), formatting spoken emails ("hi at example dot com" → "hi@example.com"), and converting spoken currency and time into digits ("two ninety nine" → "$2.99"; "three oh two" → "3:02") (Hacker News, getonit.ai). That's a real, shipped feature set — not a hypothetical one — running entirely through a local model.

None of it requires a giant model. This is the class of task — apply a known, bounded transformation to input text — that small open-weight models in the 3–4B parameter range, quantized to 4-bit formats like GGUF (for llama.cpp) or MLX, are routinely used for on consumer hardware; a recent academic benchmark of on-device LLMs specifically evaluated Qwen 2.5/3 and Llama 3 models at these sizes for exactly this kind of constrained, resource-limited deployment (arXiv:2505.15030). Rewriting a paragraph of dictated speech into clean prose is squarely inside that envelope — it's pattern transformation on text that's already in front of the model, with no need for broad world knowledge.

Where local rewrite still falls short of GPT-4-class models

The honest limits are worth stating plainly, because the same team that shipped local cleanup was the one to flag them: "small LLMs (3B, 1B) still make mistakes. They can hallucinate long, unrelated responses and occasionally repeat back a few-shot example." Their fix was defensive, not corrective — falling back to the raw, unedited transcript whenever the small model's output looked unreliable, which means "some 'ums' and 'ahs' still make it through" (Hacker News, getonit.ai).

That tradeoff shows up any time engineers swap a frontier model for a small local one on a language task: one detailed writeup of replacing GPT-4 with a local small language model for a document-processing pipeline reported the small model handling the bulk of well-defined cases reliably, while conceding "GPT-4 is genuinely better on ambiguous documents... anything requiring real inference" — and routing those harder cases to a separate review path rather than pretending the small model could cover them too (Towards Data Science). Applied to dictation: a local model is well-suited to fillers, punctuation, number formatting, and paragraph structure — narrow, repeatable jobs. Deep tonal rewrites, ambiguous phrasing, or content that needs real judgment calls are where a smaller on-device model is more likely to need a human's second look, the same way it does in any other domain.

How does a local rewrite pass actually work?

Mechanically, it's an extra step bolted onto the same pipeline that already runs local speech-to-text — see our breakdown of what GPU a local dictation app actually needs for the transcription half. The rewrite step sits immediately after transcription and before the text lands: a quantized language model, loaded once and kept warm, reads the raw transcript and returns a cleaned version, sharing the same GPU that just did the speech-to-text pass instead of opening a network connection. In the MLX-based project cited above, the combined pipeline — Parakeet transcription plus Llama 3B cleanup — landed at roughly 800ms p50 latency for a full sentence, versus under 500ms with the LLM step skipped entirely (Hacker News, getonit.ai). The added latency is the cost of the second model pass; the ceiling for what to fix first — the mess this pass is built to remove — is the same list we broke down in why filler words survive dictation in the first place.

What Inkvox is building for Pro

Inkvox's core dictation already runs this way for transcription: Whisper large-v3-turbo, quantized, on your own GPU through Vulkan — NVIDIA, AMD, or Intel, with a CPU fallback if you don't have one — transcribing a sentence in roughly 0.3–0.4 seconds on hardware like an RTX 3070, across 100+ languages. Zero bytes of audio uploaded, no account required, fully offline after the initial ~800 MB model download. See how it works.

The rewrite layer described in this article — the piece that cleans up fillers, reformats, and adjusts tone after transcription — is what we're building next, for Inkvox Pro: a local LLM running on that same GPU, doing that same class of work, still 100% local. It's in active development, not shipped yet, and we're not going to pretend otherwise — when it lands, it lands as a local pass, not a quiet trip through someone else's API. macOS support is planned to follow the same local-first model once the Windows beta hardens.

FAQ

Can dictation AI cleanup run locally instead of in the cloud?

Yes. A quantized large language model in the few-billion-parameter range — small enough to run on a consumer GPU next to a speech-to-text model — can strip filler words, fix punctuation, and reformat dictated text without any of it leaving the device. Independent local dictation projects have already shipped this pattern using small local models like Llama 3B.

Why do "local" dictation apps still send text to OpenAI or Anthropic?

Because transcription and cleanup are two separate steps, and most apps only localize the first one. Speech-to-text can run on-device, but the AI formatting or rewrite layer is frequently routed to a cloud LLM API — documented in apps like Spokenly and OpenWhispr, where the core transcription is local but the optional AI cleanup uses your own OpenAI or Anthropic API key.

Does a local LLM rewrite dictation as well as GPT-4?

Not on every task. A small quantized model handles narrow, well-defined cleanup — filler words, punctuation, number formatting, paragraph breaks — reliably. It's more prone to occasional hallucination or repetition than a frontier model, and developers who've shipped this pattern build in a fallback to the raw transcript when the small model's output looks unreliable.

What's the difference between local AI rewrite and cloud AI formatting like Wispr Flow?

Wispr Flow's AI formatting processes your speech on its cloud servers, through third-party AI providers, by its own account of how the product works. A local rewrite pass runs the equivalent cleanup step on your own GPU, using a model stored on your machine, so the dictated text never has to leave the device to get formatted.

The cleanup everyone wants from dictation — fewer fillers, better punctuation, a tone that fits the app you're typing into — doesn't require a round trip to someone else's server. It requires a small model, sitting on the same GPU that's already doing the transcription. Join the waitlist to try Inkvox, and to hear first when the local rewrite layer ships.