Hold Right Ctrl, talk, let go, and the cleaned text lands at your cursor. A Whisper model costs 1 to 2 seconds to load, so spawning a process per keypress would be unusable. linuxwhisp runs resident instead, holds two models warm, and does nothing until you press the key. Transcription is entirely local and audio never leaves the machine in any phase, which is the property the rest of the design exists to protect.
The hotkey
The obvious approach is a keyboard grab, and it is wrong. A core XGrabKey on Right
Ctrl consumes the key, which breaks Ctrl+C, Ctrl+V and every other right-hand chord system wide.
What works is XRecord, a passive tap on the X event stream, so the focused application still
receives the key normally while the daemon watches it.
That creates a problem worth being explicit about. XRecord is the X keylogging API, and it sees every keystroke on the system including passwords typed into other windows. So the discard sits above every other branch in the handler: anything that isn’t our keycode is dropped before it can be logged, buffered, or handed to another function. One key covers hold, tap-to-latch and double-tap, separated by timing rather than by carving out more keys.
Live text, and why it never gets injected
Whisper is not a streaming model. Re-transcribing a growing buffer produces text whose tail keeps changing, so a raw preview would rewrite itself constantly. LocalAgreement-2 fixes that: a word is confirmed once two consecutive passes agree on it, confirmed words render solid, everything after them renders dim. The subtle part is that agreement alone is not enough, because two passes over nearly identical short audio will happily agree on the same wrong guess and a confirmed word is never revisited. So no word ending within a second of the buffer’s edge can be confirmed at all. Without that guard, one benchmark sample went from 5.0% to 22.5% word error, permanently.
The live pass never contributes to the injected text. It is preview only. The transcript that ships comes from a second, larger model run over the complete audio after the key comes up, which is why a jittery preview can never corrupt the result.
The overlay
The status surface has to be visible and must never take focus, because if it accepts focus then
both your keystrokes and the injection go to it instead of the application you are dictating into.
It is a genuine X11 override_redirect window, unfocusable by construction rather than
by convention, with a runtime assertion so a Qt upgrade that silently changes it fails loudly
instead of breaking injection mysteriously.
It also exists to avoid what the commercial tools do. Typing raw text into the target window and then selecting it back to replace it with the cleaned version is why those tools are flaky, since it breaks against autocomplete, markdown renderers and editor auto-indent. Feedback happens in a surface linuxwhisp controls, and injection into the foreign application happens exactly once, with final text.
Cleanup, and the failure rule
Cleanup is a Haiku call over the text, never the audio, in one of two modes chosen from the focused
window’s WM_CLASS. Prompt mode strips filler and preserves identifiers, paths and
flags exactly. Prose mode handles punctuation and capitalisation.
The hard part is that people dictate messages to AI assistants, so a transcript reads like a request. Sent plainly, “this is a test of your implementation” gets answered, and the answer gets typed at your cursor. Three guards stop it: the transcript arrives inside delimiters described as a recording made before the model existed, anything under 16 characters skips the API entirely, and a response longer than twice the input is rejected as the model talking rather than cleaning. Every failure path injects the raw transcript rather than losing your words, and says so on the overlay.
What didn’t work
Four separate attempts at near-realtime output failed. Hard time cuts were 1.8× faster and pushed word error from 9.5% to 15.5%, with one sample going from 2.5% to 37.5%, because Whisper mangles the partial words on both sides of a mid-utterance seam. Cutting at detected silences was safe but almost never fired, since real dictation averages 1.6 usable silences per 20 seconds. Using the live agreement text as the final transcript cost 8 points of accuracy outright.
The tension is structural rather than a tuning problem. Confirm early and it is fast but wrong, confirm late enough to be accurate and there is no time left to save. A local cleanup model was cut for a related reason: 5 to 7 seconds against Haiku’s 1, and being memory bandwidth bound it would have turned the one idle second in the pipeline into the heaviest CPU event on the machine.
Stack
Python. faster-whisper for transcription, webrtcvad for the speech gate, sounddevice and PortAudio for capture, PyQt5 for the overlay and the correction panel, python-xlib and xdotool for the X11 work, and the Anthropic API for cleanup. 15 modules, 176 tests that run headless and offline, and exactly one module that knows the whole flow.