Whisper

OpenAI’s Whisper is one of the best open-source speech models around. Out of the box, whisper-large-v3-turbo hits about 8% word error rate (WER) on general English tests like LibriSpeech. But point it at radiology reports, esports commentary, court audio, or factory SOPs and that number can spike to 30-50%. The model just hasn’t seen enough of those niche terms in training.

You can fix this. Fine-tuning Whisper on a small set of domain audio, as little as one to three hours, with LoRA adapters cuts domain-term WER by 30-60%. The full training run fits on a single consumer GPU with 12-16 GB of VRAM. It takes a couple of hours and yields an adapter file under 100 MB. Below is the full path from data prep to deployment.

You can build a fully local meeting transcriber on Linux. Capture system audio with PipeWire. Transcribe with Faster-Whisper on your GPU. Pipe the transcript to a local LLM through Ollama for structured summaries with names, decisions, and action items. The pipeline runs on 16GB of RAM and a mid-range NVIDIA GPU, and produces notes within seconds of the call ending. No data leaves your network.

Commercial services like Otter.ai and Fireflies.ai route your audio through their servers. If your meetings cover sensitive topics like product plans, HR, or legal reviews, that’s a non-starter. A local pipeline gives you the same structured output, and nothing leaves your building.

Home Assistant AI voice control with a local LLM as the brain is practical in 2026. No Amazon, no Google, no cloud. The Assist pipeline already handles the plumbing: wake word, speech-to-text, a conversation agent, and text-to-speech, all on your own hardware. Setting that up is the easy part. The hard part is picking a local model that calls Home Assistant’s tools without guessing. The loop also has to be fast, or it will never feel like a real assistant. This guide covers both: the 2026 stack, the models the community actually trusts, and the latency budget that makes it work.

Willow gives you sub-second local voice control for Home Assistant without sending your audio to the cloud. With an ESP32-S3 Box, you can build a private smart speaker that matches the speed of commercial assistants. Every spoken word stays inside your own network. This guide walks through the full setup: hardware, server deployment, firmware flashing, pipeline config, and the fixes for the most common problems.

Why Local Voice Control Is Worth It in 2026

Say “Hey Alexa” or “OK Google” and an audio clip travels from your home to a data center. There it gets transcribed by a third-party model, passes through an intent classifier, triggers an action, and returns a response. The whole trip usually takes under two seconds. That pipeline is impressive engineering. It is also a steady stream of your household’s spoken data flowing to Amazon and Google servers, where it is logged, reviewed by contractors, and used to train future models.