Guide shows how to build an offline Python voice assistant using Whisper

Build a Voice Assistant with Python and Whisper

Build a Voice Assistant with Python and Whisper Imagine hearing your computer respond to your voice with the same nuance and clarity as a human, all running locally on your own machine without sending a single byte of data to the cloud. That’s the power of building a Voice Assistant with Python and OpenAI’s Whisper model. While cloud-based assistants like Siri or Alexa are convenient, they come with privacy trade-offs and latency issues. By combining Whisper’s state-of-the-art speech-to-text capabilities with Python’s flexibility, you can create a private, fast, and customizable voice interface that answers your questions, controls your scripts, or even just chats with you—right from your terminal. Let’s get your hands dirty and build one from scratch today. Why Whisper and Python? OpenAI’s Whisper is a transformer-based model trained on 680,000 hours of multilingual and multitask supervised data. Unlike older speech recognition tools that struggled with background noise or specific accents, Whisper delivers high accuracy across diverse environments [5]. It’s open-source, meaning you can run it locally, ensuring your conversations stay private. Python is the natural partner for this task. With libraries like SpeechRecognition, pyttsx3, and openai-whisper (or its faster variant faster-whisper), you can stitch together the entire pipeline: capturing audio, transcribing it, processing the text with an LLM, and speaking the response back [6][7]. Setting Up Your Environment Before writing code, you need the right tools. Start by creating a dedicated project directory and a virtual environment to keep dependencies clean. mkdir voice-assistant && cd voice-assistant python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Next, install the core speech processing stack. While openai-whisper works well, faster-whisper is often preferred for production due to its optimized performance and lower memory usage [2][5]. pip install SpeechRecognition==3.10.4 \ faster-whisper \ pyttsx3==2.98 \ python-dotenv==1.0.1 You’ll also need an API key if you plan to send the transcribed text to an LLM like OpenAI’s GPT or Anthropic’s Claude. Store this securely in a .env file to avoid hardcoding secrets: # .env ANTHROPIC_API_KEY=sk-ant-api03-your-key-here OPENAI_API_KEY=your-openai-key-here The Core Logic: Speech-to-Text and Text-to-Speech The heart of your assistant is the loop: Listen → Transcribe → Process → Speak. Let’s break down the first two steps. Capturing and Transcribing Audio We’ll use SpeechRecognition to capture microphone input and faster-whisper to transcribe it. The recognizer object handles the audio stream, while the whisper model converts audio waves into text. import speech_recognition as sr from faster_whisper import Whisper import os from dotenv import load_dotenv # Load API keys load_dotenv() # Initialize Whisper model (small.en is fast and accurate for English) model = Whisper("small.en") # Initialize Speech Recognition recognizer = sr.Recognizer() def listen_for_prompt(): with sr.Microphone() as source: print("Listening... (Say something)") recognizer.adjust_for_ambient_noise(source, duration=0.5) audio = recognizer.listen(source, timeout=5) # Convert audio to WAV file for Whisper audio_path = "prompt.wav" with open(audio_path, "wb") as f: f.write(audio.get_wav_data()) # Transcribe with Whisper segments, _ = model.transcribe(audio_path) text = "".join([segment.text for segment in segments]) print(f"You said: {text}") return text This function waits for you to speak, saves the audio to a temporary file, and runs it through Whisper. The small.en model is a great starting point—it’s lightweight (about 50MB) and transcribes 5 seconds of audio in under 0.5 seconds on modern hardware [6]. Speaking the Response Back Once you have text, you need to turn it into speech. pyttsx3 is a simple, offline text-to-speech engine that works without external APIs. import pyttsx3 engine = pyttsx3.init() def speak(text): print(f"Assistant: {text}") engine.say(text) engine.runAndWait() Putting It All Together: The Assistant Loop Now, let’s combine transcription and speech into a working loop. For this example, we’ll use a simple hardcoded response. In a real project, you’d send the transcribed text to an LLM API for dynamic answers [3][7]. def main(): speak("Hello! I'm your local voice assistant. What can I do for you?") while True: user_input = listen_for_prompt() if not user_input: continue # Simple logic: if user says "stop", exit if "stop" in user_input.lower(): speak("Goodbye!") break # Placeholder for LLM integration response = f"I heard you say: '{user_input}'. How can I help with that?" speak(response) if __name__ == "__main__": main() Run this script, and you’ll have a working voice assistant that listens, transcribes, and responds. The beauty here is that everything runs locally—no cloud dependencies for the speech parts, and you can swap the LLM logic to use any provider you prefer. Enhancing Your Assistant Once the basics are working, you can level up with these practical additions: Wake Word Detection: Instead of listening constantly, detect a specific phrase like “Hey Assistant” to activate the loop [2]. This saves CPU and reduces false triggers. Context Management: Pass previous conversations to your LLM so the assistant remembers context. This creates a more natural, conversational experience [3]. Custom Commands: Map specific phrases to system actions (e.g., “Open browser” → subprocess.run(["google-chrome"])). Faster Models: If you need speed, try tiny.en or base.en models. They’re less accurate but faster for real-time applications [6]. Privacy and Performance Considerations Running Whisper locally means your audio never leaves your machine, a critical feature for privacy-conscious users [5]. However, be aware that larger models (like large-v3) require significant RAM and GPU resources. For most desktop use cases, small.en or base.en offers the best balance of speed and accuracy. If you’re on a low-power device, consider using faster-whisper with quantized models, which can reduce memory usage by up to 50% while maintaining accuracy [2]. Start Building Today You now have a fully functional, private voice assistant built with Python and Whisper. The code is simple, the dependencies are open-source, and the results are immediate. Don’t just stop at transcription—integrate an LLM, add wake word detection, or connect it to your home automation system. The best part? You can do all of this today without waiting for a cloud service to approve your API key. Clone the logic, tweak the models, and make it yours. Ready to go further? Try adding a wake word or integrating Anthropic’s Claude for smarter responses. Share your project on Dev.to and let the community see what you built. The future of voice interaction is private, fast, and entirely in your hands. If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming! Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.

2 hours ago

Build a Voice Assistant with Python and Whisper

Build a Voice Assistant with Python and Whisper Imagine hearing your computer whisper back a response to your question, not through a robotic script, but with the fluid clarity of a human conversation. That’s the power of combining Python with OpenAI’s Whisper model: you’re no longer just coding a script; you’re building a voice assistant that listens, understands, and speaks back. And the best part? You can build a working prototype in under an hour, right on your laptop. Voice assistants are everywhere, from smart speakers to car dashboards, but most of them rely on cloud services that can be slow, expensive, or privacy-invasive. By using Whisper—a state-of-the-art speech-to-text model that runs locally—you gain full control over your assistant’s behavior, data, and latency. This guide walks you through building a private, offline-capable voice assistant using Python, Whisper, and a text-to-speech engine, so you can start experimenting today. Why Whisper and Python? Whisper is OpenAI’s general-purpose speech recognition model. It’s trained on diverse audio data and supports multiple languages, making it far more robust than older tools like SpeechRecognition’s built-in Google API. Unlike many cloud-based alternatives, Whisper can run locally, which means: No API fees for transcription Zero latency for network roundtrips Full privacy: your audio never leaves your machine Customizability: you can tweak the model, add wake words, or integrate with any LLM Python is the ideal language for this because of its rich ecosystem: openai-whisper for transcription, pyttsx3 for speech output, and SpeechRecognition for microphone input. Setting Up Your Environment Before writing code, you need the right tools. Create a project directory and set up a virtual environment to keep dependencies isolated: mkdir voice-assistant && cd voice-assistant python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate Now install the core packages: pip install openai-whisper SpeechRecognition pyttsx3 python-dotenv If you run into installation issues with openai-whisper (common on some systems), swap it for the faster, more compatible faster-whisper: pip install faster-whisper Create a .env file to store your API keys securely (though Whisper itself doesn’t require one for local use): # .env OPENAI_API_KEY=your-key-here # Optional, only if you use GPT later Building the Core: Speech-to-Text with Whisper The heart of your assistant is the ability to convert spoken words into text. Whisper handles this elegantly. Here’s a minimal function that loads the model and transcribes an audio file: import whisper # Load the Whisper model (choose "base" for speed, "large" for accuracy) model = whisper.load_model("base") def transcribe_audio(audio_path: str) -> str: result = model.transcribe(audio_path) return result["text"] This function takes an audio file path (e.g., recorded.wav) and returns the transcribed text. The "base" model is fast (0.5s for 5s audio on an M2 MacBook) and lightweight (74MB), while "large" offers higher accuracy but requires more resources [4]. Capturing Voice Input To capture audio from your microphone, use the SpeechRecognition library. It provides a simple interface to record and save audio: import speech_recognition as sr import wave def record_audio(filename: str = "input.wav", duration: int = 5): recognizer = sr.Recognizer() microphone = sr.Microphone() with microphone as source: print("Listening...") recognizer.adjust_for_ambient_noise(source, duration=1) audio = recognizer.listen(source, timeout=duration) # Save audio as WAV with wave.open(filename, 'wb') as f: f.write(audio.get_wav_data()) print(f"Saved audio to {filename}") return filename This function records 5 seconds of audio, saves it as input.wav, and returns the file path for transcription. Turning Text Back into Speech Once Whisper transcribes your voice, you need to respond. pyttsx3 is a cross-platform text-to-speech library that works offline: import pyttsx3 def speak(text: str): engine = pyttsx3.init() engine.say(text) engine.runAndWait() This function speaks the given text using your system’s default voice. You can customize speed, voice, and language by adjusting engine properties. Putting It All Together: The Full Assistant Now, combine everything into a working voice assistant. Here’s a complete script that records, transcribes, and responds: import whisper import speech_recognition as sr import wave import pyttsx3 # Initialize Whisper model = whisper.load_model("base") def transcribe_audio(audio_path: str) -> str: result = model.transcribe(audio_path) return result["text"] def record_audio(filename: str = "input.wav", duration: int = 5): recognizer = sr.Recognizer() microphone = sr.Microphone() with microphone as source: print("Listening...") recognizer.adjust_for_ambient_noise(source, duration=1) audio = recognizer.listen(source, timeout=duration) with wave.open(filename, 'wb') as f: f.write(audio.get_wav_data()) print(f"Saved audio to {filename}") return filename def speak(text: str): engine = pyttsx3.init() engine.say(text) engine.runAndWait() def main(): audio_file = record_audio() text = transcribe_audio(audio_file) print(f"You said: {text}") # Simple response logic if "hello" in text.lower(): response = "Hello! How can I help you today?" elif "time" in text.lower(): import datetime response = f"The current time is {datetime.datetime.now().strftime('%H:%M')}" else: response = "I'm not sure I understood that. Try saying 'hello' or 'time'." print(f"Assistant: {response}") speak(response) if __name__ == "__main__": main() Run this script, speak into your microphone, and watch your assistant respond. It’s a working prototype you can expand with wake words, LLM integration, or GUIs. Next Steps: Making It Smarter This basic assistant is functional, but you can make it powerful by integrating it with an LLM like GPT-4 or a local model like Llama 3. Instead of hardcoded responses, send the transcribed text to an API and return the AI’s reply. You can also: Add a wake word (e.g., “Hey Assistant”) using SpeechRecognition’s keyword detection Build a web interface with Streamlit for recording and playback [3] Deploy it as a desktop app with PyQt or Tkinter Use Faster-Whisper for real-time streaming transcription [2] Start Building Today You now have everything you need to build a voice assistant that listens, understands, and speaks—without relying on cloud services. The code above is ready to run, and the architecture is flexible enough to grow with your ideas. Your challenge: Modify the script to respond to a custom command like “What’s the weather?” by fetching real-time data. Share your version on Dev.to and tag it with #voiceassistant or #whisper. Let’s build the future of human-computer interaction together—one voice command at a time. If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming! Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.

1 day ago