Build a Fast & Scalable Voice Assistant with Claude in Python

🏢 Enterprise AI Consulting

Get dedicated help specific to your use case and for your hardware and software choices.

TLDR: Add voice to Claude AI and build a Claude voice chatbot in Python using fully on-device speech processing. Unlike cloud-based solutions that rely on external servers, this approach creates fast, scalable, and reliable voice interactions that run entirely locally across multiple platforms.

Claude's consumer interface includes Voice Mode, but developers need custom implementations for their applications. This guide demonstrates building a Claude Voice Assistant in Python with Picovoice's on-device voice AI platform, achieving lower latency than Claude Voice Mode and offering hands-free activation with local speech processing. Building a Claude Voice assistant requires three key components:

Wake Word Detection powered by Porcupine Wake Word to activate hands-free listening.
Speech-to-Text (STT) powered by Cheetah Streaming Speech-to-Text to convert voice input into text in real time.
Text-to-Speech (TTS) Text-to-Speech (TTS) powered by Orca Streaming Text-to-Speech to generate Claude’s responses into natural speech.

Each engine runs locally without sending data to remote servers, eliminating network latency and giving developers' full control over the voice user interface.

What You'll Build:

Wake word detection for on-device voice activation
Streaming speech recognition that transcribes speech in real-time
Claude API integration that sends text and receives responses
Text-to-speech synthesis that generates audio without external calls

What You'll Need:

Python 3.9+ installed on your development machine
Working microphone and audio output device
Picovoice AccessKey from the Picovoice Console
Claude API key from the Claude Console

Looking to integrate voice with other AI chatbots? Check out our guides to build ChatGPT voice Assistant and Perplexity voice Assistant.

Train a Custom Wake Word for Claude Voice Assistant

Sign up for a Picovoice Console account and navigate to the Porcupine page.
Enter your wake phrase such as "Hey Claude" and test it using the microphone button.
Click "Train", select the target platform, and download the .ppn model file.

For tips on designing an effective wake word, review the choosing a wake word guide.

Set Up the Python Environment

Install all required Python SDKs and dependencies with a single terminal command:

Porcupine Wake Word Python SDK: pvporcupine
Cheetah Streaming Speech-to-Text Python SDK: pvcheetah
Orca Text-to-Speech Python SDK: pvorca
Picovoice Python Recorder library: pvrecorder
Picovoice Python Speaker library: pvspeaker
Anthropic Python library: anthropic — used for Claude API integration.

pip install pvporcupine pvcheetah pvorca pvrecorder pvspeaker anthropic

System Requirements: This tutorial works on Windows, macOS, Linux, and Raspberry Pi using Python 3.9+, microphone and speakers.

Add Wake Word Activation to Claude

The following snippet captures audio from your default microphone and detects your custom wake word locally:

import pvporcupine
from pvrecorder import PvRecorder

ACCESS_KEY = "${ACCESS_KEY}"
KEYWORD_PATH = "${KEYWORD_PATH}"

porcupine = pvporcupine.create(access_key=ACCESS_KEY, keyword_paths=[KEYWORD_PATH])
recorder = PvRecorder(frame_length=porcupine.frame_length)
recorder.start()

print("Listening for wake word...")
while True:
    pcm = recorder.read()
    keyword_index = porcupine.process(pcm)
    if keyword_index >= 0:
        print("Wake word detected.")
        break

recorder.stop()

Generate Transcriptions for the Claude Voice Assistant

After wake word detection, capture audio frames and transcribe them in real-time with Cheetah Streaming Speech-to-Text:

import pvcheetah

ACCESS_KEY = "${ACCESS_KEY}"

cheetah = pvcheetah.create(
            access_key=ACCESS_KEY,
            endpoint_duration_sec= 1.0)

recorder = PvRecorder(frame_length=cheetah.frame_length)
recorder.start()

print("Speak your message…")
transcript = ""
while True:
    pcm = recorder.read()
    partial_transcript, is_endpoint = cheetah.process(pcm)
    transcript += partial_transcript
    print(partial_transcript, end="", flush=True)
    if is_endpoint:
        final_transcript = cheetah.flush()
        transcript += final_transcript
        print(final_transcript)
        break

recorder.stop()
cheetah.delete()

Each completed segment returns text, which is ready to send to Claude's API.

Send Transcribed Text to the Claude API

Once speech is transcribed, send the text to Claude using Anthropic's messages endpoint:

from anthropic import Anthropic

ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

def ask_claude(api_key: str, prompt: str, model: str = "claude-sonnet-4-20250514") -> str:
    """
    Uses Claude Sonnet 4 (May 2025 release) for balanced performance.
    Alternative models:
    - claude-opus-4-20250514: For more complex reasoning tasks
    - claude-haiku-3-20240307: For faster, simpler responses
    """
    try:
        client = Anthropic(api_key=api_key)
        response = client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}]
        )
        return response.content[0].text
    except Exception as e:
        return f"I'm having trouble connecting to Claude. Error: {e}"

# transcript = "example user request"
reply = ask_claude(ANTHROPIC_API_KEY, transcript)

This minimal integration sends text to Claude while all speech processing remains on-device.

Generate Voice Output from Claude's Responses

Transform Claude's text response into natural speech using Orca Streaming Text-to-Speech and PvSpeaker:

import pvorca
from pvspeaker import PvSpeaker
from collections import deque

ACCESS_KEY = "${ACCESS_KEY}"
orca = pvorca.create(access_key=ACCESS_KEY)
speaker = PvSpeaker(sample_rate=orca.sample_rate, bits_per_sample=16)

# Synthesize speech
# reply = response from claude
pcm_out, _ = orca.synthesize(reply)

# Play audio
speaker.start()

pcm_buffer = deque()
pcm_buffer.append(pcm_out)

while len(pcm_buffer) > 0:
    pcm = pcm_buffer.popleft()
    written = speaker.write(pcm)
    if written < len(pcm):
        pcm_buffer.appendleft(pcm[written:])

speaker.flush()
speaker.stop()

# Cleanup
speaker.delete()
orca.delete()

Complete Python Code for Voice-Enabled Claude Assistant

This implementation combines three Picovoice engines: Porcupine Wake Word, Cheetah Streaming Speech-to-Text, and Orca Streaming Text-to-Speech.

import argparse
from collections import deque
import sys
from anthropic import Anthropic
import pvporcupine
import pvcheetah
import pvorca
from pvrecorder import PvRecorder
from pvspeaker import PvSpeaker


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Porcupine + Cheetah + Orca voice interface for Claude"
    )
    parser.add_argument("--access_key", required=True, help="Picovoice AccessKey")
    parser.add_argument("--keyword_paths", nargs='+', required=True, 
                       help="Path(s) to .ppn wake-word model(s)")
    parser.add_argument("--anthropic_key", required=True, help="Anthropic API key")
    args = parser.parse_args()

    porcupine = None
    cheetah = None
    orca = None
    recorder = None
    speaker = None

    try:
        # Initialize engines
        porcupine = pvporcupine.create(
            access_key=args.access_key, 
            keyword_paths=args.keyword_paths)
        cheetah = pvcheetah.create(access_key=args.access_key, endpoint_duration_sec=1.0)
        orca = pvorca.create(access_key=args.access_key)

        print(f'Porcupine version: {porcupine.version}')
        print(f'Cheetah version: {cheetah.version}')
        print(f'Orca version: {orca.version}\n')

        # Initialize speaker
        speaker = PvSpeaker(
            sample_rate=orca.sample_rate, 
            bits_per_sample=16)

        # Initialize recorder
        recorder = PvRecorder(frame_length=porcupine.frame_length)
        recorder.start()

        print("Ready. Say the wake word… (Ctrl+C to stop)")

        # Wait for wake word
        while True:
            pcm = recorder.read()
            keyword_index = porcupine.process(pcm)
            if keyword_index >= 0:
                print("[EVENT] Wake word detected")
                break

        recorder.stop()
        recorder.delete()
        recorder = PvRecorder(frame_length=cheetah.frame_length)
        recorder.start()

        # Stream STT with Cheetah
        print("Speak your message…")
        transcript = ""
        while True:
            pcm = recorder.read()
            partial_transcript, is_endpoint = cheetah.process(pcm)
            transcript += partial_transcript
            print(partial_transcript, end="", flush=True)
            if is_endpoint:
                final_transcript = cheetah.flush()
                transcript += final_transcript
                print(final_transcript)
                break

        print("[TRANSCRIPT]", transcript)

        # Call Claude
        client = Anthropic(api_key=args.anthropic_key)
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            messages=[{"role": "user", "content": transcript}],
            max_tokens=1024,
            timeout=45
        )

        reply = response.content[0].text
        print("[REPLY]", reply)

        # Synthesize speech with Orca
        pcm_out, _ = orca.synthesize(reply)

        # Play audio
        speaker.start()

        pcm_buffer = deque()
        pcm_buffer.append(pcm_out)

        while len(pcm_buffer) > 0:
            pcm = pcm_buffer.popleft()
            written = speaker.write(pcm)
            if written < len(pcm):
                pcm_buffer.appendleft(pcm[written:])

        speaker.flush()
        speaker.stop()

    except KeyboardInterrupt:
        print("\n[EXIT] Stopping…")
    except pvporcupine.PorcupineActivationLimitError:
        print("AccessKey has reached its processing limit")
    except pvcheetah.CheetahActivationLimitError:
        print("AccessKey has reached its processing limit")
    except pvorca.OrcaActivationLimitError:
        print("AccessKey has reached its processing limit")
    finally:
        # Cleanup
        if speaker is not None:
            speaker.delete()
        
        if recorder is not None:
            recorder.delete()
        
        if orca is not None:
            orca.delete()
        
        if cheetah is not None:
            cheetah.delete()
        
        if porcupine is not None:
            porcupine.delete()

    return 0


if __name__ == "__main__":
    sys.exit(main())

Run the Claude Voice Assistant

To run the voice-enabled Claude assistant, update the model path to match your local file and have both API keys ready:

Picovoice AccessKey (copy it from the Picovoice Console)
Claude API key (available from the Claude Console)

python3 voice_claude.py \
  --access_key "$ACCESS_KEY" \
  --keyword_paths ./models/hey-claude.ppn \
  --anthropic_key "$ANTHROPIC_API_KEY" \

The Claude voice assistant is now running and ready to listen, transcribe, and respond.

Common Issues and Solutions

Resolve API Connection Issues for Claude API

Problem: "Error: Authentication failed" or "API key not valid"
Solution: Verify your Claude API key is correct in the Claude Console. Ensure billing is enabled and your account has available credits.

Problem: "Error: Rate limit exceeded"
Solution: Claude API has rate limits based on your plan. Wait a moment and retry, or upgrade your plan for higher limits.

Resolve Audio Device Issues

Problem: "Failed to initialize PvRecorder" or "Audio device not found"
Solution: Make sure to use the correct --audio_device_index parameter. List available audio devices by running the following python code:

from pvrecorder import PvRecorder
print(PvRecorder.get_available_devices())

Problem: No audio output from speaker
Solution: Check speaker volume and connections. Verify PvSpeaker initialization by running the following python code:

from pvspeaker import PvSpeaker
print(PvSpeaker.get_available_devices())

You can start building your own commercial or non-commercial projects leveraging Picovoice's self-service Console.

Start Building

Frequently Asked Questions

Can I use a different wake word instead of 'Hey Claude' for my voice assistant?

Yes. You can train any custom wake word using Picovoice Console in seconds without collecting training data. Simply enter your desired phrase (e.g., "Hey Assistant", "Computer", or your brand name), and download the trained model. The wake word guide provides best practices for selecting effective wake phrases. You can also detect multiple wake words simultaneously to support different commands.

What happens to the voice assistant if Claude API calls fail or timeout during a conversation?

Network timeouts, rate limits, or API outages can cause the Claude API requests to fail. The code includes error handling that catches these exceptions and can use Orca Streaming Text-to-Speech to provide voice feedback like "I'm having trouble connecting, please try again." Since wake word detection and speech recognition run entirely on-device, the voice interface remains functional during API failures and only Claude responses are unavailable.

Can I build a voice assistant for languages other than English?

Yes. The tutorial uses Porcupine Wake Word supporting 9 languages, Cheetah Streaming Speech-to-Text supporting 6 languages, and Orca Streaming Text-to-Speech supporting 8 languages. English, French, German, Italian, Portuguese, and Spanish are supported across all three engines. To build a voice assistant in a different language, download language-specific model files from the Picovoice Console and specify the model path when initializing each engine.