Add Voice Control to AI Cooking App in Python

🎯 Voice AI Consulting

Get dedicated support and consultation to ensure your specific needs are met.

Adding AI-powered voice control to cooking apps doesn’t have to be complicated. With Rhino Speech-to-Intent, you can enable hands-free recipe navigation and timer control in just a few lines of code, without heavy data pipelines or complex infrastructure.

In this tutorial, we'll build a Python-based hands-free cooking workflow powered by Rhino Speech-to-Intent. We will create a responsive, and low-latency workflow where users can follow recipes and track steps with their voice while cooking.

Why On-Device Speech-to-Intent is Better for Cooking Apps

Traditional cloud-based solutions use a speech-to-text pipeline, transcribing everything you say. This voice controlled cooking app uses a more efficient, modern approach on the device: end-to-end intent inference from speech.

Instead of transcribing audio, Rhino Speech-to-Intent directly understands what you mean and maps your speech to a pre-defined set of intents. Given an audio input, Rhino makes a structured decision: which intent was spoken, and which parameters (or "slots") are included. It can understand voice commands like:

"What's the next step?"

"How much salt do I need?"

With Rhino Speech-to-Intent, these commands are defined in a context file. The context lists the intents (e.g., next_step, repeat_step, set_timer) and the expected phrases for each intent.

To train a custom Speech-to-Intent model, create an account on the Picovoice Console.
Create an empty Rhino Speech-to-Intent context.
Click the "Import YAML" button in the top-right corner of the Console. Paste the YAML provided below to add the intents and expressions for the voice controlled cooking app.

Use the microphone button to test the context in-browser. This will save and build the context.
Download the context file for your platform.

YAML Context file for the Voice Controlled Cooking App:

context:
  expressions:
    set_timer:
      - "[set, start] (a) timer for $pv.TwoDigitInteger:duration minutes"
      - timer for $pv.TwoDigitInteger:duration minutes
    check_timer:
      - how much time is left
      - check (the) timer
    stop_timer:
      - "[stop, cancel, end] (the) timer"
    next_step:
      - "@go_to next step"
      - what's (the) next (step)
      - next instruction
    repeat_step:
      - repeat (the) [last, previous] step
      - repeat that
      - again please
    ingredient_info:
      - how much $ingredient:ingredient (do I need)
      - "(show me) how much (of) $ingredient:ingredient (is needed)"

  slots:
    ingredient:
      - salt
      - pepper
      - sugar
      - oil

  macros:
    go_to:
      - go to
      - take me to
      - skip to

For more details on how to build your context, refer to the Rhino Syntax Cheat Sheet.

Step 2: Set Up Your Python Environment for Voice Control

Install the Rhino Speech-to-Intent Python SDK pvrhino to run inference, and the Picovoice Python Recorder library pvrecorder to capture microphone audio.

pip install pvrhino pvrecorder

Step 3: Capture and Process Voice Commands in Real Time in Python

Now let's run Rhino Speech-to-Intent with live audio and process voice commands in real time.

Initialize Rhino Speech-to-Intent and Set up Microphone Input

The main function initializes Rhino with the ACCESS_KEY and context file, then sets up the audio recorder to capture microphone input.

# Initialize Rhino
rhino = pvrhino.create(
    access_key=access_key,
    context_path=context_path)

# Initialize recorder
recorder = PvRecorder(
    device_index=device_index,
    frame_length=rhino.frame_length)

Process Voice Commands in Real Time

Rhino continuously processes audio frames from the microphone. When a voice command is finalized, the cooking guide displays the recognized intent and executes the corresponding action.

recorder.start()
print("Listening for cooking commands... Press Ctrl+C to exit.\n")

while True:
    pcm = recorder.read()

    is_finalized = rhino.process(pcm)
    if is_finalized:
        inference = rhino.get_inference()
        process_inference(inference)
        rhino.reset()

Map Recognized Voice Commands to Cooking App Actions

Once Rhino finalizes an utterance, it produces an inference containing the recognized intent and slots. The process_inference function displays what was understood and triggers the corresponding action in the recipe app.

def process_inference(inference):
    """Process and display the voice recognition inference"""
    if inference.is_understood:
        # Display what was understood
        print("{")
        print(f"  intent : '{inference.intent}'")
        print("  slots : {")
        for slot, value in inference.slots.items():
            print(f"    {slot} : '{value}'")
        print("  }")
        print("}\n")
        # integrate these with calls to your app logic
        handle_cooking_action(inference.intent, inference.slots)
    else:
        print("Didn't understand the command.\n")

The handle_cooking_action function maps Rhino's recognized intents and their slots into recipe-specific actions.

def handle_cooking_action(intent, slots):
    """Demo: map intents to cooking actions"""

    actions = {
        "set_timer": lambda: print(f"[Demo] Setting timer for {slots.get('duration', 'unknown')} minutes"),
        "check_timer": lambda: print("[Demo] Checking remaining time"),
        "stop_timer": lambda: print("[Demo] Stopping the timer"),
    }

    if intent in actions:
        actions[intent]()  
        # Replace this with the feature your app needs (e.g. start a timer)
        # For example: start_timer(slots['duration'])
    else:
        print("[Demo] Intent not handled")
        # You can add more intents here as your app grows

Here's the complete Python script integrating all the components above:

import os
import sys
import argparse
import pvrhino
from pvrecorder import PvRecorder


def handle_cooking_action(intent, slots):
    """Handle cooking-specific actions based on intent"""
    if intent == "set_timer":
        duration = slots.get("duration", "unknown")
        print(f"Setting timer for {duration} minutes\n")
    elif intent == "check_timer":
        print("Checking remaining time\n")
    elif intent == "stop_timer":
        print("Stopping the timer\n")
    elif intent == "next_step":
        print("Moving to the next step\n")
    elif intent == "repeat_step":
        print("Repeating the last step\n")
    elif intent == "ingredient_info":
        ingredient = slots.get("ingredient", "unknown")
        print(f"Checking how much {ingredient} you need\n")
    else:
        print("Intent not handled in this demo\n")
    # Replace print statements with your app’s logic
    # (e.g. start_timer(duration))


def process_inference(inference):
    """Process and display the voice recognition inference"""
    if inference.is_understood:
        # Display what was understood
        print("{")
        print(f"  intent : '{inference.intent}'")
        print("  slots : {")
        for slot, value in inference.slots.items():
            print(f"    {slot} : '{value}'")
        print("  }")
        print("}\n")

        # Handle the cooking action
        handle_cooking_action(inference.intent, inference.slots)
    else:
        print("Didn't understand the command.\n")


def main():
    # Set up command line argument parser
    parser = argparse.ArgumentParser(
        description="Rhino Speech-to-Intent Demo for Cooking Assistant"
    )
    parser.add_argument(
        "--access_key",
        required=True,
        help="Picovoice access key"
    )
    parser.add_argument(
        "--context_file_path",
        required=True,
        help="Path to the context file (.rhn)"
    )
    parser.add_argument(
        "--audio_device_index",
        type=int,
        default=-1,
        help="Audio device index (default: -1)"
    )

    args = parser.parse_args()

    # Load command line arguments
    access_key = args.access_key
    context_path = args.context_file_path
    device_index = args.audio_device_index

    if not access_key or len(access_key.strip()) == 0:
        print(
            "Error: Please provide ACCESS_KEY"
        )
        sys.exit(1)

    if not os.path.isfile(context_path):
        print(f"Error: Context file not found at {context_path}")
        sys.exit(1)

    rhino = None
    recorder = None

    try:
        # Initialize Rhino
        rhino = pvrhino.create(
            access_key=access_key,
            context_path=context_path
        )

        # Initialize recorder
        recorder = PvRecorder(
            device_index=device_index,
            frame_length=rhino.frame_length
        )

        recorder.start()
        print(
            "Listening for cooking commands... "
            "Press Ctrl+C to exit.\n"
        )

        while True:
            pcm = recorder.read()

            is_finalized = rhino.process(pcm)
            if is_finalized:
                inference = rhino.get_inference()
                process_inference(inference)
                rhino.reset()

    except KeyboardInterrupt:
        print("\nStopping ...")
    finally:
        recorder.stop()
        recorder.delete()
        rhino.delete()


if __name__ == "__main__":
    main()

Run Your Cooking App with Voice Commands

Run the script from your terminal with the following commands. Copy your AccessKey from the Picovoice Console and replace the placeholder values with your actual ACCESS_KEY and CONTEXT_FILE_PATH. The AUDIO_DEVICE_INDEX is optional and will use the default microphone if not specified.

python voice-cooking-app.py \
    --access_key="${ACCESS_KEY}" \
    --context_file_path="${CONTEXT_FILE_PATH}" \
    --audio_device_index="${AUDIO_DEVICE_INDEX}"

The cooking app is now ready and listening for your voice commands.

Unlock New Voice Features for Cooking Apps

With Rhino Speech-to-Intent, you can extend beyond timers and recipe navigation to:

Shopping List Management: Add missing ingredients directly to a list or send them to a companion app.
Nutritional Guidance: Query calories, portions, or dietary substitutions while preparing meals.

These features can turn recipe apps into interactive, hands-free cooking companions that are personalized, responsive and private.

Interested in adding more voice-powered features? Add Porcupine Wake Word and Orca Text-to-Speech to create a full kitchen assistant responding to users.

Start Adding Voice to Your Cooking App with Picovoice

Ready to add voice control to your cooking app? The Picovoice Console is the best place to start. You can design your custom intent model for free in just a few clicks.

Start Free