Picovoice - React API

Wake Word Engine
Offline Voice Commands
Local Speech Recognition
Always Listening
npm

This document outlines how to integrate the Picovoice SDK within an application using its React API.

Requirements

React 16.8+
yarn (or npm)

Compatibility

Chrome, Edge
Firefox
Safari

The Picovoice SDKs for Web are powered by WebAssembly (WASM), the Web Audio API, and Web Workers.

All modern browsers (Chrome/Edge/Opera, Firefox, Safari) are supported, including on mobile. Internet Explorer is not supported.

Using the Web Audio API requires a secure context (HTTPS connection), with the exception of localhost, for local development.

Installation

Use npm or yarn to install the package and its peer dependencies. Each spoken language (e.g. 'en', 'de') is a separate package. For this example we'll use English:

yarn add @picovoice/picovoice-web-react @picovoice/picovoice-web-en-worker @picovoice/web-voice-processor

(or)

npm install @picovoice/picovoice-web-react @picovoice/picovoice-web-en-worker @picovoice/web-voice-processor

Language-specific Picovoice web worker packages

These worker packages are compatible with the React SDK:

Usage

The Picovoice SDK for React is based on the Picovoice SDK for Web. The library provides a React hook: usePicovoice. The hook will take care of microphone access and audio downsampling (via @picovoice/web-voice-processor) and provide wake word and inference callbacks.

The usePicovoice hook provides a collection of fields and methods shown below. You can pass the keywordEventHandler and inferenceEventHandler to respond to Picovoice wake word detection events and follow-on command inference, respectively.

usePicovoice Hook Parameters

The usePicovoice hook accepts four arguments:

The Picovoice worker factory (language specific, imported as PicovoiceWorkerFactory from the @picovoice/picovoice-web-xx-worker series of packages, where xx is the two-letter language code. (The Picovoice WebAssembly code, wrapped in a Web worker. Dependency injection is needed to keep the React SDK spoken-language-agnostic)
The PicovoiceHookArgs (what we want Picovoice to listen for and understand; see below for details)
The callback function with signature (keywordLabel: string) => void (what to do when Picovoice detects our keyword)
The callback function with signature (inference: RhinoInference) => void (what to do when Picovoice concludes its follow-on command inference)

Make sure you handle the possibility of errors with the isError and errorMessage fields. Users may not have a working microphone, and they can always decline (and revoke) permissions; your application code should anticipate these scenarios.

export type PicovoiceHookArgs = {
  porcupineKeyword: PorcupineKeyword
  rhinoContext: RhinoContext
  start?: boolean
}

The PicovoiceHookArgs accepts a single porcupineKeyword of type PorcupineKeyword.

PorcupineKeyword can be either PorcupineKeywordBuiltin or PorcupineKeywordCustom:

export type PorcupineKeywordBuiltin = {
  /** Name of a builtin keyword for the specific language (e.g. "Grasshopper" for English, or "Ananas" for German) */
  builtin: string
  /** Value in range [0,1] that trades off miss rate for false alarm */
  sensitivity?: number
}

English	French	German	Spanish
"Americano" "Blueberry" "Bumblebee" "Grapefruit" "Grasshopper" "Hey Google" "Hey Siri" "Jarvis" "Okay Google" "Picovoice" "Porcupine" "Terminator"	"Framboise" "Mon Chouchou" "Parapluie" "Perroquet" "Tournesol"	"Ananas" "Heuschrecke" "Himbeere" "Leguan" "Stachelschwein"	"Emparedado" "Leopardo" "Manzana" "Murcielago"

If you simply pass a string of a builtin keyword instead of an object, that will also work.

Use PorcupineKeywordCustom for custom keywords:

export type PorcupineKeywordCustom = {
  /** Base64 representation of a trained Porcupine keyword (`.ppn` file) */
  base64: string
  /** An arbitrary label that you want Porcupine to report when the detection occurs */
  custom: string
  /** Value in range [0,1] that trades off miss rate for false alarm */
  sensitivity?: number
}

PicovoiceHookArgs also requires a rhinoContext field of type type RhinoContext, to handle the follow-on commands:

export type RhinoContext = {
  /** Base64 representation of a trained Rhino context (`.rhn` file) */
  base64: string
  /** Value in range [0,1] that trades off miss rate for false alarm */
  sensitivity?: number
}

Imports

Using static imports for the picovoice-web-xx-worker packages is straightforward, but will impact your initial bundle size with an additional ~6MB. Depending on your requirements, this may or may not be feasible. If you require a small bundle size, see dynamic importing below.

Static Import

import React, { useState } from 'react';
// Import the specific PicovoiceWorkerFactory for the spoken language used: in this case, English (en).
import { PicovoiceWorkerFactory } from '@picovoice/picovoice-web-en-worker';
import { usePicovoice } from '@picovoice/picovoice-web-react';

const RHN_CONTEXT_BASE64 = /* Base64 representation of English-language `.rhn` file, omitted for brevity */

export default function VoiceWidget() {
  const [keywordDetections, setKeywordDetections] = useState([]);
  const [inference, setInference] = useState(null);

  const inferenceEventHandler = (rhinoInference) => {
    console.log(rhinoInference);
    setInference(rhinoInference);
  };

  const keywordEventHandler = (porcupineKeywordLabel) => {
    console.log(porcupineKeywordLabel);
    setKeywordDetections((x) => [...x, porcupineKeywordLabel]);
  };

  const {
    isLoaded,
    isListening,
    isError,
    errorMessage,
    start,
    resume,
    pause,
    engine,
  } = usePicovoice(
    PicovoiceWorkerFactory,
    {
      // "Picovoice" is one of the builtin wake words, so we merely need to ask for it by name.
      // To use a custom wake word, you supply the `.ppn` files in base64 and provide a label for it.
      porcupineKeyword: "Picovoice",
      rhinoContext: { base64: RHN_CONTEXT_BASE64 },
      start: true,
    },
    keywordEventHandler,
    inferenceEventHandler
  );

return (
  <div className="voice-widget">
    <h3>Engine: {engine}</h3>
    <h3>Keyword Detections:</h3>
    {keywordDetections.length > 0 && (
      <ul>
        {keywordDetections.map((label, index) => (
          <li key={index}>{label}</li>
        ))}
      </ul>
    )}
    <h3>Latest Inference:</h3>
    {JSON.stringify(inference)}
  </div>
)

Dynamic Import / Code Splitting

If you are shipping the Picovoice SDK for Web and wish to avoid adding its ~4-6MB to your application's initial bundle, you can use dynamic imports. These will split off the porcupine-web-xx-worker packages into separate bundles and load them asynchronously. This means we need additional logic.

We add a useEffect hook to kick off the dynamic import. We store the result of the dynamically loaded worker chunk into a useState hook. When usePicovoice receives a non-null/undefined value for the worker factory, it will start up Picovoice.

See the Webpack docs for more information about Code Splitting.

import { useState, useEffect } from "react";
// Note we are not statically importing "@picovoice/picovoice-web-en-worker" here
import { usePicovoice } from "@picovoice/picovoice-web-react";

const RHN_CONTEXT_BASE64 = /* Base64 representation of an English-language `.rhn` file, omitted for brevity */

export default function VoiceWidget() {
  const [workerChunk, setWorkerChunk] = useState({ workerFactory: null });

  useEffect(() => {
    async function loadPorcupineWorkerChunk() {
      const pvWorkerFactory = (await import("@picovoice/picovoice-web-en-worker")).PicovoiceWorkerFactory; // <-- Dynamically import the worker
      console.log("Picovoice worker chunk is loaded.");
      return pvWorkerFactory;
    }
    if (workerChunk.workerFactory === null) { // <-- We only want to load once!
      loadPorcupineWorkerChunk().then((ppnWorkerFactory) => {

        setWorkerChunk({ workerFactory: ppnWorkerFactory });
      });
    }
  }, [workerChunk]);

  const [keywordDetections, setKeywordDetections] = useState([]);
  const [inference, setInference] = useState(null);

  const inferenceEventHandler = (rhinoInference) => {
    console.log(rhinoInference);
    setInference(rhinoInference);
  };

  const keywordEventHandler = (porcupineKeywordLabel) => {
    console.log(porcupineKeywordLabel);
    setKeywordDetections((x) => [...x, porcupineKeywordLabel]);
    setInference("...")
  };

  const {
    isLoaded,
    isListening,
    isError,
    errorMessage,
    start,
    resume,
    pause,
    engine,
  } = usePicovoice(
    workerChunk.workerFactory, // <-- When this is null/undefined, it's ignored. Otherwise, usePicovoice will start.
    {
      porcupineKeyword: "Picovoice",
      rhinoContext: { base64: RHN_CONTEXT_BASE64 },
    },
    keywordEventHandler,
    inferenceEventHandler
  );

Important Note: Internally, usePicovoice performs work asynchronously to initialize, as well as asking for microphone permissions. Not until the asynchronous tasks are done and permission given will Picovoice actually be running. Therefore, it makes sense to use the isLoaded state to update your UI to let users know your application is actually ready to process voice (and isError in case something went wrong). Otherwise, they may start speaking and their audio data will not be processed, leading to a poor/inconsistent experience.