Rhino - React API

NPM
NLU
React Hooks

This document outlines how to integrate the Rhino wake word engine within an application using its React API.

Requirements

yarn (or npm)
Secure browser context (i.e. HTTPS or localhost)

Compatibility

Chrome, Edge
Firefox
Safari

The Picovoice SDKs for Web are powered by WebAssembly (WASM), the Web Audio API, and Web Workers. All audio processing is performed in-browser, providing intrinsic privacy and reliability.

All modern browsers are supported, including on mobile. Internet Explorer is not supported.

Using the Web Audio API requires a secure context (HTTPS connection), with the exception of localhost, for local development.

Installation

Use npm or yarn to install the package and its peer dependencies. Each spoken language (e.g. 'en', 'de') is a separate package. For this example we'll use English:

yarn add @picovoice/rhino-web-react @picovoice/rhino-web-en-worker @picovoice/web-voice-processor

(or)

npm install @picovoice/rhino-web-react @picovoice/rhino-web-en-worker @picovoice/web-voice-processor

Language-specific Rhino web worker packages

These worker packages are compatible with the React SDK:

Usage

The Rhino SDK for React is based on the Rhino SDK for Web. The library provides a React hook: useRhino. The hook will take care of microphone access and audio downsampling (via @picovoice/web-voice-processor) and provide a wake word detection event to which your application can subscribe.

The Rhino library is by default a "push-to-talk" experience. You can use a button to trigger the isTalking state. Rhino will listen and process frames of microphone audio until it reaches a conclusion. If the utterance matched something in your Rhino context (e.g. "make me a coffee" in a coffee maker context), the details of the inference are returned.

The useRhino hook has three parameters:

The rhinoWorkerFactory (language-specific, imported as from the @picovoice/rhino-web-xx-worker series of packages, where xx is the two-letter language code)
The rhinoHookArgs (i.e. what context we want Rhino to listen for)
The inferenceCallback function with signature (inference: RhinoInference) => void (what to do when Rhino concludes an inference)

Make sure you handle the possibility of errors with the isError and errorMessage fields. Users may not have a working microphone, and they can always decline (and revoke) permissions; your application code should anticipate these scenarios.

export function useRhino(
  rhinoWorkerFactory: RhinoWorkerFactory | null,
  rhinoHookArgs: RhinoHookArgs,
  inferenceCallback: (inference: RhinoInference) => void
): {
  isLoaded: boolean
  isListening: boolean
  isError: boolean
  isTalking: boolean
  errorMessage: string | null
  start: () => void
  pause: () => void
  pushToTalk: () => void
  resume: () => void
}

Note that you can pass in null for the rhinoWorkerFactory argument. This is intentional. If you are loading the @picovoice/rhino-web-xx-worker package with a dynamic import(), it will not be ready immediately. The hook will wait until RhinoWorkerFactory is non-null to begin.

Use the pushToTalk function to start Rhino (typically connected to a button). If you want to use Rhino with a wake word instead, see the Picovoice SDK for React which combines the two engines.

export type RhinoHookArgs = {
  /** The context to instantiate */
  context: RhinoContext
  /** Immediately start the microphone upon initialization */
  start: boolean
}

export type RhinoContext = {
  /** Base64 representation of a trained Rhino context (`.rhn` file) */
  base64: string
  /** Value in range [0,1] that trades off miss rate for false alarm */
  sensitivity?: number
}

The inferenceCallback provides a RhinoInference object:

export type RhinoInference = {
  /** Rhino has concluded the inference (isUnderstood is now set) */
  isFinalized: boolean
  /** The intent was understood (it matched an expression in the context) */
  isUnderstood?: boolean
  /** The name of the intent */
  intent?: string
  /** Map of the slot variables and values extracted from the utterance */
  slots?: Record<string, string>
}

Imports

You can use Rhino by importing the worker package statically or dynamically. Static is more straightforward to implement, but will impact your initial bundle size with an additional ~4MB. Depending on your requirements, this may or may not be feasible. If you require a small bundle size, see dynamic importing below.

Static Import

import React, { useState } from 'react';
import { RhinoWorkerFactory } from '@picovoice/rhino-web-en-worker';
import { useRhino } from '@picovoice/rhino-web-react';

const RHN_CONTEXT_CLOCK_64 = /* Base64 representation of English language clock_wasm.rhn, omitted for brevity */

function VoiceWidget(props) {
  const [latestInference, setLatestInference] = useState(null)

  const inferenceEventHandler = (rhinoInference) => {
    console.log(`Rhino inferred: ${rhinoInference}`);
    setLatestInference(rhinoInference)
  };

  const {
    isLoaded,
    isListening,
    isError,
    isTalking,
    errorMessage,
    start,
    resume,
    pause,
    pushToTalk,
  } = useRhino(
    // Pass in the factory to build Rhino workers. This needs to match the context language below
    RhinoWorkerFactory,
    // Initialize Rhino (in a paused state) with the clock context.
    // Immediately start processing microphone audio,
    // Although Rhino itself will not start listening until the Push to Talk button is pressed.
    {
      context: { base64: RHN_EN_CLOCK_64 },
      start: true,
    }
    inferenceEventHandler
  );

return (
  <div className="voice-widget">
    <button onClick={() => pushToTalk()} disabled={isTalking || isError || !isLoaded}>
      Push to Talk
    </button>
    <p>{JSON.stringify(latestInference)}</p>
  </div>
)

The inferenceEventHandler will log the inference to the browser's JavaScript console and display the most recent one. Use the push-to-talk button to activate Rhino.

Important Note: Internally, useRhino performs work asynchronously to initialize, as well as asking for microphone permissions. Not until the asynchronous tasks are done and permission given will Rhino actually be running. Therefore, it makes sense to use the isLoaded state to update your UI to let users know your application is actually ready to process voice (and isError in case something went wrong). Otherwise, they may start speaking and their audio data will not be processed, leading to a poor/inconsistent experience.

Dynamic Import

If you are shipping Rhino for the Web and wish to avoid adding its ~4MB to your application's initial bundle, you can use dynamic imports. These will split off the rhino-web-xx-worker packages into separate bundles and load them asynchronously. This means we need additional logic.

We add a useEffect hook to kick off the dynamic import. We store the result of the dynamically loaded worker chunk into a useState hook. When useRhino receives a non-null/undefined value for the worker factory, it will automatically start up Rhino.

See the Webpack docs for more information about Code Splitting.

import React, { useState, useEffect } from 'react';
import { RhinoWorkerFactory } from '@picovoice/rhino-web-en-worker';
import { useRhino } from '@picovoice/rhino-web-react';

const RHN_CONTEXT_CLOCK_64 = /* Base64 representation of English language clock_wasm.rhn, omitted for brevity */

function VoiceWidget(props) {
  const [latestInference, setLatestInference] = useState(null)

  const inferenceEventHandler = (rhinoInference) => {
    console.log(`Rhino inferred: ${rhinoInference}`);
    setLatestInference(rhinoInference)
  };

  const [workerChunk, setWorkerChunk] = useState({ factory: null });

  useEffect(() => {
    async function loadRhino() {
      // Dynamically import the worker
      const rhnEnWorkerFactory = (await import('@picovoice/rhino-web-en-worker'))
        .RhinoWorkerFactory;
      console.log('Rhino worker (EN) chunk is loaded.');
      return rhnEnWorkerFactory;
    }

    if (workerChunk.factory === null) {
      loadRhino().then(workerFactory => {
        setWorkerChunk({ factory: workerFactory });
      });
    }
  }, [workerChunk]);

  const {
    isLoaded,
    isListening,
    isError,
    isTalking,
    errorMessage,
    pushToTalk,
  } = useRhino(
    workerChunk.factory,
    {
      context: { base64: RHN_EN_CLOCK_64 },
      start: true,
    },
    inferenceEventHandler
  );