Rhino - React Native API
This document outlines how to integrate Rhino Speech-to-Intent engine within an application using its React Native API.
Requirements
- Node
- yarn or npm
- CocoaPods
- Android SDK 16+
- JDK 8+
- Xcode 9+
Refer to React Native's environment setup page for specifics on how to prepare your machine for React Native development.
Compatibility
- React Native 0.62.2+
- Android 4.1+ (API 16+)
- iOS 9.0+
Installation
To install Porcupine into you React Native project add the following native modules to your project:
yarn add @picovoice/react-native-voice-processoryarn add @picovoice/rhino-react-native
or
npm i @picovoice/react-native-voice-processor --savenpm i @picovoice/rhino-react-native --save
Link the iOS package
cd ios && pod install && cd ..
NOTE: Due to a limitation in React Native CLI autolinking, these two native modules cannot be included as transitive dependencies. If you are creating a module that depends on rhino-react-native and/or react-native-voice-processor, you will have to list these as peer dependencies and require developers to install them alongside.
Permissions
To enable recording with the hardware's microphone, you must first ensure that you have enabled the proper permission on both iOS and Android.
On iOS, open your Info.plist and add the following line:
<key>NSMicrophoneUsageDescription</key><string>[Permission explanation]</string>
On Android, open your AndroidManifest.xml and add the following line:
<uses-permission android:name="android.permission.RECORD_AUDIO" />
Finally, in your app JS code, be sure to check for user permission consent before proceeding with audio capture:
let recordAudioRequest;if (Platform.OS == 'android') {// For Android, we need to explicitly askrecordAudioRequest = this._requestRecordAudioPermission();} else {// iOS automatically asks for permissionrecordAudioRequest = new Promise(function (resolve, _) {resolve(true);});}recordAudioRequest.then((hasPermission) => {if(hasPermission){// Code that uses Rhino}});async _requestRecordAudioPermission() {const granted = await PermissionsAndroid.request(PermissionsAndroid.PERMISSIONS.RECORD_AUDIO,{title: 'Microphone Permission',message: '[Permission explanation]',buttonNeutral: 'Ask Me Later',buttonNegative: 'Cancel',buttonPositive: 'OK',});return (granted === PermissionsAndroid.RESULTS.GRANTED)}
Usage
The module provides you with two levels of API to choose from depending on your needs.
High-Level API
RhinoManager provides a high-level API that takes care of audio recording. This class is the quickest way to get started.
The constructor RhinoManager.create
will create an instance of a RhinoManager using a context file that you pass to it.
async createRhinoManager(){try{this._rhinoManager = await RhinoManager.create('/path/to/context/file.rhn',inferenceCallback);} catch (err) {// handle error}}
NOTE: the call is asynchronous and therefore should be called in an async block with a try/catch.
The inferenceCallback
parameter is a function that you want to execute when Rhino makes an inference.
The function should accept an object, which will be a JSON representation of the inference.
inferenceCallback(object){console.log(JSON.stringify(inference));}
You can override also the default Rhino model file and/or the inference sensitivity. These optional parameters can be passed in like so:
this._rhinoManager = await RhinoManager.create("/path/to/context/file.rhn",inferenceCallback,'path/to/model/file.pv',0.25);
Once you have instantiated a RhinoManager, you can start audio capture and intent inference by calling:
let didStart = await this._rhinoManager.process();
When RhinoManager returns an inference result via the inferenceCallback, it will automatically stop audio capture for you. When you wish to result, call .process()
again.
Once your app is done with using RhinoManager, be sure you explicitly release the resources allocated for it:
this._rhinoManager.delete();
As you may have noticed, there is no need to deal with audio capture to enable intent inference with RhinoManager. This is because it uses our @picovoice/react-native-voice-processor module to capture frames of audio and automatically pass it to the inference engine.
Low-Level API
Rhino provides low-level access to the inference engine for those who want to incorporate speech-to-intent into a already existing audio processing pipeline.
Rhino
is created by passing a context file to its static constructor create
:
async createRhino(){try{this._rhino = await Rhino.create('/path/to/context/file.rhn');} catch (err) {// handle error}}
As you can see, in this case you don't pass in an inference callback as you will be passing in audio frames directly using the process
function. The JSON result that is returned from process
will have up to four fields:
- isFinalized - whether Rhino has made an inference
- isUnderstood - if isFinalized, whether Rhino understood what it heard based on the context
- intent - if isUnderstood, name of intent that were inferred
- slots - if isUnderstood, dictionary of slot keys and values that were inferred
let buffer = getAudioFrame();try {let result = await this._rhino.process(buffer);// inference result example:// {// isFinalized: true,// isUnderstood: true,// intent: 'orderDrink',// slots: {// size: 'medium',// coffeeDrink: 'americano',// sugarAmount: 'some sugar'// }// }}} catch (e) {// handle error}
For process to work correctly, the audio data must be in the audio format required by Picovoice.
The required audio format is found by calling .sampleRate
to get the required sample rate and .frameLength
to get the required frame size. Audio must be single-channel and 16-bit linearly-encoded.
Finally, once you no longer need the inference engine, be sure to explicitly release the resources allocated to Rhino:
this._rhino.delete();
Custom Context
You can create custom Rhino context models using Picovoice Console.