Local LLM-Powered Voice Assistant for Web Browsers

AI Assistants and AI Agents are changing our lives. A Large Language Model (LLM) is the standard component of modern GenAI assistants. A voice-based LLM assistant can provide a more natural, efficient, and convenient user experience. Voice Agents also unlock use cases in call centers and customer support. However, Voice Assistants require additional voice AI models to work harmoniously with the core LLM "brain" to create a satisfactory end-to-end user experience.

The recent GPT-4o audio demo proved that an LLM-powered voice assistant done correctly could create an awe-inspiring experience. Unfortunately, GPT-4o's audio feature does not run locally. Prompts and audio data are sent to a server were cloud-based inference is sent and a response is sent back.

What if we need to build an LLM-powered web application and privacy regulations block you from making API requests with voice data? Picovoice runs locally, meaning voice processing and LLM inference are performed locally without the user's data traveling to a third-party API. Products built with Picovoice are private by design, compliant (GDPR, HIPPA, ...), and real-time without unreliable network (API) latency.

First, Let's Play!

Before building an LLM-powered voice assistant, let's check out the experience. Picovoice Web SDKs can be run in:

• Chrome / Edge
• Firefox
• Safari

Phi-2 on Browsers

The video below shows Picovoice's LLM-powered voice assistant running Microsoft's Phi-2 model on Safari, FireFox and Chrome-based browsers. Running an LLM locally on browsers enables use cases that can't afford to rely on connectivity, such as healthcare, finance, and legal, where privacy and compliance are paramount.

Anatomy of an LLM Voice Assistant

There are four things an LLM-powered AI assistant needs to do:

1. Pay attention to when the user utters the wake word.
2. Recognize the request (question or command) the user is uttering.
3. Generate a response to a request using LLM.
4. Synthesize speech from LLM's text response.

1. Wake Word

A Wake Word Engine is a voice AI software that understands a single phrase. Every time you use Alexa, Siri, or Hey Google, you activate their wake word engine.

2. Streaming Speech-to-Text

Once we know the user is talking to us, we must understand what they say. This is done using Speech-to-Text. For latency-sensitive applications, we use the real-time variant of speech-to-text, also known as Streaming Speech-to-Text. The difference is that a Streaming Speech-to-Text transcribes speech as the user talks. In contrast, a normal speech-to-text waits for the user to finish before processing (e.g., OpenAI's Whisper).

3. LLM Inference

Once the user's request is available in text format, we need to run the LLM to generate the completion. Once the LLM inference starts, it generates the response piece-by-piece (token-by-token). We use this property of LLMs to run them in parallel with speech synthesis to reduce latency (more on this later). LLM inference is very compute intensive, and running it on the device requires techniques to reduce memory and compute requirements. A standard method is quantization (compression).

4. Streaming Text-to-Speech

A Text-to-Speech (TTS) engine accepts text and synthesizes the corresponding speech signal. Since LLMs can generate responses token-by-token as a stream, we prefer a TTS engine that can accept a stream of text inputs to lower the latency. We call this a Streaming Text-to-Speech.

Soup to Nuts

This section explains how to code a local LLM-powered voice assistant in JavaScript. You can check the entire script at LLM-powered voice assistant recipe in the picovoice cookbook github repository.

1. Voice Activation

Install Picovoice Porcupine Wake Word Engine:

yarn add @picovoice/porcupine-web

or using npm:

npm install -D @picovoice/porcupine-web

Import the package, initialize an instance of the wake word engine, and start processing audio in real time:

import { PorcupineWorker } from '@picovoice/porcupine-web';

// Model (.pv)
const porcupineModel = {
  publicPath: ${MODEL_RELATIVE_PATH},
  // or
  base64: ${MODEL_BASE64_STRING},
};

// custom keyword (.ppn)
const keywordModel = {
  publicPath: ${KEYWORD_RELATIVE_PATH},
  // or
  base64: ${KEYWORD_BASE64_STRING},
  label: ${KEYWORD_LABEL},
}

function keywordDetectionCallback(keyword) {
  console.log(`Porcupine detected keyword: ${keyword.label}`);
}

const porcupine = await PorcupineWorker.create(
  "${ACCESS_KEY}",
  [keywordModel],
  keywordDetectionCallback,
  porcupineModel
);

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console.

Replace ${MODEL_RELATIVE_PATH} with the model path relative to the public directory or ${MODEL_BASE64_STRING} with the base64 string of the model file.

Replace ${KEYWORD_RELATIVE_PATH} with the keyword file path relative to the public directory or ${KEYWORD_BASE64_STRING} with the base64 string of the keyword file. Replace ${KEYWORD_LABEL} with a label to identify the keyword.

To learn more about Porcupine, check out the Porcupine docs.

2. Speech Recognition

Install Picovoice Cheetah Streaming Speech-to-Text Engine:

yarn add @picovoice/cheetah-web

or using npm:

npm install -D @picovoice/cheetah-web

Import the package, initialize an instance of the streaming speech-to-text engine, and start transcribing audio in real time:

import { CheetahWorker } from '@picovoice/cheetah-web';

let transcript = "";

function transcriptCallback(cheetahTranscript: CheetahTranscript) {
  transcript += cheetahTranscript.transcript;
  if (cheetahTranscript.isEndpoint) {
    transcript += ". ";
  }
  if (cheetahTranscript.isFlushed) {
    transcript += "\n"
  }
}

const cheetahModel = {
  publicPath: ${MODEL_RELATIVE_PATH},
  // or
  base64: ${MODEL_BASE64_STRING},
}

const handle = await CheetahWorker.create(
  ${ACCESS_KEY},
  transcriptCallback,
  cheetahModel,
  {
    endpointDurationSec: ${ENDPOINT_DURATION_SEC}
  }
);

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console and ${ENDPOINT_DURATION_SEC} with the duration of silence at the end of the user's utterance to make sure they are done talking. The longer it is, the more time the user has to stutter or think in the middle of their request, but it also increases the perceived delay.

Replace ${MODEL_RELATIVE_PATH} with the model path relative to the public directory or ${MODEL_BASE64_STRING} with the base64 string of the model file.

To learn more about Cheetah, check out the Cheetah docs.

3. Response Generation

Install picoLLM Inference Engine:

yarn add @picovoice/picollm-web

or using npm:

npm install -D @picovoice/picollm-web

Import the package, initialize an instance of the LLM inference engine, create a dialog helper object, and start responding to the user's prompts:

import { PicoLLMWorker } from '@picovoice/picollm-web';

const picoLLMModel = {
  modelFile: ${MODEL_FILE_CONTENT},
}

const picoLLM = await PicoLLMWorker.create(
  "${ACCESS_KEY}",  
  picoLLMModel,
);
const dialog = picoLLM.getDialog();
dialog.addHumanRequest(prompt);

const streamCallback = (token) => {
  // do something with token
};

const res = await picoLLM.generate(dialog.prompt(), {
  streamCallback: streamCallback
});
// LLM is done generating the response, do something!
dialog.addLLMResponse(res.completion);

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console.

Replace ${MODEL_FILE_CONTENT} with picoLLM model, either a path to the resource, a file type or blob type./

To learn more about picoLLM, check out the picoLLM docs.

Note that the LLM's .generate function provides the response in pieces (i.e., token by token) using its streamCallback input argument. We pass every token as it becomes available to the Streaming Text-to-Speech and when the .generate function returns we notify the Streaming Text-to-Speech model that there is no more text and flush any remaining synthesized speech.

4. Speech Synthesis

Install Picovoice Orca Streaming Text-to-Speech Engine:

yarn add @picovoice/orca-web

or using npm:

npm install -D @picovoice/orca-web

Import the package, initialize an instance of Orca, and start synthesizing audio in real time:

import { OrcaWorker } from '@picovoice/orca-web';

const orcaModel = {
  publicPath: "${MODEL_RELATIVE_PATH}",
  // or
  base64: "${MODEL_BASE64_STRING}",
};

const orca = await OrcaWorker.create(
  "${ACCESS_KEY}",
  orcaModel
);
const orcaStream = await orca.streamOpen();

while (!llmDoneGeneratingResponse()) {
  const pcm = await orcaStream.synthesize(getNextLlmResponsePiece());
  if (pcm !== null) {
    // handle pcm
  }
}
const flushedPcm = await orcaStream.flush();
if (flushedPcm !== null) {
  // handle pcm
}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console.

Replace ${MODEL_RELATIVE_PATH} with the model path relative to the public directory or ${MODEL_BASE64_STRING} with the base64 string of the model file.

To learn more about Orca, check out the Orca docs.

What's Next?

The voice assistant we've created above is sufficient but basic. The Picovoice platform allows you to create more complex, multi-dimensional AI software products by mixing and matching our tools. For instance, we could add:

• Personalization: We could let the AI assistant not only know what is being said, but who is saying it. This could allow us to create personal profiles on each speaker, with a database of past interactions that allow us to inform how to respond to future interactions. We can achieve this with the Picovoice Eagle Speaker Recognition Engine.

• Multi-Turn Conversations: While saying the wake word is a voice activation mechanism for long-running, always-on systems, it becomes cumbersome when we have to go back-and-forth with the assistant during a prolonged multi-turn conversation. We could switch to a different form of activation after the initial interaction, to smooth conversations out. Using the Picovoice Cobra Voice Activity Detection Engine, we could simply detect when the speaker is speaking and when they are waiting for a response.