picoLLM Inference Engine
Node.js Quick Start

Platforms

• Linux (x86_64)
• macOS (x86_64, arm64)
• Windows (x86_64)
• Raspberry Pi (4, 5)

Requirements

• Picovoice Account & AccessKey
• Node.js 16+
• npm

Picovoice Account & AccessKey

Signup or Login to Picovoice Console to get your AccessKey. Make sure to keep your AccessKey secret.

Quick Start

Setup

1. Install Node.js.

2. Install the picollm-node npm package:

npm install @picovoice/picollm-node

3. Download a picoLLM model file (.pllm) from Picovoice Console.

Usage

1. Create an instance of the engine:

const { PicoLLM } = require("@picovoice/picollm-node");

const pllm = new PicoLLM(
    "${ACCESS_KEY}",
    "${MODEL_PATH}");

2. Generate a prompt completion:

const res = await pllm.generate("${PROMPT}");
console.log(res.completion);

3. To interrupt completion generation before it has finished:

pllm.interrupt();

4. When done, be sure to release the resources explicitly:

pllm.release()

Demo

For the picoLLM Node.js SDK, we offer a demo application that demonstrates how to use it to generate text from a prompt or in a chat-based environment.

Setup

Install the picoLLM demo package:

npm install -g @picovoice/picollm-node-demo

This package installs command-line utilities for the picoLLM Node.js demos.

Usage

Use the --help flag to see the usage options for the completion demo:

picollm-completion-demo --help

Run the following command to generate text:

picollm-completion-demo --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}

For more information on our picoLLM demos for Node.js or to see a chat-based demo, head over to our GitHub repository.

Resources