picoLLM Inference Engine
Python Quick Start

Platforms

• Linux (x86_64)
• macOS (x86_64, arm64)
• Windows (x86_64)
• Raspberry Pi (4, 5)

Requirements

• Picovoice Account & AccessKey
• Python 3
• PIP

Picovoice Account & AccessKey

Signup or Login to Picovoice Console to get your AccessKey. Make sure to keep your AccessKey secret.

Quick Start

Setup

1. Install Python 3.

2. Install the picollm Python package using PIP:

pip3 install picollm

3. Download a picoLLM model file (.pllm) from Picovoice Console.

Usage

1. Create an instance of the inference engine:

import picollm

pllm = picollm.create(
    access_key='${ACCESS_KEY}',
    model_path='${MODEL_PATH}')

2. Generate a prompt completion:

res = pllm.generate(prompt='${PROMPT}')
print(res.completion)

3. To interrupt completion generation before it has finished:

pllm.interrupt()

4. When done, be sure to release the resources explicitly:

pllm.delete()

Demo

For the picoLLM Python SDK, we offer demo applications that demonstrate how to use it to generate text from a prompt or in a chat-based environment.

Setup

Install the picollmdemo Python package using PIP:

pip3 install picollmdemo

This package installs command-line utilities for the picoLLM Python demos.

Usage

Run the demo by entering the following in the terminal:

picollm_demo_completion --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console, ${MODEL_PATH} with the path to a model file downloaded from Picovoice Console, and ${PROMPT} with a prompt string.

To get information about all the available options in the demo, run the following:

picollm_demo_completion --help

For more information on our picoLLM demos for Python or to see a chat-based demo, head over to our GitHub repository.

Resources