picoLLM Inference Engine
Python Quick Start

Platforms

• Linux (x86_64)
• macOS (x86_64, arm64)
• Windows (x86_64, arm64)
• Raspberry Pi (3, 4, 5)

Requirements

• Picovoice Account & AccessKey
• Python 3.9+
• PIP

Picovoice Account & AccessKey

Signup or Login to Picovoice Console to get your AccessKey. Make sure to keep your AccessKey secret.

Quick Start

Setup

1. Install Python 3.

2. Install the picollm Python package using PIP:

pip3 install picollm

3. Download a picoLLM model file (.pllm) from Picovoice Console.

Usage

1. Create an instance of the inference engine:

import picollm

pllm = picollm.create(
    access_key='${ACCESS_KEY}',
    model_path='${MODEL_PATH}')

2. Generate a prompt completion:

res = pllm.generate(prompt='${PROMPT}')
print(res.completion)

3. To interrupt completion generation before it has finished:

pllm.interrupt()

4. When done, be sure to release the resources explicitly:

pllm.delete()

Vision models

To run a VLM such as qwen3-vl-2b-it:

res = pllm.generate_with_image(
    prompt='${PROMPT}',
    image_width=${IMAGE_NUM_PIXELS_WIDTH},
    image_height=${IMAGE_NUM_PIXELS_HEIGHT},
    image=${IMAGE_DATA});
print(res.completion)

Replace ${PROMPT} with a text prompt. For the image, you will need to get image height and width in number of pixels and the raw pixel values of the image in 8-bit, RGB format.

OCR models

To run an OCR (Optical Character Recognition) model such as deepseek-ocr-2:

res = pllm.generate_ocr(
    image_width=${IMAGE_NUM_PIXELS_WIDTH},
    image_height=${IMAGE_NUM_PIXELS_HEIGHT},
    image=${IMAGE_DATA});
print(res.completion)

For the image, you will need to get image height and width in number of pixels and the raw pixel values of the image in 8-bit, RGB format.

Embedding models

To run an embedding model such as embeddinggemma-300m:

res = pllm.generate_embeddings(prompt='${PROMPT}');
for embedding in range(len(res)):
  print(embedding)

Replace ${PROMPT} with a text prompt that you want to generate embeddings for.

Demo

For the picoLLM Python SDK, we offer demo applications that demonstrate how to use it to generate text from a prompt or in a chat-based environment.

Setup

Install the picollmdemo Python package using PIP:

pip3 install picollmdemo

This package installs command-line utilities for the picoLLM Python demos.

Usage

Run the demo by entering the following in the terminal:

picollm_demo_completion --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console, ${MODEL_PATH} with the path to a model file downloaded from Picovoice Console, and ${PROMPT} with a prompt string.

To get information about all the available options in the demo, run the following:

picollm_demo_completion --help

For more information on our picoLLM demos for Python or to see a chat-based demo, head over to our GitHub repository.

Resources