picoLLM Inference Engine
.NET API

API Reference for the picoLLM .NET SDK (NuGet).

namespace: Pv

PicoLLM.Create()

public static PicoLLM Create(
    string accessKey,
    string modelPath,
    string device = null)

Factory method for picoLLM Inference Engine.

Parameters

• accessKey string : AccessKey obtained from Picovoice Console.
• modelPath string : Absolute path to the file containing LLM parameters.
• device string : String representation of the device (e.g., CPU or GPU) to use for inference. If set to best, picoLLM picks the most suitable device. If set to gpu, the engine uses the first available GPU device. To select a specific GPU device, set this argument to gpu:${GPU_INDEX}, where ${GPU_INDEX} is the index of the target GPU. If set to cpu, the engine will run on the CPU with the default number of threads. To specify the number of threads, set this argument to cpu:${NUM_THREADS}, where ${NUM_THREADS} is the desired number of threads. If set to null, best device will be used.

Returns

• PicoLLM : An instance of picoLLM Inference Engine.

Throws

• PicoLLMException

PicoLLM.GetAvailableDevices()

public static string[] GetAvailableDevices()

Lists all available devices that picoLLM can use for inference. Each entry in the list can be used as the device argument of the .create() factory method or the PicoLLM constructor.

Returns

• string[] : List of all available devices that picoLLM can use for inference.

Throws

• PicoLLMException

PicoLLM

public class PicoLLM : IDisposable { }

Class for the picoLLM Inference Engine.

PicoLLM can be initialized either using the static constructor create(). Resources should be cleaned when you are done using the delete() method.

PicoLLM.Model

public string Model { get; private set; }

Getter for model's name.

Returns

• string : Model name.

PicoLLM.ContextLength

public int ContextLength { get; private set; }

Getter for model's context length.

Returns

• int : Context length.

PicoLLM.Version

public string Version { get; private set; }

Getter for version.

Returns

• string : Version string.

PicoLLM.MaxTopChoices

public int MaxTopChoices { get; private set; }

Getter for maximum number of top choices.

Returns

• int : Maximum number of top choices.

PicoLLM.Dispose()

public void Dispose()

Releases resources acquired by picoLLM.

PicoLLM.Generate()

public PicoLLMCompletion Generate(
    string prompt,
    int? completionTokenLimit = null,
    string[] stopPhrases = null,
    int? seed = null,
    float presencePenalty = 0,
    float frequencyPenalty = 0,
    float temperature = 0,
    float topP = 1,
    int numTopChoices = 0,
    Action<string> streamCallback = null)

Given a text prompt and a set of generation parameters, creates a completion text and relevant metadata.

Parameters

• prompt string : Prompt.
• completionTokenLimit int? : Maximum number of tokens in the completion. If the generation process stops due to reaching this limit, the .endpoint parameter in PicoLLMCompletion output will be PicoLLMEndpoints.COMPLETION_TOKEN_LIMIT_REACHED. Set to null to impose no limit.
• stopPhrases string[] : The generation process stops when it encounters any of these phrases in the completion. The already generated completion, including the encountered stop phrase, will be returned. The endpoint parameter in PicoLLMCompletion output will be PicoLLMEndpoints.STOP_PHRASE_ENCOUNTERED. Set to null to turn off this feature.
• seed int? : The internal random number generator uses it as its seed if set to a positive integer value. Seeding enforces deterministic outputs. Set to null for randomized outputs for a given prompt.
• presencePenalty float : It penalizes logits already appearing in the partial completion if set to a positive value. If set to 0.0, it has no effect.
• frequencyPenalty float : If set to a positive floating-point value, it penalizes logits proportional to the frequency of their appearance in the partial completion. If set to 0.0, it has no effect.
• temperature float : Sampling temperature. Temperature is a non-negative floating-point value that controls the randomness of the sampler. A higher temperature smoothens the samplers' output, increasing the randomness. In contrast, a lower temperature creates a narrower distribution and reduces variability. Setting it to 0 selects the maximum logit during sampling.
• topP float : A positive floating-point number within 0 and 1. It restricts the sampler's choices to high-probability logits that form the top_p portion of the probability mass. Hence, it avoids randomly selecting unlikely logits. A value of 1. enables the sampler to pick any token with non-zero probability, turning off the feature.
• numTopChoices int : If set to a positive value, picoLLM returns the list of the highest probability tokens for any generated token. Set to 0 to turn off the feature. The maximum number of top choices is .max_top_choices.
• streamCallback Action<string> : If not set to null, picoLLM executes this callback every time a new piece of completion string becomes available.

Returns

• PicoLLMCompletion : Completion result.

Throws

• PicoLLMException

PicoLLM.Interrupt()

public void Interrupt()

Interrupts .Generate() if generation is in progress. Otherwise, it has no effect.

Throws

• PicoLLMException

PicoLLM.Tokenize()

public int[] Tokenize(
    string text,
    bool bos,
    bool eos)

Tokenizes a given text using the model's tokenizer. This is a low-level function meant for benchmarking and advanced usage. .Generate() should be used when possible.

Parameters

• text string : Text.
• bos bool : If set to true, the tokenizer prepends the beginning of the sentence token to the result.
• eos bool : If set to true, the tokenizer appends the end of the sentence token to the result.

Returns

• int[] : Tokens representing the input text.

Throws

• PicoLLMException

PicoLLM.Forward()

public float[] Forward(int token)

Performs a single forward pass given a token and returns the logits. This is a low-level function for benchmarking and advanced usage. .Generate() should be used when possible.

Parameters

• token int : The input token for the forward pass.

Returns

• float[] : The logits resulting from the forward pass.

Throws

• PicoLLMException

PicoLLM.Reset()

public void Reset()

Resets the internal state of LLM. It should be called in conjunction with .Forward() when processing a new sequence of tokens. This is a low-level function for benchmarking and advanced usage. .Generate() should be used when possible.

Throws

• PicoLLMException

PicoLLM.GetDialog()

public PicoLLMDialog GetDialog(
    string mode = null,
    int? history = 0,
    string system = null)

Return the PicoLLMDialog object corresponding to the loaded model. The model needs to be instruction-tuned and have a specific chat template.

Parameters

• mode string : Some models (e.g., phi-2) define multiple chat template modes. For example, phi-2 allows both qa and chat templates.
• history int? : History refers to the number of latest back-and-forths to include in the prompt. Setting history to null will embed the entire dialog in the prompt.
• system string : System instruction to embed in the prompt for configuring the model's responses.

Returns

• PicoLLMDialog : Constructed dialog object.

Throws

• PicoLLMException

PicoLLMException

class PicoLLMException(Exception)

Error thrown if an error occurs within picoLLM Inference Engine.

Exceptions

class PicoLLMActivationException(PicoLLMException)
class PicoLLMActivationLimitException(PicoLLMException)
class PicoLLMActivationRefusedException(PicoLLMException)
class PicoLLMActivationThrottledException(PicoLLMException)
class PicoLLMIOException(PicoLLMException)
class PicoLLMInvalidArgumentException(PicoLLMException)
class PicoLLMInvalidStateException(PicoLLMException)
class PicoLLMKeyException(PicoLLMException)
class PicoLLMMemoryException(PicoLLMException)
class PicoLLMRuntimeException(PicoLLMException)
class PicoLLMStopIterationException(PicoLLMException)

PicoLLMUsage

public class PicoLLMUsage

Usage information.

• PromptTokens int : Number of tokens in the prompt.
• CompletionTokens int : Number of tokens in the completion.

PicoLLMEndpoint

public enum PicoLLMEndpoint

Reasons for ending the generation process.

• END_OF_SENTENCE : 0
• COMPLETION_TOKEN_LIMIT_REACHED : 1
• STOP_PHRASE_ENCOUNTERED : 2
• INTERRUPTED : 3

PicoLLMToken

public class PicoLLMToken

Generated token and its log probability.

• Token string : Token.
• LogProb float : Log probability.

PicoLLMCompletionToken

public class PicoLLMCompletionToken

Generated token within completion and top alternative tokens.

• Token PicoLLMToken : Token.
• TopChoices PicoLLMToken[] : Top choices.

PicoLLMCompletion

public class PicoLLMCompletion

LLM completion result.

• Usage PicoLLMUsage : Usage information.
• Endpoint PicoLLMEndpoint : Reason for ending the generation process.
• CompletionTokens PicoLLMCompletionToken[] : Generated tokens within completion and top alternative tokens.
• Completion string : Completion string.

PicoLLMDialog

public class PicoLLMDialog

Dialog is a helper class that stores a chat dialog and formats it according to an instruction-tuned LLM's chat template. Dialog is the base class. Each supported instruction-tuned LLM has an accompanying concrete subclass.

PicoLLMDialog.PicoLLMDialog()

protected PicoLLMDialog(int? history = null, string system = null)

Constructor.

Parameters

• history int? : History refers to the number of latest back-and-forths to include in the prompt. Setting history to null will embed the entire dialog in the prompt.
• system string : System instruction to embed in the prompt for configuring the model's responses.

Throws

• PicoLLMInvalidArgumentException

PicoLLMDialog.AddHumanRequest()

public void AddHumanRequest(string content)

Adds a human's request to the dialog.

Parameters

• content string : Human's request.

Throws

• PicoLLMInvalidStateException

PicoLLMDialog.AddLLMResponse()

public void AddLLMResponse(string content)

Adds LLM's response to the dialog.

Parameters

• content string : LLM's response.

Throws

• PicoLLMInvalidStateException

GemmaChatDialog

public class GemmaChatDialog : PicoLLMDialog

Dialog helper for gemma-2b-it and gemma-7b-it.

GemmaChatDialog.Prompt()

public override string Prompt()

Creates a prompt string for gemma-2b-it and gemma-7b-it models.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

Llama2ChatDialog

public class Llama2ChatDialog : PicoLLMDialog

Dialog helper for llama-2-7b-chat, llama-2-13b-chat, and llama-2-70b-chat.

Llama2ChatDialog.Prompt()

public override string Prompt()

Creates a prompt string for llama-2-7b-chat, llama-2-13b-chat, and llama-2-70b-chat models.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

Llama3ChatDialog

public class Llama3ChatDialog : PicoLLMDialog

Dialog helper for llama-3-8b-instruct and llama-3-70b-instruct.

Llama3ChatDialog.Prompt()

public override string Prompt()

Creates a prompt string for llama-3-8b-instruct and llama-3-70b-instruct models.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

Llama32ChatDialog

public class Llama32ChatDialog : PicoLLMDialog

Dialog helper for llama-3.2-1b-instruct and llama-3.2-3b-instruct.

Llama32ChatDialog.Prompt()

public override string Prompt()

Creates a prompt string for llama-3.2-1b-instruct and llama-3.2-3b-instruct models.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

MistralChatDialog

public class MistralChatDialog : PicoLLMDialog

Dialog helper for mistral-7b-instruct-v0.1 and mistral-7b-instruct-v0.2.

MistralChatDialog.Prompt()

public override string Prompt()

Creates a prompt string for mistral-7b-instruct-v0.1 and mistral-7b-instruct-v0.2 models.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

MixtralChatDialog

public class MixtralChatDialog : PicoLLMDialog

Dialog helper for mixtral-8x7b-instruct-v0.1. This class inherits methods from MistralChatDialog.

Phi2Dialog

public class Phi2Dialog : PicoLLMDialog

Dialog helper for phi-2. This is a base class, use one of the mode-specific subclasses.

Phi2Dialog.Phi2Dialog()

protected Phi2Dialog(
    string humanRequestsTag,
    string llmResponsesTag,
    int? history = null,
    string system = null)
    : base(history, system)

Constructor for the Phi2Dialog class.

Parameters

• humanRequestsTag string : Tag for human input.
• llmResponsesTag string : Tag for LLM input.
• history int? : History refers to the number of latest back-and-forths to include in the prompt. Setting history to null will embed the entire dialog in the prompt.
• system string : System instruction to embed in the prompt for configuring the model's responses.

Phi2Dialog.Prompt()

public override string Prompt()

Creates a prompt string for phi-2 model.

Returns

• string : Formatted prompt.

Throws

• PicoLLMInvalidStateException

Phi2QADialog

public class Phi2QADialog : Phi2Dialog

Dialog helper for phi-2 in qa mode. This class inherits methods from Phi2Dialog.

Phi2ChatDialog

public class Phi2ChatDialog : Phi2Dialog

Dialog helper for phi-2 in chat mode. This class inherits methods from Phi2Dialog.

Phi3Dialog

public class Phi3Dialog : PicoLLMDialog

Dialog helper for phi3.

Phi35Dialog

public class Phi35Dialog : Phi3Dialog

Dialog helper for phi3.5.