picoLLM - LLM Compression Benchmark

Prerequisites

• Git
• Python 3.8+
• PIP

Usage

1. Clone the repository:

git clone https://github.com/Picovoice/llm-compression-benchmark.git

2. Install the dependencies:

pip3 install -r requirements.txt

3. Measure the MMLU score for a given model:

python3 mmlu.py \
--compression picoLLM \
--picollm-access-key ${PICOLLM_ACCESS_KEY} \
--model-uri ${MODEL_URI}

4. Measure the ARC score for a given model:

python3 arc.py \
--compression picoLLM \
--picollm-access-key ${PICOLLM_ACCESS_KEY} \
--model-uri ${MODEL_URI}

5. Measure the perplexity score for a given model:

python3 perplexity.py \
--compression picoLLM \
--picollm-access-key ${PICOLLM_ACCESS_KEY} \
--model-uri ${MODEL_URI}

Where ${MODEL_URI} is the URI of the model to benchmark. You must replace ${PICOLLM_ACCESS_KEY} with your Picovoice Access Key obtained from the Picovoice Console.