GPTQ - LLM Compression Benchmark

Prerequisites

• Git
• Python 3.8+
• PIP

Usage

1. Clone the repository:

git clone https://github.com/Picovoice/llm-compression-benchmark.git

2. Install the dependencies:

pip3 install -r requirements.txt

3. Measure the MMLU score for a given model:

python3 mmlu.py \
--compression GPTQ \
--model-uri ${MODEL_URI}

4. Measure the ARC score for a given model:

python3 arc.py \
--compression GPTQ \
--model-uri ${MODEL_URI}

5. Measure the perplexity score for a given model:

python3 perplexity.py \
--compression GPTQ \
--model-uri ${MODEL_URI}

Where ${MODEL_URI} is the URI of the model to benchmark.