Amazon Transcribe Streaming Benchmark

Prerequisites

• Ubuntu 20.04 (x86_64)
• Git
• Python
• PIP
• AWS Account

Usage

Setup

1. Clone the repository:

git clone https://github.com/Picovoice/speech-to-text-benchmark.git

2. Install the dependencies:

pip3 install -r requirements.txt

Word Accuracy Benchmark

1. Run the benchmark:

python3 benchmark.py \
--engine AMAZON_TRANSCRIBE_STREAMING \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--aws-profile ${AWS_PROFILE} \
--aws-location ${AWS_LOCATION}

• To measure the Punctuation Error Rate (PER) add --punctuation to the command.

Word Emission Latency Benchmark

1. Follow the instructions here to generate word timing information.

2. Run the benchmark:

python3 benchmark_latency.py \
--engine AMAZON_TRANSCRIBE_STREAMING \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--aws-profile ${AWS_PROFILE} \
--aws-location ${AWS_LOCATION}