IBM Watson Benchmark

The IBM Watson Natural Language Understanding service is a cloud offering that is capable of extracting metadata from text. For speech input, Watson Speech to Text can be used to transcribe audio files. For domain-specific contexts, the customization interface can be used to create a custom language model to improve speech recognition performance.

Prerequisites

• Ubuntu 20.04 (x86_64)
• Git
• Python
• PIP
• IBM Watson Account

Usage

1. Clone the repository:

git clone https://github.com/Picovoice/speech-to-intent-benchmark.git

2. Install the dependencies:

pip3 install -r requirements.txt

3. Create a NLU service.

4. Create a standard plan Speech to Text service.

5. Create a Knowledge Studio service and create a new Workspace.

6. In your new Workspace, upload the previously created type system data/watson/entity_types.json.

7. In the Rules page under Rule-based Model, create a class for each entity type.

8. In the Dictionaries page, import data/watson/barista_dictionaries.zip. Select the corresponding entity type and corresponding rule-class for each dictionary.

9. In the Versions page, go to the Rule-based Model Type Mapping tab and map each entity type to the corresponding class.

10. Return to the Rule-based Model page and save for deployment. You should see a model with version number 1.0. Deploy this model to Natural Language Understanding, and take note of your model ID.

11. Run the benchmark:

python3 src/bench.py \
--engine IBM_WATSON \
--noise cafe \
--ibm_watson_model_id ${YOUR_MODEL_ID} \
--ibm_watson_stt_apikey ${YOUR_STT_API_KEY} \
--ibm_watson_stt_url ${YOUR_STT_URL} \
--ibm_watson_nlu_apikey ${YOUR_NLU_API_KEY} \
--ibm_watson_nlu_url ${YOUR_NLU_URL}

python3 src/bench.py \
--engine IBM_WATSON \
--noise kitchen \
--ibm_watson_model_id ${YOUR_MODEL_ID} \
--ibm_watson_stt_apikey ${YOUR_STT_API_KEY} \
--ibm_watson_stt_url ${YOUR_STT_URL} \
--ibm_watson_nlu_apikey ${YOUR_NLU_API_KEY} \
--ibm_watson_nlu_url ${YOUR_NLU_URL}

Result