Natural Language Understanding (NLU) is a subtopic of Natural Language Processing. It focuses on “comprehension”. NLU deals with users’ intents and what they mean instead of what they say. Thus, some people refer to it as Intent Detection or Intent Detector.
Initial studies on NLU started in the mid-60s at MIT, Bobrow’s Ph.D. dissertation followed by Weizenbaum’s ELIZA, a mock psychotherapist chatbot in 1966. Then Winograd built a more sophisticated system SHRDLU in 1971, again at MIT.
How does NLU work?
NLU analyses text input to understand what humans mean by extracting Intent and Intent Details. Then NLU matches what it reads with what corresponds in the training data.
For example, when NLU reads:
"show me sneakers"
It breaks it into Intent and Intent Details, such as Entity:
Intent: "filter"
Entity - category: "sneakers"
However, this information should exist in the training data. NLU brings up whatever it has in the training data. When the training data does not have the exact corresponding Intent and Intent Details, NLU cannot comprehend them accurately.
Sometimes, this mismatch leads to funny conversations between machines and humans. Below is a snippet of a conversation between the Late Night Show host Stephen Colbert and Siri in its early days. Yet, this mismatch further frustrates already-frustrated customers when NLU doesn’t perform in enterprise applications.
Five most commonly used NLU terms
1. Conversational AI
Conversational AI focuses on enabling interactions between machines and humans. In other words, Conversational AI applications imitate human intelligence and have dialogues with them. NLU is a crucial component of Conversational AI. When machines do not understand humans properly, humans do not continue with the conversation. Along with accuracy, human-centered and iterative product design principles are critical for the success of Conversational AI applications such as chatbots and voicebots.
2. Corpus
A Corpus is a large collection of machine-readable texts from natural language. A Corpus consists of anything based on written or spoken language, from newspapers, recipes, podcasts or even social media posts. A Corpus may include other data formats as well. For example, Corpus for image recognition has images such as drawings linked to the texts.
3. Utterance
Cambridge dictionary defines Utterance as “something that someone says.” It refers to the smallest unit of speech with a clear beginning and ending. Yet, it doesn’t have to be a complete sentence. NLU processes an Utterance, a user’s input, and interprets it to understand its meaning.
4. Intent
Intents capture the general meaning of an utterance. The Intent of the Utterances “show me sneakers" and “I want to see running shoes” is the same. The user intends to “see” or “filter and retrieve” certain products.
5. Entities
Understanding Intents is just one part of the problem. Extracting details and understanding choices are as vital as understanding intents. Entities are also known as Slots or Intent Details. Despite the same intent, the utterances “show me sneakers" and “I want to see running shoes” have different Entities: “sneakers” and “running shoes.”
For more technical and academic information on NLU, Stanford’s Natural Language Understanding class is a great source. Check the articles comparing NLU vs. NLP vs. NLG and NLU vs. SLU or learn more about LLMs and LLM applications. Don’t forget to review the buyer’s NLU guide and comparison of top NLU software before making a decision.
