Natural Language Understanding (NLU) is a subtopic of
Natural Language Processing. It focuses on “comprehension”.
NLU deals with users’ intents and what they mean instead of what they say. Thus, some people refer to it as
Intent Detection or
Initial studies on
NLU started in the mid-60s at MIT, Bobrow’s Ph.D. dissertation followed by Weizenbaum’s ELIZA, a mock psychotherapist chatbot in 1966. Then Winograd built a more sophisticated system SHRDLU in 1971, again at MIT.
How does NLU work?
NLU analyses text input to understand what humans mean by extracting
Intent Details. Then
NLU matches what it reads with what corresponds in the training data.
For example, when NLU reads:
"show me sneakers"
It breaks it into
Intent Details, such as
Entity - category: "sneakers"
However, this information should exist in the training data.
NLU brings up whatever it has in the training data. When the training data does not have the exact corresponding
NLU cannot comprehend them accurately.
Sometimes, this mismatch leads to funny conversations between machines and humans. Below is a snippet of a conversation between the Late Night Show host Stephen Colbert and Siri in its early days. Yet, this mismatch further frustrates already-frustrated customers when
NLU doesn’t perform in enterprise applications.
Five most commonly used NLU terms
1. Conversational AI
Conversational AI focuses on enabling interactions between machines and humans. In other words,
Conversational AI applications imitate human intelligence and have dialogues with them.
NLU is a crucial component of
Conversational AI. When machines do not understand humans properly, humans do not continue with the conversation. Along with accuracy, human-centered and iterative product design principles are critical for the success of
Conversational AI applications such as chatbots and voicebots.
Corpus is a large collection of machine-readable texts from natural language. A
Corpus consists of anything based on written or spoken language, from newspapers, recipes, podcasts or even social media posts. A
Corpus may include other data formats as well. For example,
Corpus for image recognition has images such as drawings linked to the texts.
Cambridge dictionary defines
Utterance as “something that someone says.” It refers to the smallest unit of speech with a clear beginning and ending. Yet, it doesn’t have to be a complete sentence.
NLU processes an
Utterance, a user’s input, and interprets it to understand its meaning.
Intents capture the general meaning of an utterance. The
Intent of the
Utterances “show me sneakers" and “I want to see running shoes” is the same. The user intends to “see” or “filter and retrieve” certain products.
Intents is just one part of the problem. Extracting details and understanding choices are as vital as understanding intents.
Entities are also known as
Intent Details. Despite the same intent, the utterances “show me sneakers" and “I want to see running shoes” have different
Entities: “sneakers” and “running shoes.”
For more technical and academic information on
NLU, Stanford’s Natural Language Understanding class is a great source. Check the articles comparing NLU vs. NLP vs. NLG and NLU vs. SLU or learn more about LLMs and LLM applications. Don’t forget to review the buyer’s NLU guide and comparison of top NLU software before making a decision.