Overfitting and Underfitting are modeling errors in statistics. Artificial Intelligence algorithms are mathematical prediction models. They learn to classify or predict data accurately after being trained on a sample dataset, also called training data. Overfitting or Underfitting causes AI models to fail at classifying or predicting accurately. While they are technical problems, product managers and executives should also be aware of these concepts in order to make informed purchase or development decisions.
What’s Overfitting?
Overfitting occurs when a mathematical model fits too closely to the training dataset. Machine learning models are trained on sample datasets. When the model is too complex or trained for too long on the sample dataset, it fits too closely to the training data sets, losing the ability to generalize the models. As a result, the model cannot perform accurately when unseen data is introduced.
What’s Underfitting?
Underfitting happens when a model is unable to capture the relationship between the input and output accurately, generating a high error rate on both the training set and unseen data.
In case of overtraining or model complexity that causes overfitting, the intuitive reaction is to pause training earlier or to reduce the model complexity by eliminating less relevant inputs. However, stopping too early or excluding important features causes Underfitting.
Overfitting vs. Underfitting
Overfitting and Underfitting happen when machine learning models cannot establish a dominant trend within the training dataset.
In the case of Overfitting, we see a low error rate when tested on the training data and high error rates when tested on a new dataset. Underfitting returns high error rates on both training and test datasets as the models fail regardless of the data set, hence less variance.
To overcome Underfitting, researchers continue to train the model and introduce more inputs. As Underfitted models learn more, their error rate decreases and variance increases. Machine learning researchers aim to find the “sweet spot” between Underfitting and Overfitting so that models can find a dominant trend to predict or classify unseen data accurately.
Why should you care about overfitting and underfitting?
Development teams may underestimate the cost of Overfitting and Underfitting while comparing buy, open-source, and build options. Finding the balance between Overfitting and Underfitting requires expertise and a diverse dataset, increasing the hidden costs of open-source and build options. It’s one of the reasons why enterprises choose to buy. However, Overfitting is still a crucial concept to understand before making the purchase decision.
Evaluating the accuracy of AI models is a part of the purchase process. There are several approaches to it:
- Compare the accuracy metrics provided by AI vendors
- Provide test data to the AI vendors and ask them to run a benchmark
- Run own benchmark using own data
The first approach is unreliable as most vendors only disclose the results if not qualitative comments, such as “the best” or “the most accurate,” not the data sets. Since vendors have control over both test data and the model, they can cherrypick the results. Similarly, the second approach is also unreliable because vendors can easily Overfit their models using the test data provided by prospects before running the tests. Only the third approach gives enterprises control over the data, model, and environment to test alternative models.
If you’re not sure how to compare AI models, check out Picovoice’s open-source benchmark frameworks or reach out to Picovoice Consulting to get a reproducible benchmark developed for you.
Consult an Expert
