Compute-Optimal Large Language Models

Improving the performance of a machine learning model by increasing its size is typically the first and most straightforward approach. As a result, language models have been getting bigger and more cloud-dependent. However, bigger ML models do not always perform better. Training larger models for better performance comes at a cost and is not the most efficient approach.

Larger ML models require powerful GPUs that only a few companies can afford: Big Tech. Yet it is expensive even for them. Alexa was close to losing $10 billion . Although Microsoft GitHub AI Co-pilot is a paid service, users cost Microsoft up to $80 per month . It’s not surprising that Microsoft has started exploring alternative approaches . Amazon has announced its plans to introduce a monthly fee for Alexa, as the cost for inference of the model in the cloud is substantial . Most enterprises cannot afford billions of dollars. Besides, the environmental cost is on all of us. This "bigger is better" approach is exclusive and expensive.

Training Compute-Optimal Large Language Models:

Training Compute-Optimal Large Language Models by Hoffman et al. of DeepMind investigates the optimal model size and number of tokens required to train a transformer model under a given compute budget. The study argues that everyone, including OpenAI, DeepMind, and Microsoft, has been training large language models with suboptimal use of computing resources because a large language model stops learning after the learning rate has fully decayed. Researchers demonstrated this by training a model, Chinchilla with 70B parameters, and showing that it outperforms Gopher with 280B parameters and GPT-3 with 175B parameters. Chinchilla was trained on three to five times as much data as the competitive models had.

Meta’s LLaMA

In February 2023, Meta released LLaMA, the first open-source LLM, which is competitive with commercial alternatives. Meta researchers claimed that the 13B parameter LLaMA model outperforms GPT-3 with 175B parameters on many NLP benchmarks. Later in July, Meta released LLaMA-2 trained on 40% more data than LLaMA and doubled the context length. LLaMA-2 with the same parameters outperformed its predecessors , e.g., Llama-2 with 13B parameters defeats LLaMA with 13B parameters. After Chinchilla, LLaMA has been another example to show the model size is not a good proxy to predict the performance of a large language model. Furthermore, solo developers took LLaMA weights and the Alpaca* training schemes to run their own LLaMA on PCs, phones, and even a Raspberry Pi microcontroller.

These results are not surprising. Picovoice has released efficient voice AI models that outperform the alternatives. For example, Leopard and Cheetah, with 20MB model sizes , are more accurate than large cloud-dependent models of Google Speech-to-Text. If you’re interested in making large language models more efficient, talk to Picovoice Consulting and leverage their expertise!