Hitting the weights

Models are getting better as companies are doing the job upfront to feed improved datasets prior to training. LLama 3.1 models, released in late July, are applauded for taking the lead of open source. A blog post from Meta AI on the improvements of pre-training:

Compared to previous versions of Llama, we improved both the quantity and quality of the data we use for pre- and post-training. These improvements include the development of more careful pre-processing and curation pipelines for pre-training data, the development of more rigorous quality assurance, and filtering approaches for post-training data.

As expected per scaling laws for language models, our new flagship model outperforms smaller models trained using the same procedure. We also used the 405B parameter model to improve the post-training quality of our smaller models.

Foundational providers are paying for domain experts and generalists to annotate data prior to model training. Let’s take a look at the mission statement and job description of an ‘AI Tutor’ posted by Elon Musk’s xAI:

xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. As an AI Tutor, you will be responsible for generating consistently high-quality and accurately labeled data through various methods to facilitate the training of NLP algorithms. Collaborating closely with the technical staff, you will create datasets for model training, benchmarking, and overall advancement. You will play a crucial role in the improvement of our products and services by ensuring high-quality and accurately labeled data.

What this equates to is models that are pruned to include only high quality pre-training data. The jump from GPT-2 to GPT-3 was due to the addition of web corpus data, allowing the model to learn on a massive amount of data. However, we know that the reliability of sources vary. Google notably blundered early releases of their Gemini-powered search feature due to low quality data attributed to sources, like Reddit.

If instead, models are pre-trained only on high quality data, such as those produced from authoritative sources: academia, finance, programming, literature, the results may skew toward providing more efficient areas to productivity-oriented questions.

Mixing Modality

Early this Summer, OpenAI announced GPT-4o (Omni), which is a model capable of understanding text, speech, and imagery. A combination model was inevitable, but this is the first of the foundational providers to offer this capability to the public. Developers are able to leverage multi-modal for new products and markets, such as voice AI or image recognition. Currently, these capabilities are being leveraged for customer service and text/data extraction from documents. There are more up-and-coming business opportunities for startups to get creative and leverage these new models.

Open Source

Meta leads in providing high quality models to developers and researchers looking to adapt models to their uses. The Llama 3.1 family competes with foundational model providers on capability (Artificial Analysis). This means that without having to front the costs of API usage from foundational providers, nascent teams can generate high quality products at no cost.

Conclusion

In the first update, we discussed innovations at the hardware level, allowing teams to go faster. In this discussion, we discuss the improvements in model capabilities. Let’s combine the two. Imagine a product leveraging an accelerated architecture, like Groq, and a superb open source model, such as Llama 3.1. This unlocks software that can function at the speed of thought, providing new breakthroughs in end-user capability.

While current software can be served at high speeds in the cloud, it rarely provides enhanced productivity speed to users. Even if Excel 365 is able to run anytime, anywhere, it still takes users the same time to write their functions and format their data as it did upon release.

However, with LLM inference occurring at lightning speeds, LLM-enabled software can run 24/7, providing new insights over mass amounts of data. Take another look at my introduction to agents to get a better understanding of how this works. There is still much to be discovered in use-cases for LLMs beyond chatbots and small features. Similar to breakthroughs from the capabilities of web and mobile, artificially intelligent software will inevitably find its groove.