Every day, more people are adopting AI. Whether it be for work or personal use, users are demanding a lot from artificial intelligence to augment their lives and workflows. One of the key demands for AI adoption is speed. To satisfy current and future needs for applications that demand high speed, such as customer service, tokens/second (the general measurement for a large language model’s speed) throughput is being addressed at the hardware level.
While several startups and scale-ups are receiving million/billion dollar fundraising rounds to build the next breakthrough models, a few companies focused on getting close to the metal are creating businesses with strong intellectual property value. These businesses aim to capitalize on current trends and demands for AI, while creating a new market for businesses that will be founded on the ability to leverage high-throughput models. Strap in as we learn about these businesses looking to break record speeds in large language model inference.
Groq
I’ve had the pleasure of meeting Groq founder Jonathan Ross. Ross was responsible for developing the TPU at Google, where his knack for hardware unlocked the commercial possibility for high speed machine learning applications. After leaving Google, Ross founded Groq in 2016. Since its inception, Groq has received investments from Tiger Global Management, TDK Ventures, and notably Chamath Palihapitya’s Social Capital. According to Artificial Analysis, Groq is the current leader in throughput for all API providers.
Groq went viral earlier this year after acquiring cloud partner Definitive Intelligence, allowing Groq to serve open source LLMs to developers around the world. Currently, Groq serves over 300,000 developers, consistently launching up-to-date open source LLMs on their GroqCloud platform, such as Meta’s Llama 3.1 family of models. Try Groq’s lightning fast inference speed chatbot, GroqChat. I’ve been able to produce and replicate prompts resulting in over 1000+ tokens/s.
Groq’s core business is offering a usage based API through its GroqCloud platform to startups and smaller enterprises. Additionally, Groq is working on obtaining its FedRAMP certification to onboard government and public sector clients who may rely on Groq’s hardware to provide mission-critical services.
Etched
Similar to Groq, Etched has moved to Silicon Valley and raised a $120M Series A to build their Sohu chip. CEO Gavin Uberti is hedging that the current transformer architecture will persist. Investors include Peter Thiel and Stanley Druckenmiller. Druckenmiller, whose known expertise is finance, has ventured to invest in several AI-focused companies, such as Perplexity. Etched is a noticeably younger company, both in terms of time and the age of the team, but has developed strong partnerships with clients who have demand for their product. Etched has partnered with TSMC to develop their chips, which already have million of dollars in anticipated orders. Etched does not yet have a demo to showcase their hardware, but anticipate that it will arrive in the near future.
Coming Up Next
Today was an outlook on the future of hardware necessary to meet the demand in speed that users want from AI. Next, we will discuss the updates in models that have been released throughout 2024. Notably, OpenAI has brought multi-modal models to the forefront, while Elon Musk’s xAI has begun training on a 100,000 NVIDIA H100 cluster. Also, we will discuss theory on model pre-training and how Meta is leading against open source competition.
Sources
Artificial Anaylsis, https://artificialanalysis.ai/leaderboards/providers
GroqChat, https://groq.com/
TechCrunch, https://techcrunch.com/2024/06/25/etched-is-building-an-ai-chip-that-only-runs-transformer-models/