Aug 1, 2025

August 1, 2025, 6:05 pm IDT The current landscape of AI inference is a “bloodbath in the finance department,” as Robert Wachen, Co-Founder and COO of Etched, starkly put it during his presentation at the AI Engineer World’s Fair. He highlighted a critical juncture where AI models are finally delivering on their promise, yet the underlying infrastructure is struggling to keep pace, leading to unsustainable costs and performance limitations. Wachen elaborated on why the general-purpose hardware paradigm, primarily GPUs, has hit a scaling wall, paving the way for a new era of specialized AI chips. Wachen articulated that when AI models reach a tipping point in quality and cost, an “unstoppable surge of inference demand” follows. This demand is exponential, with the number of tokens processed doubling every three months. However, the compute density of GPUs has shown diminishing returns since 2022, improving by only 15% in three years. General-purpose chips have simply grown larger and more power-hungry, moving from air-cooled to liquid-cooled, and from single to multiple stitched dies. This brute-force scaling is economically unviable, creating a “massive delta between supply and demand” that current hardware cannot bridge. The solution, according to Etched, lies in specialized chips. For the first time in history, the spend on AI models is orders of magnitude greater than the cost of custom chips. Wachen explained that if a company is spending $100 billion on an AI model, a mere 0.1% investment in a custom chip can yield over a thousand percent performance improvement. This shift isn’t merely about silicon; it necessitates a complete rethinking of data center infrastructure, including interconnections, thermal management, and power delivery. Etched’s answer to this challenge is Sohu, touted as the world’s first transformer ASIC. This specialized chip is designed specifically for AI inference, making deliberate trade-offs, such as not supporting high-precision data types or training. These optimizations allow Sohu to deliver remarkable performance: it is over 15 times faster, 10 times cheaper, and consumes 9 times less power for tasks like generating 1 million Llama 70B tokens. Such specialized hardware unlocks previously intractable real-time AI applications. Wachen cited real-time video generation, where serving just one million users with an 8B model would require an astonishing 400,000 H100 chips, costing $10 billion in CapEx. This is simply not sustainable. He also pointed to real-time code generation, where a “latency cliff” means suggestions delivered beyond 100-300 milliseconds are largely ignored, leading to wasted compute. Current solutions involve small, low-quality models or expensive, inefficient low batch sizes. Lastly, enterprise peak load problems, common in cloud computing, are exacerbated by AI’s unpredictable demands, forcing companies to over-provision by hundreds of millions of dollars just to handle intermittent surges. Many enterprises, Wachen noted, “will not roll out this product until this problem is solved.” Etched’s strategy is to specialize in the stable elements of AI models while allowing for flexibility in evolving components. The company boasts a team of over 150 engineers from leading AI and chip companies, having secured supply chain partnerships and sold out its initial production run. This focused approach is positioned to meet the burgeoning, yet currently unaddressable, demand for real-time AI at scale.