Benchmark Blindness - The Land Mines Ahead - AI and Real Business
I think tech benchmarks miss the point, or distract from it a bit.
The real economics and national competitiveness matter more.
Benchmarks
There have been a lot of comments about Kimi K3 and how it is a huge leap forward, even as good as Anthropic's Fable 5 in some areas (benchmarks).
Also comments that in some cases it is only half the cost of competing platforms, not the 20x we have seen before. As if half teh cost is a bit of a let down.
Post AI Super Hype Early Phase Business Will Choose Economic Value
In the real business world, paying twice as much for a slightly better anything is a bad idea and hard to sell. The extra marginal impact you get from the very best leading-edge model over a close competitor is negligible, while a top open-source (China) model can be 60x cheaper.
You Are Not Buying a Thing, You Are Buying an Ongoing Partnership
Also, you are not buying a set thing at a point in time. You are making a commitment to a development shop for what they have now and what it will evolve into over the next couple of years.
Top Chinese models were 9 months behind; now they are 2-3 months behind. Where does that trend end up in 2027? Also, Chinese firms don't have a huge speculative capital bubble hanging over their heads. Nor an almost unimaginable capital consumption rate. Oh, and minor detail, no shortage of power. China added more power, at a lower cost structure (wind/solar) to their grid in one month than teh US did in a year.
It's Enterprise IT and IT's Fault
Historically, IT vendors were forced by enterprises to do a feature-function bake-off, and enterprise procurement shops were generally not capable of looking at real lifetime value/cost structure. Layer on top of that, the technologists building these things largely think they are competing with other technologists to see who is the leader, rather than the fact that they are really competing with the sales teams from the other companies.
So you get benchmark wars. Which kind of misses the point and flies in the face of the entire history of technology, whether for business or warfighting. The real game is about impact. I can shoot a tank with a $178,000 Javelin, at which point the enemy knows exactly where I am and then can try to kill me, or I can use a $22,000 drone. We know how that turns out. 65% of Russia's tanks are dead, and the others have about 10% utilization because if they wander onto a battlefield, a drone shows up. Congrats, Ukraine, who have brilliantly extended cost structure thinking the entire ecosystem through supply chains and manufacturing and even unto product development cycle times.
Leading Edge Capabilities
While the benchmark win is impressive, as are the leading-edge capabilities, that is really not what you use most. Most AI platforms are workhorses for your enterprise. Does anyone know anyone who uses all the leading-edge functions in Excel or PowerPoint?
The top five most common hospital surgeries add up to 5 million a year. They also do 1000 Orbitozygomatic Craniotomies. They don't benchmark each other on who is best at Orbitozygomatic Craniotomy.
Cost Structure and Impact
Those are the factors that will win the AI battle, not who is temporarily marginally ahead of the other. Although again the recent trends indicate Chinese open-source models may well pull ahead. Chinese models can also be 20x+ cheaper to train. They are also making great progress on cheaper chip substructures to run all of this.
Conclusion - Land Mines Ahead
I do love a good benchmark like the one below that gets extra points for the Salvador Dali-esque Pelican on a Bike part (yes, pelican on a bike). We definitely do need tech benchmarks and smart trend data.
But in the 'love of tech' and the tech-versus-tech frenzy, which is typical of the early speculation-based economics pre-bubble explosion in new tech waves, we have lost sight of the fact that the real battle is about cost structure and impact. If you have not read the genius economist Carolotta Perez (Cambridge) on tech history and speculative bubbles, you should.
One last canary-in-the-coal-mine bit: 80% of startups in an a16z survey opted to use Chinese open-source models.
Cost structure and impact.
