Google is ending support for a widely used browser benchmark after determining that the test had actually become detrimental to web performance.
Coding to the test actually hurt real-world performance
Part of that is simply that the test was designed for where the web was five years ago. Browsers are now efficient at what Octane was designed to test, and at the same time, websites have started being built in different ways.
Octane, Google writes, “didn’t capture important use cases for the modern web” and often doesn’t work “the way real-world code does.”
On top of that, developers were essentially cheating to get higher scores on the test, even though it resulted in worse real-world performance. In one case, Google says, developers began taking advantage of a bug that gave a 15 percent performance bump in Octane, even though it had zero effect on the actual web.
Google warns against relying on benchmarks too much
The team is right that this isn’t a problem unique to Octane. While benchmarking tools are helpful, there’s always been a distinct cap on their usefulness, especially as they age.
In one example, in 2013, major Android phone manufacturers were caught rigging their phones to over-perform when running benchmarks, leading consumers to believe they may have been more capable in daily use than they actually were.
Google isn’t saying benchmarks should be put aside altogether, but its announcement this week is an important reminder that benchmarks can easily outlive their usefulness.