In the latest battle of the ongoing browser wars, Google has released to the open source community a browser benchmark dubbed RoboHornet that aims to test how well browsers handle Web technologies beyond JavaScript. Though the benchmark is still an alpha release, Microsoft has already dismissed it as a mere "micro-benchmark" that fails to represent Web technologies used in the "real world."
Popular browser benchmarks such as Sunspider, Kraken, and Octane test raw JavaScript performance, "which is rarely the bottleneck we have in our apps," according to Paul Irish, a Google Chrome and Web development community leader. "These days, our performance bottlenecks are oftentimes DOM, <canvas> API methods, [and] SVG. Those are our priorities."
Enter RoboHornet, which according to project lead Alex Komoroske "encompasses all aspects of browser performance and everything that matters to Web developers, like performance of layout and localStorage."
RoboHornet, which is actually a modular suite of performance tests, uses the Benchmark.js framework to run and measure tests. Currently, it's "officially" compatible with the five most popular browsers: Chrome, Firefox, Internet Explorer, Opera (on Windows), and Safari (on OS X). "At this early alpha stage, RoboHornet is not guaranteed to be compatible with mobile browsers," according to the benchmark's wiki. "Working on mobile is an explicit goal of RoboHornet as the importance of performance of mobile browsers continues to grow."
In its current form, RoboHornet benchmarks browser performance for "major pain points" of jQuery, Google Apps, Google Maps, Ember, Handlebars, and Cappuccino. For example, it measures the speed for adding rows and columns to existing tables; converting a 2D canvas to a data URI; resizing SVGs; table-rendering after innerHTML; scrolling using scrollTop; and localStorage read and write performance.
By turning the benchmark over to GitHub and the open source community, RoboHornet becomes "a living, dynamic benchmark that aims to use the collective efforts of the Web development community and ultimately get browser vendors to fix real-world performance pain points," wrote Komoroske.
Developers are welcome to propose other browser pain points to a committee of JavaScript experts from such organizations as JSPerf, YUI, Google, and Facebook. The community votes for which browser features they want the benchmark to support, and if the committee gives its blessing, those items are added to the benchmark.
Though the project is in its early stages, Microsoft has already pooh-poohed it. "Like all micro-benchmarks, RoboHornet is a lab test that only focuses on specific aspects of browser performance," wrote Roger Capriotti, director of Internet Explorer Marketing.
However, the folks behind the RoboHornet project don't consider it a micro-benchmark. "Although RoboHornet's benchmarks are small and specially made, they are different from a 'micro-benchmark' because they are directly motivated by real-world performance pain points and are designed to evolve over time as the browser landscape evolves," according the project's FAQ page.
Microsoft's dismissal of the benchmark didn't stop the company from using it to build its own benchmark, dubbed RoboHornet Pro, which uses "modern browser capabilities like CSS3 Animations, CSS3 Transforms, CSS3 Text Shadows, custom WOFF fonts, Unicode, Touch, and more" to test Web performance "in the context of a real-world scenario," Capriotti wrote.
Surprise, surprise, Microsoft reported that Internet Explorer 10 passed the RoboHornet Pro test with flying colors, while Chrome "[slowed] to a crawl and stops animating the screen, because it wasn't designed to handle a benchmark load in the context of a real-world scenario."
Tom's Hardware, meanwhile, put RoboHornet through its paces in a series of tests of the various supported browsers on Windows 7, Windows 8 RTM, OS X Mountain Lion, Ubuntu 12.04 LTS, and Android. Cutting right to the chase, Chrome beat out IE9, Opera, and Firefox when run on Windows 7 64-bit. IE10 trounced Chrome and company on Windows 8 RTM 64-bit. Safari came out on top when Mac OS X was the platform of choice. On Ubuntu, Chrome was again the winner, topping Opera and Firefox.
The caveat to these results: RoboHornet is still in alpha and already has plenty of reported issues. It will be a matter of time before we know whether it will become a de facto benchmark for browser performance.
This story, "Microsoft dismisses Google's open source browser benchmark," was originally published at InfoWorld.com. Get the first word on what the important tech news really means with the InfoWorld Tech Watch blog. For the latest developments in business technology news, follow InfoWorld.com on Twitter.