Battle Brewing Over Intel-ARM Benchmarking

There is a new fight brewing over a recent report from ABI Research, which claimed that Intel's most powerful "Clover Trail+" System on a Chip (SoC) for mobile devices is a better performer than ARM-based chips from Qualcomm, Nvidia, and Samsung, even while using significantly less power.

In early June, the research firm published the results of benchmark testing that pitted a Lenovo K900 smartphone powered by a dual-core, 2-GHz Atom Z2580 and featuring Intel's XMM6360 modem chipset against this quartet of rival Android handsets featuring ARM-based processors:

• The Samsung Nexus 10 ($279.99 at Amazon) (dual-core, 1.7-GHz Samsung Exynos 5250 SoC with ARM A15 cores)
• The Samsung Galaxy S4 i9500 (eight-core, 1.6-GHz Samsung Exynos Octa SoC with ARM A15 and ARM A7 cores)
• The Samsung Galaxy S4 i377 (quad-core, 1.9-GHz Qualcomm Snapdragon APQ8064T SoC with ARM-based Krait 300 cores)
• The Asus Nexus 7 (quad-core, 1.3GHz Nvidia Tegra 3 SoC with ARM A9 cores)

ABI Research reported that for its CPU test, Intel's x86-based, Clover Trail+ chipset was only challenged in terms of performance by the Samsung Exynos Octa and the Qualcomm Snapdragon APQ8064T, with the Atom Z2580 scoring a 5,547, the Exynos Octa scoring a 5,277, and the APQ8064T scoring a 5,378. The Exynos 5250 scored a 3,104 on the CPU test while Nvidia's Tegra 3 scored a 2,886.

Significantly, the ABI Research benchmarkers found that Intel's Atom Z2580 CPU used a lot less power to match its two closest rivals on performance, "with only 0.85A of average current versus 1.38A for the Samsung Exynos Octa and 1.79A for the Qualcomm APQ8064T."

Benchmarking of other chipset functions, including RAM, display, 2D and 3D graphics, etc., produced similar results, according to ABI Research. In almost all cases, either the Intel chipset easily bested the competition on pure performance or matched it while beating rival chips on current consumption.

If so, these results could prove very significant. If Intel's top Clover Trail+ products are matching the most powerful ARM-based SoCs available in performance while also providing longer battery life, we may soon see a major shift in the power structure of the market for smartphone chips.

And just wait until Intel releases its even more advanced, 22-nanometer "Bay Trail" products with a 64-bit instruction set later this year, right?

Not so fast, say some interested observers who this week published criticisms of the ABI Research report. Jeff Bier of BDTI and Jim McGregor of TIRIAS Research both had some nits to pick with the ABI Research team's methodology, starting with a failure to identify the benchmarking tools used for its testing.

Continue Reading: The Trouble With Benchmarking>

The Trouble With Benchmarking

Bier, McGregor, and others think ABI Research used the AnTuTu Android benchmark suite, "a smartphone benchmark app popular with consumers," Bier noted. The BDTI analyst said his team recently looked at the Lenovo K900's Atom Z2580 and Samsung S4 i9500's Exynos Octa using code taken from the older NBench benchmarking suite, which is the basis of the RAM test in AnTuTu.

Bier said BDTI discovered something odd—the NBench-based compiler "removed a key element of the benchmark" when testing the Atom Z2580 but left it in for the Exynos Octa. In short, the Atom Z2580 was allowed to "skip some steps" during the RAM benchmarking test while the Exynos Octa was required to "perform all the operations specified in the benchmark source code."

That's just one instance of a potential flaw in ABI Research's methodology, but it could sour the validity of a "benchmark that purports to compare processors in an apples-to-apples manner," Bier claimed.

On the other hand, the BDTI analyst said the removal of some testing requirements for Intel's SoC by the NBench-based compiler, while unfair in terms of laboratory benchmarking, could be "a good thing in real-world application development."

In a post for EE Times, McGregor said he also had problems with ABI Research making a "blanket statement [that] essentially proclaims that Intel has surpassed the entire ARM ecosystem in mobile processors for the all-important, high-end smartphone segment."

Describing benchmarking in general as "plagued with many issues," such as being "subject to manipulation" by vendors, the TIRIAS Research analyst listed a number of specific problems associated with testing mobile processors like the Atom Z2580.

"The first is the complexity and variety of mobile processors and devices. No two mobile processors or devices are designed alike. The second issue is that it is difficult to test for actual usage models. Smartphones, for example, are used for a variety of functions including communication, content creation, and entertainment," McGregor noted.

"In addition, this usage varies, not only between individuals, but for each individual depending on his or her current requirements. And usage models are continuously changing with new applications, content, and devices."

In the end, this dustup may simply underscore the need for a wider array of benchmarking tools for mobile devices, one that matches the robustness of testing resources for the more mature PC market, said Pat Moorhead, principal analyst at Moor Insights & Strategy.

"Benchmarking controversies aren't anything new, just look at BAPCO Sysmark and MobileMark. These same issues are moving from the PC market to the mobile market. The bottom line is that the mobile industry needs more industry-standard benchmarks that better represent what users want to do with their devices," Moorhead said.