The Meltdown and Spectre vulnerabilities still loom over the world’s computer users as companies scramble to mitigate them. Like many PC owners of late, I’ve been biting my nails waiting for Intel to issue stable versions of its Spectre-mitigating microcode updates and for the blue team’s hardware partners to bake them into new firmware for their products.
I’m still waiting on a microcode update for my aging Haswell desktop, but Dell issued a new firmware update for my Alienware 13 R3 last week that promised some Spectre-cide™ goodness for the Core i7-7700HQ inside. Before I patched that system, though, I wanted to record some pre-patch performance numbers to see just how much I was giving up by fully protecting my system against these vulnerabilities (or as fully as possible, at least).
Part of the fear surrounding Meltdown and Spectre, aside from the potential for the leakage of sensitive data, is the potential performance hit from patching those vulnerabilities. Intel has claimed from the beginning that any performance hit from the Spectre and Meltdown patches would “not be significant,” was “highly workload-dependent,” and might change with time as the mitigations for those vulnerabilities were refined. Early tests from some sources foretold potentially large slowdowns for servers, but just like Intel says, the potential slowdowns range from “minimal” to “measurable” depending on workload, according to Red Hat. Those projections are more for data centers running atop Linux than Windows desktops, though.
Microsoft does have some ideas for the kinds of performance hits we should expect on PCs, and its work suggests any performance changes will be CPU-dependent and operating-system-dependent. Redmond expects that systems with Skylake and newer CPUs might experience “single-digit slowdowns,” while “some owners” of Haswell and older parts should expect to notice the slowdowns from these patches. Those projections are for systems running Windows 10, too. Microsoft advises that older versions of Windows on Haswell and older chips will experience a noticeable performance hit.
With all that in mind, my Kaby Lake laptop running Windows 10 should be the best-case scenario for post-Spectre patch performance. Until Intel issues stable microcode updates for Haswell and older CPUs, we won’t be able to gauge just what the effects of Spectre-cide might be for those systems—and that assumes big PC companies and motherboard makers plan to go to the effort of issuing new firmware for their older products in the first place.
One thing is for certain: this story is far from finished. Spectre and Meltdown represent classes of attacks, not a single bug that can be exterminated with a single patch. It’s important to remember that the microcode updates Intel is issuing are mitigations, not curatives. These patches may make Spectre much harder to exploit, but if there’s one group one never wants to underestimate, it’s security researchers. We may yet see new and novel ways of exposing privileged data through processor side channels. For now, whatever margin of safety Intel’s patches claim to afford seems worth having. Let’s see how much performance we’re trading for those safeguards.
Our testing methods
Our test system was configured as follows:
| Alienware 13 R3 | |
| CPU | Intel Core i7-7700HQ |
| Memory | 16 GB (2x 8 GB) DDR4-2666 |
| Graphics card | Nvidia GeForce GTX 1060 6GB |
| Storage | Samsung PM961 512 GB NVMe SSD |
There are a lot of moving parts in testing the performance implications of Spectre and Meltdown between operating system updates and microcode changes. Microsoft has already issued the operating-system-level patches necessary to stop Meltdown cold on affected PCs, though, and I let Windows automatically update on my personal system, so the effects of those changes are already in place whether I like them or not. I’m not particularly concerned about the effects of Meltdown mitigations here, either, since Microsoft calls out Spectre patches as the ones with the largest potential performance impact.
Still, I isolated our before-and-after comparisons to just one change: a Spectre-related firmware patch on an otherwise fully-updated Windows 10 machine. I ran my tests back-to-back, immediately before and immediately after I applied the necessary firmware from Dell. That fact means we should be looking at the performance impact of that one patch and little else.
We tested using Windows’ Balanced power plan. To ensure accuracy, we ran each benchmark at least three times and took the median of the results. Our tests were conducted with the system connected to AC power.
Our testing methods are generally publicly available and reproducible. If you have questions regarding our methods or results, leave a comment on this article or join us in our forums.
JavaScript
First off, we have several synthetic JavaScript benchmarks that give us a good idea of single-threaded performance changes. These benchmarks should be broadly relevant for the speed and responsiveness users enjoy when scrolling through Facebook, Twitter, and other anxiogenics in this modern age.
Across all of our JavaScript results, we get mid- to high-single-digit percentage decreases in performance. The fully-patched Core i7-7700HQ falls 6.7% behind its unfettered self in Jetstream, 9.7% behind in Octane, and 7.3% behind in Speedometer, though Kraken doesn’t seem to be hurt much by the new microcode.
On the whole, a geometric mean of the performance changes we saw suggests a 6.1% loss in these benchmarks. That may not sound like much, but in this day and age, that kind of figure can encompass an entire generational change’s worth of performance for Intel CPUs. To be fair, most recent systems feel sufficiently snappy in day-to-day use, but every little bit helps.
Feeling out the whole hog with PCMark 10 Extended
I also tried to get a sense of the Spectre patch’s effects on my system using Futuremark’s PCMark 10 Advanced suite. That benchmark’s Extended test gathers data on a range of typical desktop tasks, including app start-up times, video conferencing, web browsing, word processing, and spreadsheet manipulation. It also gives us a high-level view of gaming performance, thanks to its integrated Fire Strike graphics and physics tests.
Even better, Futuremark fully details just what its benchmarks actually test instead of spitting out an opaque index value. Thanks to that information, I can spitball about just what’s caused a dip in performance for a given benchmark. My thanks to Futuremark for providing a license key for this handy software.
While PCMark’s gaming tests are no replacement for a test of several real-world applications with frame-time data behind them, they do let us see whether gaming performance potential changes much, if at all, before and after I apply the Spectre microcode patch. Sorry to disappoint, but few games are actually CPU-bound these days at real-world resolutions and settings. Spectre mitigations are unlikely to have a major effect on gaming performance to begin with.
The PCMark 10 Essentials test encompasses app start-up times, video conferencing performance, and web browsing tasks. The biggest concern among these results is the 13.5% drop in app load time scores post-Spectre patch. That result could mean that some PCs could feel significantly more sluggish than before when users go to load software for the first time in a day. Just how much users actually feel the slowdowns PCMark 10 exposes may depend on how often they put their PCs to sleep versus performing cold shutdowns, though. Windows’ SuperFetch caching could also mitigate some of the pain.
Other I/O-heavy interactions with a system might not be as fortunate, but few of our CPU benchmarks (and few applications in general) are actually storage-bandwidth-bound on the desktop. We may need to explore that topic in greater depth separately.
Outside of I/O-bound work, the Essentials video conferencing score fell just a couple of percent, and web browsing performance remained essentially unchanged. For folks who truly don’t lean on their PCs that much, this test suggests that Intel is correct when it says that most users won’t notice a performance impact from its mitigations.
The Productivity portion of the PCMark suite also turns up some big swings pre- and post-Spectre patch. For spreadsheet jockeys like myself, the 9% performance increase in PCMark 10 is welcome, but folks who spend more time in Word (as represented by LibreOffice Writer) might find their text-entry work a little more sluggish post-Spectre patch. Futuremark’s detailed discussion of its word-processing benchmark suggests a fair bit of saving and loading documents is going on behind the scenes. If that work meaningfully involves the SSD in its evaluation, that could explain the performance drop we see in what should otherwise be a fairly lightweight task.
The Digital Content Creation test simulates photo editing, rendering and visualization, and video-editing work. None of these applications show more than a couple of percentage points’ worth of performance decreases pre- and post-Spectre patching, and that’s well within the 3% margin of error that Futuremark suggests is typical for controlled testing with PCMark. Creative pros that aren’t bottlenecked by I/O bandwidth would seem to be largely safe from the side effects of Spectre-cide.
PC enthusiasts might worry about gaming performance the most among the potential performance drops that might arise from Spectre patches. Few games are I/O-bound, however, and Futuremark’s 3DMark Fire Strike Graphics and Physics tests certainly are not. As with the Digital Content Creation test, the gaming performance potential demonstrated by 3DMark is, for all intents and purposes, unchanged pre- and post-Spectre update.
Conclusions
The best thing I can say about Intel’s Spectre patch is that it doesn’t seem to harm the day-to-day performance of recent PCs much—at least going by Dell’s version for my Alienware 13 R3. As I’ve already lamented, though, per-core CPU performance gains of any kind are worth their weight in gold these days, and even the single-digit losses I saw across most of my light-usage benchmarks sting like a tarantula hawk when single-digit performance increases are all we’ve gotten from the blue team over the past few years.
Perversely, AMD might be able to take some good news out of this mess. Ryzen CPUs weren’t that far behind Intel parts in lightly-threaded workloads to begin with, and the company doesn’t believe its Ryzen CPUs are affected by the Meltdown exploit at all. AMD does expect to offer optional firmware updates for customers concerned about Spectre, but it doesn’t seem to be as concerned as Intel is about the need for a patch. The effects of Meltdown and Spectre patches on Intel systems and the potential performance benefits of second-generation Ryzen CPUs with higher clock speeds could help AMD close the gap even further soon.
For those whose PCs use Skylake or newer CPUs inside, it’s probably OK to go ahead and patch up with Spectre-cide if your motherboard maker or big OEM has a firmware update available. Outside of some potentially troubling behavior in I/O-bound workloads that we need to explore further, the performance trade-off from the patch just isn’t big enough to risk running unprotected. That’s a small bit of relief in what has otherwise been a rough year for PC enthusiasts. It remains to be seen just how much similar patches for Haswell and older PCs are going to hurt performance, but until we get stable microcode from Intel and firmware updates with that microcode from motherboard makers, the unease of running an unpatched system and the potential performance pitfalls of those patches for older processors are going to haunt us. All we can do is wait.
Share
