Image: Google/Natascha EiblWe have a pretty good idea that fixes for the Meltdown and Spectre CPU exploits plain hurt on newer hardware, but Microsoft and Intel said it would be even worse on older hardware. No one has said just how bad yet, though, so we decided to find out. Spoiler alert: The results are ugly.
Most motherboard vendors aren’t supporting anything older than 6th-generation (Skylake), so we turned to the oldest hardware for which we could get a microcode and software update: a 5th-gen Broadwell laptop.
How we tested
If you don’t remember the Dell XPS 13 9343, this is the laptop that put the XPS 13 in the lead with its “InfinityEdge” bezel.
Inside it has a 5th-gen Intel Core i5-5200U, 4GB of LPDDR3, and a 128GB SATA Samsung PM851 SSD. We updated it to the Windows 10 Home Fall Creators Update and then installed the latest BIOS and drivers for the laptop.
Although Intel has asked vendors to halt patches for Broadwell and Haswell due to spontaneous reboots, our XPS 13 didn’t exhibit any instability or crashes over several days of testing. Performance changes, however, are another thing.
IntelThe oldest thing we cold get our hands on that also received the patches was a Broadwell laptop.
For our first Meltdown/Spectre exploration with the Surface Book, we simply tested the storage subsystem before and after Microsoft pushed out the firmware, a strategy that gave us little control. For our Broadwell test, we relied on Steve Gibson’s InSpectre utility, which confirms whether you’re completely patched against the exploits and also lets you disable the protection.
Just to make sure InSpectre was working, I also ran Alex Ionescu’s SpecuCheck, which reports more detail on the patch status but doesn’t let you manually switch it off.
Of course, the ultimate verification was seeing the results in the benchmarks. Flipping the protection off or on immediately flipped the results.
The kinda good news
We already know from testing at TechSpot and other hardware sites that the impact on a lot of CPU-intensive tasks can be fairly minimal. We saw as much in most of our tests.
First up: Cinebench R15.038. This popular CPU-based 3D renderer is built on a real engine. For the most part, we see barely a 2 percent difference, which is within the margin of error.
IDGCinebench R15 testing on a 5th-gen Broadwell laptop shows minimal impact from Spectre and Meltdown patches.
Moving on, we used VeraCrypt’s default AES encryption test, which taxes the CPU. As with Cinebench, we saw basically no effect.
IDG
VeraCrypt’s encryption benchmark also shows no impact.
Next we ran WinRAR 5.40’s built-in test and yet again found a very minimal impact from Spectre and Meltdown fixes.
IDGWinRaR’s built-in test also shows little impact to the CPU.
We ran 7-Zip’s built-in compression test, where the fix for Spectre and Meltdown moved the needle about 4 percent. That’s probably the most we saw among several performance tests.
IDG7-Zip says meh.
Our last yawner is using Handbrake 0.9.9 to encode a 4K-resolution file using the Standard Profile. The result shows a small advantage pre-patch, but not enough to care.
IDGYawn. Again.
Here’s the bad news
We could show you another four charts where the bars all look nearly identical or shift by a smidgen, but we won’t. Instead, we’ll hit you with the bad news right now.
Most of the tests we’ve run are pure CPU tests. Some of them may rely on memory performance, but they’re designed stay within the cores themselves for the most part. That’s what a good CPU benchmark does.
BAPCo’s SYSMark 2014 SE is a different kind of beast, though. It’s built on a dozen or so real applications such as Photoshop, Excel, PowerPoint, and Word. It measures a computer’s overall performance in multi-tasking as well as how it responds to commands. Although it is tinged by accusations of politics, there really is no other test like it for accurately measuring real-world usage.
The overall test score on the 5th-gen Core i5 shows just where the Meltdown and Spectre fix hurts. The overall score shows a 12-percent drop in performance. This is a serious shortfall, but probably not a deal-breaker.
IDGSYSMark 2014 SE is perhaps the most comprehensive benchmark for the PC today.
That overall score though, rolls up several sections of the test into one score. If you dive into the results, you can see the variations from test to test. The data and financial analysis score, for example, reflects performance for generating a sales forecast, creating charts, and other Excel-related tasks. It shows about a 3-percent impact from enabling the Spectre and Meltdown fix.
IDGSYSMark 2014 SE’s data analysis shows a minor three-percent performance degradation.
Where performance starts to dive is in media creation. In SYSMark 2014 SE Media Creation, we saw a performance drop in excess of 10 percent. BAPCo describes this portion of the test as follows: “Create a panoramic image using an image editing application, combine a set of photos into one high dynamic range (HDR) image, and adjust and prepare both images for print. Transcode the video to a format suitable for web publishing using a video editing application. Add visual features to an existing architectural model, and render ‘sketch’ style views using an architectural modelingapplication.”
IDGSYSMark 2014 SE’s Media Creation test shows a fairly small hit as well.
It gets even worse in Office Productivity, which is described as: “Read and manipulate notes from a notebook. Browse multiple websites which include a blog, online shopping site, wiki site and social networking site. Archive a diverse set of files into a single encrypted file. Convert a PDF document into an editable word processing document. Perform and mail merge. View a complex presentation that include multimedia and export it to PDF. Combine multiple scanned pages from a complex document into an encrypted PDF document using optical character recognition (OCR). Read, create and search for emails. Create and execute a rule on email inbox. Use a spreadsheet program to do data analysis.” The hit is almost 12 percent.
It gets worse
What’s crazy is that’s not even the really bad news. Besides testing general performance, SYSMark 2014 SE includes a module that measures system responsiveness—how long it takes for the PC to respond when doing certain tasks. In this case BAPCo says: “Launch a variety of creativity and productivity applications. Browse a large number of websites. Import photos into a catalog. Export a large set of photos from a catalog. Decompress the encrypted and unencrypted archive files. Copy a large number of files in the background. Save presentation documents. Install and uninstall applications in the background. Switch between various web browser tabs. Save a variety of documents to the file system. Switch between various applications.”
Intel’s own tests on 8th-gen and 7th-gen laptops put the performance drop at 14 percent, while 6th-gen Skylake takes a hard 21-percent fall. Our tests put the 5th-gen Broadwell at 23 percent in the hole.
IDGUgly, plain ugly, is the only way to describe the 5th-gen Core i5’s performance with the Spectre and Meltdown fix.
SYSMark 2014 SE uses real applications to measure performance and responsiveness. Even so, it’s probably hard for the average person to relate to the performance hit from the Spectre and Meltdown fix.
To see when the pain points show up, we did something we’ve done a hundred times: decompress a single file. For every PC we test, we copy the CinebenchR15.038 file and unzip it using the Windows 10 File Explorer interface. It’s a 82MB Zip file with 2,516 files in 129 folders, and it decompresses to 199MB.
I’m not a fan of stopwatch-based tests, but for this test I simply decompressed the file to several unique folders multiple times and measured how long it took. I then flipped the Spectre and Meltdown fixes either on or off and repeated the task.
As you can see, the results are ugly. We go from taking about 24 seconds to decompress a file to nearly 31 seconds, roughly a 28 percent difference with the Spectre and Meltdown patch switched off (faster) and on (slower).
IDGHere’s how much longer you’ll wait to unzip compressed files with small files in them.
Sure, some will say, big whoop, what’s six more seconds? Think beyond this 30-second task to a 10-minute task. You’re now looking at several more minutes due to the Spectre and Meltdown fix. Most of the performance hits seem to come from storage-related tasks, which our Surface Book tests and Techspot’s tests already showed.
Here, for example, is the performance hit running Crystal Disk Mark 6 using a queue depth of 32 and a single thread on the 5th-gen Core i5. Ouch.
IDGMost of the performance hits from the Meltdown and Spectre patch show up in disk I/O on small file sizes.
SiSoft Sandra’s storage test of the file system I/O also shows significant hits to small file access with the patch in place.
IDGSiSoft Sandra 2017 shows a similar hit in 4K read performance.
What this all means
The results here reinforce what we learned from tests on newer CPUs. Generally, performance in common tasks will be hard for the average person to notice most of the time. So yeah, breathe a sigh of relief.
And then sometimes, it’ll just hit you in the face, with wait times taking 25 percent more on I/O-intensive tasks, such as decompressing a file. Sometimes it’ll be a no-big-deal few seconds, and sometimes it’ll be a teeth-gnashing wait that didn’t happen the last time you did the same thing. It just all depends on what you’re doing.
The bright side, for those who actually get security updates, is that you even got them. It’s better to take an occasional performance slowdown and know you’re secure. Think of all the consumers who are likely to simply be abandoned with their older hardware, and the only fix is to buy a new computer.
That’s basically going to be story of the Meltdown and Spectre fix on computers for the foreseeable future, as this slow-motion train wreck continues.