Linus Torvalds talks frankly about Intel security bugs
Meltdown-Spectre
At The Linux Foundation's Open Source Summit North America in Vancouver, Linus Torvalds, Linux's creator, and Dirk Hohndel, VMware VP and chief open source officer, had a wide-ranging conversation about Linux security, open-source developer, and quantum computing.
Torvalds would really like his work to get back to being boring. It hasn't been lately because of Intel's CPU Meltdown and Spectre security bugs. The root cause behind these security holes was speculative execution.
In speculative execution, when a program does a calculation, which might go several ways, the processor assumes several results and works on them. If it's wrong, it goes back to the beginning and restarts with the correct data. Because CPUs are so fast these days, it's much quicker to do this than to have the hardware sit idle waiting for data.
Torvalds "loves speculative execution. CPUs must do this." But, Torvalds is annoyed that "people didn't think about the problems of taking shortcuts with speculative execution. We knew speculative work that wasn't used had to be thrown away." It wasn't. That problem is now baked in most modern processors. The long-term fix is a new generation of Intel CPUs.
This ticked Torvalds off. Linux, programmers and the other operating system developers, had to scramble to fix the hardware vendors' problems. Torvalds said, "It's not fair. When we screw up, it's fair, we have to fix it. But it feels less fair when we have to fix someone's else's problems."
Earlier at the conference, Greg Kroah-Hartman, the stable Linux kernel maintainer, went into more detail about why fixing the first Spectre class bugs was such a problem for Linux developers in particular.
The problems were know about in July of 2017, Kroah-Hartman explained, but "it wasn't until October 25 of last year that the kernel community heard rumors of the flaw. That's a long time, and we only heard rumors because another very large operating system vendor told Intel to get off their tails and tell us about it."
Then, Kroah-Hartman continued, "When we get a kernel security bug, it goes to the Linux kernel security team, we drag in the right people, we work with the distributions getting everyone on the same page and push out patches" Not this time. "Intel siloed SUSE, they siloed Red Hat, they siloed Canonical. They never told Oracle, and they wouldn't let us talk to each other."
Torvalds added that with the "security issues kept under wraps, we couldn't do our usual open methods. This made fixing the bugs much more painful than it should be." "It really wasn't working," continued Kroah-Hartman, so we "yelled at [Intel] and pleaded, and we finally got them to allow us to talk to each other the last week of December. All of our Christmas vacations were ruined. Intel messed up."
Worse still, Kroah-Hartman said, "Debian wasn't allowed to be part of the disclosure, so most of the world was caught with their pants down, and that's not good."
Since then, Torvalds said, "Intel has gotten much better." But, even now, "I don't know what the hardware bug schedule is."
Still, when the next Spectre variant showed up, Foreshadow, the Linux kernel developers were notified ahead of time. Because of this, the Linux community could use their battle-tested open methods to patch this bug promptly.
Also, added Torvalds "The good news is the bugs have become more esoteric. They impact fewer and fewer cases. Intel and other hardware vendors are really getting down to the dregs of the hardware security bugs."
For the rest of 2018, Torvalds said, "Every three months hardware bugs would show up. There have been eight serious bugs this year. There were only two or three in all the years before."
Even now though the Spectre problems still haunt the Linux development process. The "Linux 4.19 merge window was not good. Usually it's not a big deal. It's just two weeks of me sitting in front of my computer. But the new security issue popped up in early in the merge window, which just added the usual frustration due to having patches that weren't public. This made it particularly stressful."
For all the pain this has caused, there's also been one good, unexpected result. Kroah-Hartman explained that Linux and Windows programmers are working together on CPU security bugs.
"We now have this wonderful back channel. We're talking to each other and we're fixing bugs for each other," he said. Who would have thought even five years ago we'd see this?
Related Stories: