Why fixing the Java flaw will take so long

OpenJDK community blasts Oracle's 'zero day' Java security patch

By now you've heard about the latest, very serious problem with Oracle's Java runtime. You may also have heard that it could take a very long time to fix. Here's why: The flaw uncovered by security researchers last week devolves not to one issue, but to a series of issues, one knocking into the other like dominoes. Oracle has fixed one of the dominos with a patch, but there are likely to be other ways to tip over the entire row.

Emergency response

The vulnerability patched by Oracle resides in a version of Java 7 designed to extend Web browsers. The defect made it possible for a malicious Java applet on a Web page to execute arbitrary code on the underlying computer.

While this sort of defect would usually be kept secret until a fix was available, it was disclosed last week because malicious crackers had already found the defect and were exploiting it as part of a dirty-tricks toolkit used by scammers and other thieves, giving Oracle zero days to fix the code. As more researchers evaluated this "zero-day exploit," it became clear it was exceptionally serious.

With terrific speed, Oracle's engineers created a fix for the problem over the weekend and released it Monday. Yet security researchers weren't impressed. Why was that? I asked Oracle to brief me, but I was refused and simply referred to a blog posting on the subject, which offered little explanation.

Instead I turned to the open source community for help. Java 7 is actually based on an open source project called OpenJDK, and Oracle had also released patches for that. I was able to quickly find explanations of both the defect and the fix.

Problem analysis

Running arbitrary code via Java in a browser is supposed to be impossible, because of a feature known as the "Java sandbox." Capabilities of Java that affect the machine it's running on are restricted to apps that have gained necessary permissions from the machine owner. To reach outside the sandbox and touch the machine -- including to run arbitrary code -- involves getting permission from a part of the Java runtime called the Security Manager.

But this exploit was able to bypass the Security Manager and gain permission to access the machine. A combination of subsystems -- including JMXBeanServer, the Rhino JavaScript classloader, and the new invokedynamic support that was added in Java 7 -- were used in a way very similar indeed to a previously reported defect that had not been completely fixed back in August.

Oracle has prevented the exploit from succeeding in an emergency release by fixing just the defective new implementation of invokedynamic. A bug allowed "reflection" -- the runtime discovery of the characteristics of classes -- to disclose inner details of Java classes that should have been hidden. That's been fixed, but there don't seem to have been changes to the other subsystems used in the exploit.

That means there's every chance that crackers could find a new way to exploit the same defects. Those other subsystems interact with each other and with the Security Manager in subtle, complex ways, and it's perfectly reasonable for the software engineers creating the emergency fix to have decided not to mess with them for fear of creating worse issues than they were solving.

Solution critique

It's that cascade of subtly interacting subsystems that worries security researchers so much. The problems seems to be less a defect in a single subsystem and more a consequence of the interaction of apparently correct subsystems.

While Oracle's quick fix appears to have broken the exploit chain in this instance, researchers fear that building another chain could be possible -- and may already have happened within the shadows of the black-hat cracker community. They fear that no single developer has the overall knowledge of all of the subsystems involved to safely create a rapid fix, so it will take a process of experimentation stretching over many months to work out what must be re-engineered to make new exploit chains impossible. Oracle seems to agree: It has set the default security level in Java to "high," just in case.

The security researchers were at pains to point out that the same cascade of complex interactions could affect any runtime model using a rich programming interface protected by a sandbox -- a model common to Android, Flash, .Net, and other runtimes.

How could the situation be improved? I was told that Oracle's secrecy does not help. Just as I was unable to get anyone to explain the issue to me, so Oracle refuses to collaborate over security issues with community members -- as the MySQL community also discovered. One community member told me: "I think people are dissuaded from posting or working on such patches since it is so clear Oracle has a policy to not talk about bugs/patches when they are marked as 'security issues.'"

One benefit of open source is Eric Raymond's observation that "many eyes make all bugs shallow." Perhaps if that all-seeing community of development experts was better leveraged, this issue could be addressed sooner than the "six months to two years" some are predicting.

This story, "Why fixing the Java flaw will take so long," was originally published at InfoWorld.com. Get the first word on what the important tech news really means with the InfoWorld Tech Watch blog. For the latest developments in business technology news, follow InfoWorld.com on Twitter.

Copyright © 2013 IDG Communications, Inc.