Problem solve Get help with specific problems with your technologies, process and projects.

Ensuring the integrity of a JVM

I am working on a DARPA project to secure a widely distributed, mobile-agent computing environment called UltraLog (www.ultralog.net). The agents are developed using the Java Expert System Shell (JESS). I want to protect the mobile agents from malicious hosts and malicious Java Virtual Machines (JVM). I have plenty of ideas about malicious hosts, but I'm struggling with the JVM problem. Aside from signing the java runtime executable {rt.jar}, do you have any other ideas on how to ensure the integrity of a JVM? The main issue is preventing a malicious insider from replacing the Java runtime with a malicious one. Digital certificates are one approach, but we have already found numerous instances in the UltraLog environment where a determined individual can work around the digital certificate requirement and gain unauthorized access. Thanks!

In any system there are parts of it that you simply have to have faith that it works. You have to just trust that your CPU works. You have to trust your operating system. You have to trust your virus scanner.

This isn't merely some technical situation, either. It is a mathematical truth, part of "computability theory" that you cannot compensate for a sufficiently malicious machine.

Most people have a relatively easy problem to solve -- they have a base of trusted software, and they protect against malicious, subordinate software, for example, hostile applets running within a JVM. You, however, have the opposite problem. You have Java code you trust, and you don't trust the CPU they run on (that's what a JVM is -- it's a CPU implemented in software).

Alas, because of computability theory, you're out of luck. There's nothing you can do. Let's suppose that you actually are running your Java code on a hostile machine that will do whatever it can to subvert you.

Suppose you do something like check a digital signature against rt.jar. Who computed the signature? The JVM that you don't trust did. Why do you trust it? Why couldn't it just tell you, "Yup, that's a valid signature, I'd stake my reputation on it." For that matter, why not verify the signature of a legitimate rrt.jar, and then load code from some other one? After all, it's the JVM, it can do whatever it wants, and what it wants to do is deceive you. Certificates don't do anything -- a certificate is nothing more than a digital signature. If you hand a certificate to an evil verifier, the verifier will just tell you the cert is good.

Suppose you write a class that verifies the JVM. Why wouldn't the JVM run correctly during its execution? Or perhaps even simpler, it doesn't run your verification code at all, it just returns you the correct answer. How can you tell?

This problem you're trying to guard against is one where you are writing software, and you want it to run only on the real computer you intend it to run on, not some emulation of it. This is impossible. You can't do it. However, this isn't bad news, it's actually good news. Since it's impossible to guard against an evil JVM, you can just go on and not worry about it.

This was last published in February 2002

Dig Deeper on Web application and API security best practices

Have a question for an expert?

Please add a title for your question

Get answers from a TechTarget expert on whatever's puzzling you.

You will be able to add details on the next page.

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.