Problem solve Get help with specific problems with your technologies, process and projects.

Implementing runtime process integrity checking for Unix & Linux

I was wondering if I can get some comments and/or suggestions on how to implement Runtime process integrity checking for at least Unix/Linux world. I have initiated this effort by writing a LKM for Linux that intercepts sys_execve() system call. Then I pass the full path of the TO BE executed binary to some user-level Java code that actually computes the hash and checks it against an already established database of binary file hashes... The kind of stuff Tripwire does but more on-the-fly and need-based nature. So this way, I can check the integrity of a binary whenever it is ran. The idea of using Java to implement the actual logic of integrity checking is to make it more portable while writing only small pieces of code to intercept the system calls on different systems.

So, I guess I am asking your suggestions/comments on the idea of runtime process integrity checking and possibly the best (or better) way to do it.

This is a subject that I find very interesting.

The basic idea you have is a very good one. Integrity checking an executable before you execute it is a great idea. There are a lot of people doing similar things, too, including rule sets to govern what gets executed, using digital signatures instead of mere hashes and so on. Take a look at Immunix and in particular their "CryptoMark" system (which is still in progress), and RSBAC for their rule-set system, which includes a way to plug in your own rules, which could be an integrity checker. There are also a number of other efforts to solve the same basic problem.

The first comment I'd make is to note that if you have a database of hashes, this becomes your vulnerable spot. That database can be modified, and modifications can either allow rogue code to execute anyway or can cause a denial of service -- the attacker can rewrite it with zeros and simply watch you scurry.

This is the reason why systems like CryptoMark are based on digital signatures. Now, of course, the problem with a digital signature is that there is some signing key that signs files, and it can be replaced or damaged, and you have to protect the secret portion of it, too. So this is not a perfect solution, either.

My next comment is that loadable modules have a certain operational elegance, but for security, a thing that can be loaded can be unloaded or replaced. It may be better to compile that module directly into the kernel, forcing an attacker to replace your whole kernel, not just a module. But there are always tradeoffs. I've done these sorts of things both ways.

My next comment is why user-space code, and why Java? I'm a huge fan of Java, but this doesn't strike me as a place where you'd want the overhead and complexity of passing to user space from the kernel, as well as the overhead of Java. Computing a hash of a file is a pretty easy thing to do, and most of it is math. It would be a lot faster in compiled code. I have some other questions about what the interface between execve() and your checker is. How does the path get to your checker and the answer get back? Those are weak points in your system and have to be protected. And to repeat myself, are you really cranking up an entire JVM just to keep from writing up the real C from this pseudocode:

file = fopen(path)
while (!eof(file))
shs_update(fread(file, chunksize));
return memcmp(lookup_in_db(path), shs_final());

I've elided away a lot of things like variables, and you have to be extra careful in the kernel. But that's where it belongs. Portability isn't really an issue because the code you're going to use is just ANSI C. There may be kernel issues, but let's face it, you have kernel issues in every kernel. Properly designed, the actual checker would be portable in C.

Now -- if your real goal in this is that you want to set up communications between the kernel and a JVM, that's different. That's an interesting project, and this would be a fun test project for it. But that's a different reason.

Let me know if I can help any more.

This was last published in October 2001

Dig Deeper on Alternative operating system security