This is a subject that I find very interesting. The basic idea you have is a very good one. Integrity checking an executable before you execute it is a great idea. There are a lot of people doing similar things, too, including rule sets to govern what gets executed, using digital signatures instead of mere hashes and so on. Take a look at Immunix and in particular their "CryptoMark" system (which is still in progress), and RSBAC for their rule-set system, which includes a way to plug in your own rules, which could be an integrity checker. There are also a number of other efforts to solve the same basic problem. The first comment I'd make is to note that if you have a database of hashes, this becomes your vulnerable spot. That database can be modified, and modifications can either allow rogue code to execute anyway or can cause a denial of service -- the attacker can rewrite it with zeros and simply watch you scurry. This is the reason why systems like CryptoMark are based on digital signatures. Now, of course, the problem with a digital signature is that there is some signing key that signs files, and it can be replaced or damaged, and you have to protect the secret portion of it, too. So this is not a perfect solution, either. My next comment is that loadable modules have a certain operational elegance, but for security, a thing that can be loaded can be unloaded or replaced. It may be better to compile that module directly into the kernel, forcing an attacker to replace your whole kernel, not just a module. But there are always tradeoffs. I've done these sorts of things both ways. My next comment is why user-space code, and why Java? I'm a huge fan of Java, but this doesn't strike me as a place where you'd want the overhead and complexity of passing to user space from the kernel, as well as the overhead of Java. Computing a hash of a file is a pretty easy thing to do, and most of it is math. It would be a lot faster in compiled code. I have some other questions about what the interface between execve() and your checker is. How does the path get to your checker and the answer get back? Those are weak points in your system and have to be protected. And to repeat myself, are you really cranking up an entire JVM just to keep from writing up the real C from this pseudocode: file = fopen(path)
return memcmp(lookup_in_db(path), shs_final());
I've elided away a lot of things like variables, and you have to be extra careful in the kernel. But that's where it belongs. Portability isn't really an issue because the code you're going to use is just ANSI C. There may be kernel issues, but let's face it, you have kernel issues in every kernel. Properly designed, the actual checker would be portable in C. Now -- if your real goal in this is that you want to set up communications between the kernel and a JVM, that's different. That's an interesting project, and this would be a fun test project for it. But that's a different reason. Let me know if I can help any more.