They need to get the compression algorithm. That's the simplest way to do it. Probably they could crack the algorithm, but it's probably simpler to just steal a copy of it.
Are you considering doing this, or are you just asking?
The reason I ask is that there's an important difference between coding and encryption. Coding is a different way of writing things. For example, this message is coded in ASCII. A coding is just a way of saying that this group of bits means that group of symbols. Anyone who knows the coding algorithm can translate the bits into symbols.
Encryption is an algorithm that is a function. That function takes a parameter, the key, and when applied to your set of bits, translates it into another set of bits that someone can't undo without the key.
It is certainly possible to make codings where the coding itself can be thought of as the key. That's what you're suggesting. If your attacker, however, gets one of your coding machines and takes it apart, then you've lost the secrecy of the entire system. With encryption, the security revolves around the keys. Your attacker can have a machine just like yours, and they still can't steal your data unless they steal your keys.
If you are building a real-world system, the best solution is to compress something, then encrypt the compressed data. That way, you can pick the most efficient compression algorithm for your data (for example, LZ compression works well on text, and JPEG works well on pictures), and then encrypt that. With today's silicon, symmetric encryption (which is what you'll do on bulk data going through an OC3) is basically for free. It is more likely that a fancy compression algorithm will be much more expensive than the encryption. It will also be much more secure.
Many people have tried to devise systems that combine some sort of compression with some sort of confidentiality features. I haven't seen one that works as well as the simple mechanism of compress and encrypt.
For more information on this topic, visit these other SearchSecurity.com resources:
Best Web Links: Encryption
Tech Tip: Tales from the crypto
This was first published in April 2002