A buffer overflow occurs when a program or process tries to store more data in its allocated data storage area, or buffer, than was originally intended. Since buffers are created to contain a finite amount of data, the extra information can overflow into adjacent buffers. When this occurs, it corrupts or overwrites the valid data held in them. Overflowing a buffer for a local variable in a function can overwrite the return address of that function. (The return address is the next instruction the process should execute once the function completes.) This can cause a segmentation fault that can crash the program. In certain conditions, the hacker will receive a shell prompt after the crash, which gives them control of the computer. And more sophisticated attacks look to overwrite the return address with a pointer to the code they wish to run, instead of trying to attack a computer just by trying to crash it.
Stack-based buffer overflow attacks are the most common, but let's look at the heap-based Buffer Overrun in JPEG Processing (GDI+) exploit to see how ingenious buffer overflow attacks can be.
The Microsoft dynamic link library file called GDIPlus.dll contains libraries for the Graphical Device Interface Plus (GDI+) application programming interface (API). This allows programmers to represent graphical objects and transmit them to output devices, such as monitors and printers. This DLL includes the capability to process JPEG image files, but it normalizes the declared length of the area designed for comments in a JPEG file prior to checking its value. This can cause a heap-based buffer overflow. Heap-based buffer overflow vulnerabilities occur if the unchecked copy of data is written to a buffer that is located on the heap. This means non-executable stack protection mechanisms can be bypassed, ultimately leaving the system vulnerable and allowing the hacker to point the next process to the code they wish to run. Ironically, they can store this in a comment area of the JPEG file. Now, if the hacker wishes to exploit this flaw, he/she only needs the victim to view the doctored image.
Buffer overflow exploits are common because programs written in relatively low-level programming languages, such as assembly language, C and C++, do not perform automatic bounds. This process checks on arrays or pointers and requires the programmer to manually manage the size of allocated memory. While a hacker can't guarantee that their exploit code will work every time, given the success of many viruses and worms, they can have a very high success rate. To see a Java applet demonstrating how buffer overflows work visit: http://nsfsecurity.pr.erau.edu/bom_docs/Demos/script.html. There is also a good beginner's tutorial called Writing Buffer Overflow Exploits at: http://www.securiteam.com/securityreviews/5OP0B006UQ.html.
This was first published in December 2005