adimas - Fotolia

Get started Bring yourself up to speed with our introductory content.

How do buffer overflow attacks work?

Buffer overflow attacks are simple exploits that can give an attacker control over a program or process. Learn how these attacks work and how to make sure they don't happen to you.

Buffer overflows are a favorite exploit for hackers. The majority of vendors' available patches fix unchecked buffer problems, but what about applications developed in-house? They are just as susceptible as commercial applications to buffer overflow attacks. It is therefore critical to understand how they work and perform vulnerability testing on homegrown applications prior to deployment.

A buffer overflow is an exploit that takes advantage of a program that accepts input from a client or other software process. It occurs when a program or process attempts to write more data to a fixed-length block of memory, or buffer, than the buffer is allocated to hold. Exploiting a buffer overflow enables an attacker to control, crash or modify the program. There are different categories of buffer overflow attacks.

Types of buffer overflow attacks

Buffer overflow attacks are categorized based on the location of the buffer in the program or process memory. There are four types of attacks:

  1. Stack-based buffer overflow: In this type of attack, the program or process being exploited uses a memory object known as a stack to store user input. The stack is a continuous space in memory used to organize data associated with function calls, including function parameters, local variables and management information, such as frame and instruction pointers.
  2. Heap-based attacks: The heap is a memory structure used to manage dynamic memory. Programmers often use the heap to allocate memory whose size is not known at compile time, where the amount of memory required is too large to fit on the stack or where the memory is intended to be used across function calls. Heap-based attacks flood the memory space reserved for a program or process, but the difficulty involved with performing such an attack makes them rare.
  3. Integer overflow attacks: This is when an integer is used in an arithmetic operation, and the result of the calculation is a value in excess of the maximum size of the integer. Most programming languages define maximum sizes for integers. When those sizes are exceeded, the result may cause an error, or it may return an incorrect result that is "wrapped around" within the integer length limit.
  4. Unicode overflow attacks: These types of attacks use Unicode characters to create buffer overflows in programs that are expecting all input to be ASCII characters.

How buffer overflow attacks work

Stack-based buffer overflow is the most common of these types of attacks. Normally, the stack is empty until the targeted program requires user input, like a username or password. At that point, the program writes a return memory address to the stack, and then the user's input is placed on top of it. When the stack is processed, the user's input gets sent to the return address specified by the program.

However, a stack has a finite size. The programmer who develops the code must reserve a specific amount of space for the stack. If the user's input is longer than the amount of space reserved for it within the stack and the program does not verify that the input will fit, then the stack will overflow. This in itself isn't a huge problem, but it becomes a huge security hole when it's combined with malicious input.

Stack-based buffer overflow attack

For example, suppose a program is waiting for a user to enter her name. Rather than enter the name, the hacker would enter an executable command that exceeds the stack size. The command is usually something short. In a Linux environment, for instance, the command is typically EXEC("sh"), which tells the system to open a command prompt window, known as a root shell in Linux circles.

Yet, overflowing the buffer with an executable command doesn't mean that the command will be executed. The attacker must then specify a return address that points to the malicious command. The program partially crashes because the stack overflowed. It then tries to recover by going to the return address, but the return address has been changed to point to the command specified by the hacker. The hacker must know the address where the malicious command will reside. To get around needing the actual address, the malicious command is often padded on both sides by NOP instructions, a type of pointer. Padding on both sides is a technique used when the exact memory range is unknown. Therefore, if the address the hacker specifies falls anywhere within the padding, the malicious command will be executed.

How to perform a buffer overflow attack

The last part of the equation is getting around the executable program's permissions. Most OSes have some sort of mechanism to control the access level of the user who's currently logged on, and executable programs typically require a higher level of permissions. These programs therefore run either in kernel mode or with permissions inherited from a service account. When a stack overflow attack runs the command found at the new return address, the program thinks it is still running. This means that the command prompt window that has been opened is running with the same set of permissions as the application that was compromised. Generally speaking, this often means that the attacker will gain full control of the OS.

How to protect yourself from buffer overflow attacks

There are a few options to protect yourself from buffer overflow attacks, including:

  • staying up to date with vendor-issued patches and software updates;
  • enabling runtime protections in OSes;
  • using address space layout randomization;
  • marking areas of memory as either executable or nonexecutable with Data Execution Prevention;
  • manually testing for buffer overflows; and
  • using a programming language that doesn't allow buffer overflow attacks -- such as Java, Python or .NET -- when possible.

Developers should also be sure to protect against overflows by checking input values before processing. They can reference the Open Web Application Security Project Top 10, which has identified buffer overflows as a top 10 threat for many years.

This was last published in May 2019

Dig Deeper on Application attacks (buffer overflows, cross-site scripting)