Chapter 7 Detection: Prelude

Prelude is defined by his conceptor (Yoann Vandoorselaere) as a general-purpose hybrid intrusion detection system. It is divided in two parts, a sensor (Prelude NIDs) and a report server. Information is available at:

ttp://www.prelude-ids.org/

Shortly, Prelude allows network intrusion detection, and the report server concentrates different mes­sages alert, which are generated by Prelude agents. Any system may report alert messages to the Prelude server if a convenient plugin exists. We are very interested in it because of the Libsafe ability to report buffer overflow exploit attempts to Prelude, as well as its aim to manage polymorphic shellcode attacks.

7.1   Prelude and Libsafe

Libsafe is able to send alert messages to Prelude. The feature has just been approved by Libsafe main-tainers, which means that this ability is now available in the releases (it used to be a patch for previous releases). We simply must be careful that libprelude.so is in the search path for dynamically loaded libraries (it is OK if it is contained in LD.LIBRARY_PATH, for instance).

Using both Prelude and Libsafe allows an administrator to catch alert messages concerning buffer overflows on any Libsafe-enabled machine and to redirect them to his report server.

7.2   Shellcode detection with Prelude

There exists some attacks based on buffer overflow that are executed remotely, e.g the shellcode is encapsulated in packets. These exploits can be detected by Prelude used in combination with the shellcode plugin. The installation of prelude and the shellcode plugins are detailed in 8.4.

7.2.1 Principle

In order to be aligned with the return address (see page 12) the buffer may need to be padded with NOP. This is what we will search inside the packets: an unusual amount of NOP.

In practice, there are several kind of NOP instructions (x86 opcodes): \x90, which is the real nop, or instructions that do nothing such as \x50, which is push eax, ...

The plugin will go through the packet and try to match the pattern read with the different NOP instructions available. If the amount of NOP instruction matched is too high then an error is raised.

The different NOP instructions are stored in a hash table (for the efficiency of the pattern matching); there is one hash table per architecture. Actually the plugin knows opcode for 3 different architectures (intel 86, hp alpha, and sparc).

7.2.2 Implementation

This is implemented as a plugin for prelude-nids, and the idea is to count the continuous bytes which may represent a danger. When a threshold is reached, Prelude considers it is an attack, and raises an alert.

The final step of the NOP search is coded in check_tlb, called by detect_shellcode for each known architecture; to every architecture corresponds a structure containing among others a counter, and if the choosen limit is reached, an alert must be sent:

/*

* we want contiguous chain of NOP. reset counter.

*/

if ( ! found )

arch->nop_counter = 0;

if ( arch->nop_counter == arch->nop_max ) { arch->continue_checking = -1; raise_alert(packet, arch);

}

This allows Prelude to detect shellcode encapsulated in IP packets, as we will see later in an example.

7.3   A new danger: plymorphic shellcodes

7.3.1 Where the danger lies...

As we have seen, a well-known kind of shellcode consists of a large amount of NOP bytes followed by the real code, and this means that this well-defined pattern can be discovered without great difficulties, at least in theory. In practice, some NIDS such as Prelude do it.

To make this detection task more difficult, a new generation of shellcode has been created: polymorphic shellcodes. A first step has been to replace the NOP sequence by a more sophisticated set of instructions which finally has no effect, for instance empiling and depling several times the same register without changing its content. . .

Another trick is the shellcode encryption: the instructions we could clearly saw in the previous case are now encrypted, and the decryption mechanism is a third part of the shellcode. This means that the final code to transmit will be quite larger than in the previous case, which is a serious drawback, but much harder to discover. This depends on the encryption system: a xor based encryption is the most classical way to hide malevolent code without making it grow too much.

7.3.2 How to discover it ?

Some methods have been discussed but most of them have the same strong drawback: they require too much CPU resource, and will last too long. Nevertheless, detecting such shellcodes is not easy and implies high resources.

A first possibility consists in trying to emulate the code decryption and execution, but this would need too much CPU time.

A second method is based on signatures; a set of known dangerous encrypted instructions could be used to compare the packet bytes to, but this would be excessively hard to trust as it would probably raise lots of false alerts.

Another method may consist in decoding the shellcode using several decryption methods, and each time compare the result to well-known shellcode signatures. This would require brute forcing and cannot reasonably be considered as a realistic solution.

So far, we see that the solutions are very theoretical. Nevertheless it seems that another solution may be quite efficient: it s quite the same principle as NOP detection, as it consists in discovering series of instructions without effect. Instead of only looking for continuous NOP instructions, it is possible to also count each unvalid instruction, which may be defined by the NIDS itself or configured by an administrator. The aim is to define which sets of instructions have no effect and, if they are met too often, may be considered as part of a shellcode padding.

Prelude for example detects polymorphic shellcodes by using this last technique.