Protecting Against Breakpoints, Tracers, and Debuggers

Now let's take a more in-depth look at ways to protect against crackers' attempts at using breakpoints, tracers, and debuggers against your programs.

Detecting Tracers Using the Trap Flag

This is one of my favorite tricks. It allows you to detect any tracer including SoftICE, TRW, FrogsICE, and ProcDump.

My friend EliCZ discovered this trick based on elegant idea. If the Trap flag is set in the Eflags, EXCEPTION SINGLE STEP will be launched. In short, this means that the SEH service will be called before the instruction is performed. We can use the fact that if a program is traced, the SEH service will not be called, and we can set a mark there to show whether it was called or not. If not, a tracer is active.

.386

.MODEL FLAT,STDCALL locals jumps UNICODE=0 include w32.inc

;While you do not need any API calls for detection, you will use them for printing out the result ;and for ending the program



.DATA

message1 db "Tracer detection using the Trap flag",0 message3 db "Tracer found",0 message2 db "Tracer not found",0

DelayESP dd 0 ;the ESP register saves here

Previous dd 0 ;the ESP register saves the address of the

;previous SEH service here
mark db 0 ;sets a mark showing whether it went into the

;xhandler

.CODE Start:

call real_start ;jumps to the real start of the program



;A new SEH service in case of an error (xhandler)

inc mark ;increases the eax by one to show that it isn't

;being traced

sub eax,eax ;nulls the eax because of the future setting of

;the original SEH service
ret ;returns to the program

real_start:

xor eax,eax

push dword ptr fs:[eax]

mov fs:[eax], esp





pushfd

or byte ptr [esp+1], 1



popfd



pop dword ptr fs: [eax]

pop ebx

;nulls the eax register

;saves the original SEH service

;sets your new SEH service. Because you jumped ;by means of CALL, the following return address ;is saved into the stack. This will be your new ;SEH service. ;saves the flag registers

;sets the Trap flag inEflags, which means that

;the program will launch EXCEPTION SINGLE STEP,

;and your new SEH service will be called ;restores the flag registers with the previously ;set Trap flag. The whole process runs nop ;and xhandler is called

;sets the previous SEH service and clears the ;stack

;decreases the mark by 1. If it is -1



js jump

;(OFFFFFFFFh) after the decrease, thena tracer ;is active in memory because the xhandler hasn't ;been called ;and the program jumps

continue:

call MessageBoxA,O, offset message2,\ offset message1,O call ExitProcess, -1

jump:

call MessageBoxA,O, offset message3,\ offset message1,O call ExitProcess, -1

ends

end Start



This is a great detection method that is almost impossible for crackers to discover (when it is well hidden), especially because tracers seem to have a problem emulating it correctly.



To make it more difficult for the cracker to discover this trick, be sure that the protected application doesn't print out messages or warnings. The best response is to simply end the application incorrectly and punish the cracker that way, though you might use your imagination to come up with other responses.



This trick works in all Windows versions.





Detecting Breakpoints by Searching for Int 3h



To make debugging more difficult, it is very important to test for the presence of breakpoints. If a breakpoint is set in a debugger, there will be an Int 3h instruction at a particular place in memory.



The following example program first searches its code in memory for the value 0CCh, which is the prefix of the Int 3h instruction. There is one problem, though, which the program doesn't take into account: It is likely, in most applications, that the value 0CCh will exist somewhere in the code. In such a case, the testing routine should save these locations somewhere. Then, if it finds the value 0CCh, it will first test to see if it is where it was found before.



Once the program finishes this test, it tests to see if a breakpoint was set on an API call used by the program. It searches addresses for imported functions one by one in the IAT table, found after the program code. These are pointers to the beginnings of the API functions, and the 0CCh value would be found there if a breakpoint was set to this API function.



Another way to find the addresses of the API function routines would be to find them in the file's PE header, where this information is saved.



.386

.MODEL FLAT,STDCALL

locals

jumps

UNICODE=O

include w32.inc

Extrn SetUnhandledExceptionFilter : PROC

.DATA

message1 message3 message2 delayESP previous

db "Breakpoint detection by searching for Int 3h",O

db "Breakpoint found",O

db "Breakpoint not found",O

dd O ;the ESP register saves here

dd O ;the ESP register will save the address of the

;previous SEH service here
current_import dd O ;the address of the imported function currently

;being tested is saved here

.code

Start:



;Sets SEH in case of an error

mov [delayESP], esp push offset error

call SetUnhandledExceptionFilter mov [previous], eax

lea

mov

edi, Start

ecx, End-Start+1

;the starting address of the program will be put ;into the edi

;the length of the program, that is, the number ;of tested bytes, will be put into the ecx

mov

add

eax,OBCh eax, 1Oh

;puts OCCh into the eax (this is the prefix of
;the INT 3h instruction). Note! Do not use
;mov eax,OCCh or you will find this value in
;the program and think that it is a breakpoint.
repnz scasb ;searches for a breakpoint in the program

test ecx,ecx ;tests to see if ecx=O; if not, it has found a

;breakpoint

jne found ;and the program jumps

lea eax, Imports+2 ;puts the IAT (first function) into the eax

;register

mov dword ptr [current import], eax

start_searching:

mov eax,

dword ptr [current_import] cmp dword ptr [eax], O

je notfound mov ecx, [eax]



mov ecx, [ecx]

mov eax, OBCh add eax, 1Oh





cmp byte ptr [ecx], al





je found

add [current_import], 6 jmp start_searching found:

mov eax, 1 jmp farther

notfound:



;puts the imported function to be ;tested into the eax

;tests to see if it is O; if so, thenthis is ;the end of the Import table (the table of the ;imported functions)

;the program jumps if it hasn't found any ;breakpoints

;this reads the address where the imported ;function is located; that is the address of the ;beginning of its routine

;reads the address of the beginning of the ;routine of the imported function

;puts OCCh into the eax (the prefix of the INT ;3h instruction). Note! Do not use mov eax, ;OCCh or you will find this value in the program ;and think that it is a breakpoint. ;tests for the breakpoint's presence in the ;tested imported function

;the program will search for OCCh (actually Int ;3h) at the beginning of its routine ;and if a breakpoint is found it will end ;sets to another imported function in the IAT

;jumps to search for a breakpoint in the next ;imported function

;sets eax to 1 to show that the program has ;found a breakpoint

xor eax, eax ;sets eax to O to show that it hasn't found a

;breakpoint

farther:

push eax ;saves the return value

;Sets the previous SEH service

push dword ptr [previous]

call SetUnhandledExceptionFilter

pop eax ;restores the return value

test eax, eax jnz jump

continue:

call MessageBoxA,0, offset message2,\ offset message1,0 call ExitProcess, -1

jump:

call MessageBoxA,0, offset message3,\ offset message1,0 call ExitProcess, -1

error: ;sets a new SEH service in case of an error

mov esp, [delayESP] push offset continue ret

End:

Imports: ;this label must be at the very end of the

;program, since the IAT begins there

ends

end Start



This trick is limited by the fact that it cannot find debug breakpoints that do not change the program code. Therefore, it is good to combine this trick with a way to detect debug breakpoints.



This trick works in all Windows versions.





Detecting Breakpoints by CRC



This example shows a different method of breakpoint detection that I think is even better than the previous one. It's based on the CRC calculation for the particular program and the checks that run during the course of the program. A program's current CRC is found at the CRC label. The CRC label changes with alterations in the program, and is therefore adjusted after each alteration. This method is even effective against changes performed in memory.



This trick is simple: First, you run the debugger, and once it returns from the CRC32 routine, the EAX register will contain the current CRC of the program code in the memory. Next, you put this value into the CRC label, and when a protected application is launched, the program calculates the CRC of the code in the memory. If the CRC isn't the same as the value at the CRC label, the program has been changed, which tells you that either there were changes performed in the program code or there is a breakpoint set in the code.



.386

.MODEL FLAT,STDCALL

locals jumps

UNICODE=0

include w32.inc

Extrn SetUnhandledExceptionFilter : PROC

.data

message1 db "Breakpoint detection by CRC",0

message3 db "Breakpoint or a change in the program found",0 message2 db "Breakpoint not found",0

delayESP dd 0 ;the ESP register saves here previous dd 0

;the ESP register will save the address of the ;previous SEH service here

;Following is a table of the values necessary for the program's CRC calculation.

CRCTable dd 00000000h, 77073 096h, 0EE0E612Ch, 9 90 951BAh

dd 076DC419h, 706AF4 8Fh, 0E963A53 5h, 9E64 95A3h

dd 0EDB8832h, 79DCB8A4h, 0E0D5E91Eh, 97D2D988h



Start:



;Sets SEH in case of an error

mov [delayESP],esp push offset error

call SetUnhandledExceptionFilter mov [previous], eax

lea esi, CRCTable ;address of the CRC table needed for the

;calculation

lea edi, Start ;beginning of data for which the CRC will be

;calculated (beginning of your program)

mov ecx, End-Start+1 ;length of the data in bytes (length of your

;program)

call CRC32 ;jump to calculate the CRC

push eax ;saves the calculated CRC

;Sets the previous SEH service

push dword ptr [previous] call SetUnhandledExceptionFilter

pop eax ;restores the calculated CRC

cmp dword ptr, CRC, eax ;compares the calculated CRC with the saved one

jnz jump ;if the CRCs do not match, the program was

;either changed or a breakpoint was set

continue:

call MessageBoxA,O, offset message2, offset message1,O call ExitProcess, -1

jump:

call MessageBoxA,O, offset message3, offset message1,O call ExitProcess, -1

error: ;sets a new SEH service in case of an error

mov esp, [delayESP] push offset continue ret

;The routine for the CRC calculation ;input:

;ESI = CRCTable address

;EDI = beginning of the data address

;ECX = number of bytes for the calculation;

;output

;EAX = 32-bit CRC result

CRC32:

mov eax, OFFFFFFFFh mov edx,eax CRC32_1: push ecx

xor ebx,ebx

mov bl,byte ptr [edi]

inc edi

xor bl,al

shl bx,1 shl bx,1

add ebx,esi

mov cx,word ptr [ebx+2]

mov bx,word ptr [ebx]

mov al,ah

mov ah,dl

mov dl,dh

xor dh,dh

xor ax,bx

xor dx,cx

pop ecx

loop CRC32_1

not ax not dx push dx push ax pop eax

ret

End:

;initializes the eax and the edx

;saves the ecx into OFFFFFFFFh

;the number of bytes for the calculation

;reads 1 byte from the data ;and moves to the next





;sets according to the read byte in the CRC

;table

;reads a 32-bit

;the value from the CRC table

;moves DX:AX to the right by 8 bytes





;XOR CX:BX into DX:AX

;restores the number of the non-calculated data ;inbytes

;jumps if not all has been calculated yet, and ;decreases the number of bytes in the ecx by 1 ;the final calculation of the CRC

;saves the 16-bit values ;into the stack

;and after it has been read back, the complete ;32-bit CRC will be inthe eax register ;return

ends

end Start

Checking changes in a program through the CRC is one of the best ways to protect software against changes in the code. It isn't a bad idea to use the trick in more places too, to prevent the simple removal of protection. The program will also be well protected against viruses and breakpoints.



Note I will not discuss the mathematics of the CRC calculation here because it is easy to find detailed documentation about it. You can find several examples on the CD with this book, including some for the C language.



This trick cannot be used to control breakpoints at the API calls, or at least not in such a way that the CRC results for the API functions would be saved somewhere in the program. (Their code is different for various Windows versions.) The only way to do this is to calculate the CRC of the API functions that the program uses right at the start of the program, and to check this CRC when the functions are called. (Doing so may slow the program significantly, though.)



The previous example tests the CRC for the code section of the program, but it doesn't check the data. It's a good idea to have an application check even its own data.



This trick works in all Windows versions.





Detecting Debug Breakpoints



In Chapter 7, in the "Finding an Active Debugger Through the DR7 Debug Register" section, I demonstrated how to use the DR7 debug register to determine whether a debugger is active in memory. X86 processors have other registers besides DR7, though, so we'll take a look at the DR0 through DR3 registers here, which contain debug breakpoints.



Unfortunately, it is only possible to work with these registers in ring0. As such, we can use the same trick for switching over into ring0 that I used in previous examples in Chapter 7.



.386

.MODEL FLAT,STDCALL locals

jumps

UNICODE=O

include w32.inc

Extrn SetUnhandledExceptionFilter : PROC

Interrupt equ 5 ;the interrupt numbers 1 or 3 will make

;debugging more difficult

.DATA

message1 db "Debug breakpoint detection",O message2 db "Debug breakpoint not found",O message3 db "Debug breakpoint found",O

delayESP dd O ;the ESP register saves here

previous dd O ;the ESP register will save the address of the

;previous SEH service here

.CODE Start:

;Sets SEH in case of an error

mov [delayESP], esp push offset error

call SetUnhandledExceptionFilter mov [previous], eax

push edx

sidt [delayesp-2] ;reads IDT into the stack

pop edx

add edx, (Interrupt*8)+4 ;reads the vector of the required interrupt

mov ebx,[edx]

mov bx,word ptr [edx-4] ;reads the address of the old service of the

;required interrupt

lea edi,InterruptHandler

mov [edx-4],di

ror edi,16

mov [edx+2],di

push ds

push es

int Interrupt

pop es pop ds

mov [edx-4],bx ror ebx,16 mov [edx+2],bx push eax



;sets the new interrupt service

;saves registers for security

;jumps into Ring0 (a newly defined INT 5h ;service)

;restores the registers

;sets the original INT 5h interrupt service ;saves the return value

;Sets the previous SEH service

push dword ptr [previous]

call SetUnhandledExceptionFilter

pop eax test eax,eax jnz jump

;restores the return value ;tests to see if eax=0

;if not, the program has found a debug ;breakpoint and it ends

continue:

call MessageBoxA,0, offset message2,\ offset message1,0 call ExitProcess, -1

jump:

call MessageBoxA,0, offset message3,\ offset message1,0 call ExitProcess, -1

error: ;sets a new SEH service if there is an error

mov esp, [delayESP] push offset continue ret

;Your new service INT 5h (runs in Ring0)

InterruptHandler:

mov eax, dr0

test ax,ax

jnz Debug_Breakpoint

mov eax,dr1

test ax,ax

jnz Debug_Breakpoint

mov eax,dr2

test ax,ax

jnz Debug_Breakpoint

mov eax,dr3

test ax,ax

jnz Debug_Breakpoint

iretd

;reads a value from the DR0 debug register ;tests to see if a breakpoint was set ;if so, the program jumps

;reads a value from the DR1 debug register ;tests to see if a breakpoint was set ;if so, the program jumps

;reads a value from the DR2 debug register ;tests to see if a breakpoint was set ;if so, the program jumps

;reads a value from the DR3 debug register ;tests to see if a breakpoint was set ;if so, the program jumps

;if a breakpoint was not set the program will ;return 0 into the eax register



Debug_Breakpoint:

mov eax,1

iretd

;sets the value 1 into the eax register to show ;that breakpoints are active ;jump back into Ring3

ends

end Start



This technique is one of the few ways to discover debug breakpoints, and it makes it possible to delete them without stopping the application in the debugger. However, rather than delete them, I recommend going to an incorrect end of the application.



Unfortunately, the trick works only in Windows 9x because of the need to switch over into ring0. One way to use it in Windows NT, Windows 2000, and Windows XP is to place it into a Sys driver running in ring0. The same is true for all tricks that exploit ring0.





Detecting User Debuggers



The following trick will only discover a user debugger—mostly older types of debuggers that work on a different principle than SoftICE and TRW. It will, for example, find a debugger that is located in WinDasm.



.386

.MODEL FLAT,STDCALL

locals jumps

UNICODE=O

include w32.inc

Extrn SetUnhandledExceptionFilter : PROC

.data

message1

message3

message2

delayESP

previous

.code

Start:

db "User debugger detection",O

db "Debugger found",O

db "Debugger not found",O

dd O dd O



;the ESP register saves here

;the ESP register will save the address of the ;previous SEH service here



;Sets SEH in case of an error

mov [delayESP], esp push offset error

call SetUnhandledExceptionFilter mov [previous], eax

xor eax,eax ;nulls the eax register

mov al, fs:[2Oh] ;reads a value. If the value is other thanO, a

;debugger is active
push eax ;saves the value

;Sets the previous SEH service

push dword ptr [previous]

call SetUnhandledExceptionFilter

pop eax ;restores the value

test eax,eax ;tests to see if the read value was O

jnz jump ;if not, then a debugger is active in memory and

;the program jumps

continue:

call MessageBoxA,O, offset message2,\ offset message1,O call ExitProcess, -1

jump:

call MessageBoxA,O, offset message3,\ offset message1,O call ExitProcess, -1

error:

;sets a new SEH service in case of an error

mov esp, [delayESP] push offset continue ret

ends

end Start



While this trick will not find either TRW or SoftICE, it works reliably with older debuggers.

Detecting User Debuggers Using the API Function IsDebuggerPresent



You can use the API function IsDebuggerPresent to find a debugger that is active in memory. You needn't enter any parameters for calling this function. If a debugger was found, the EAX register will contain something other than 0.



.386

.MODEL FLAT,STDCALL

locals jumps

UNICODE=0

include w32.inc

Extrn SetUnhandledExceptionFilter : PROC
Extrn IsDebuggerPresent : PROC

.data

message1

message3

message2

delayESP

previous

.code

Start:

db "User debugger detection with the API IsDebuggerPresent function",0

db "Debugger found",0

db "Debugger not found",0

dd 0 ;the ESP register saves here

dd 0 ;the ESP register saves the address of the

;previous SEH service here



;Sets SEH in case of an error

mov [delayESP], esp push offset error

call SetUnhandledExceptionFilter mov [previous], eax

call IsDebuggerPresent

push eax ;saves the return value

;Sets the previous SEH service

push dword ptr [previous]

call SetUnhandledExceptionFilter

pop eax ;restores the return value

test eax,eax ;tests to see if the return value was 0

jnz jump ;if not, a debugger is active in memory and the

;program jumps

continue:

call MessageBoxA,0, offset message2,\ offset message1,0 call ExitProcess, -1

jump:

call MessageBoxA,0, offset message3,\ offset message1,0 call ExitProcess, -1

error: ;sets a new SEH service in case of an error

mov esp, [delayESP] push offset continue ret

ends

end Start

This trick will not find either SoftICE or TRW, but it will certainly find user debuggers such as the one in WinDasm.