Introduction to the debug-server Automated Debugging Tool

TOC

  1. 1. Dependencies
  2. 2. Usage Example
    1. 2.1. Simple Debugging Example
    2. 2.2. Automatic Breakpoints
    3. 2.3. Setting Breakpoints in lib Libraries
    4. 2.4. Debugging Network Applications
    5. 2.5. Disabling Randomization
    6. 2.6. Example of Using strace
    7. 2.7. Pausing at the Entry Point
  3. 3. Open Source Support

Repository: https://github.com/Ex-Origin/debug-server

debug-server

This tool does not require users to install a large number of debugging tools in remote or custom environments. Using the gdbserver program, it automatically attaches to the target.

In addition, this tool supports one-click launch of the strace program to observe current system calls and the current memory mapping addresses of the program.

The main program is written in C, making it easy to compile and use on Linux systems, helping you solve cross-architecture issues.

The usage is as follows:

Usage: debug-server [-hmsvn] [-e CMD] [-p PID] [-o CMD]

General:
  -e CMD   service argv
  -p PID   attach to PID
  -o CMD   get pid by popen
  -h       print help message
  -m       enable multi-service
  -s       halt at entry point
  -v       show debug information
  -n       disable address space randomization
  -u       do not limit memory

The debug-server provides several simple interfaces:

Dependencies

The remote environment needs to have gdbserver and strace installed to ensure the service operates correctly.

The local environment needs to have gdb-multiarch and pwntools installed to ensure the service operates correctly.

Usage Example

The remote and local environments can be the same, but to highlight the convenience of debug-server for embedded systems, the remote environment used here is an aarch64 architecture system.

The remote environment uses a non-desktop Debian GNU/Linux 12 (bookworm) aarch64 architecture.

The local environment uses a desktop Ubuntu 24.04 LTS x86_64 architecture.

An example of remote testing is as follows:

// aarch64-linux-gnu-gcc -g echo.c -o echo
#include <unistd.h>

int main()
{
    while(1)
    {
        char buf[0x100] = {0};
        int result = 0;
        write(STDOUT_FILENO, "Input: ", 7);
        result = read(STDIN_FILENO, buf, sizeof(buf)-1);
        write(STDOUT_FILENO, "Output: ", 8);
        write(STDOUT_FILENO, buf, result);
    }
    return 0;
}

The corresponding dependencies are as follows:

~ # ldd ./echo 
    linux-vdso.so.1 (0x0000ffffbd14a000)
    libc.so.6 => /lib/libc.so.6 (0x0000ffffbcf30000)
    /lib/ld-linux-aarch64.so.1 (0x0000ffffbd10d000)

Enter the following command in the remote environment to start the debugging service for the target program:

~ # ./debug-server -e ./echo
2024-05-19 18:16:25 | INFO    | Start debugging service, pid=160, version=1.3.3

Then, use gdbpwn.py locally to connect to the corresponding remote IP:

$ gdbpwn.py 192.168.1.8
2024-05-19 18:17:28,277 : INFO : Connecting to 192.168.1.8:9545
2024-05-19 18:17:28,282 : INFO : It has connected successfully
2024-05-19 18:17:28,282 : INFO : Start gdb client

Next, insert the required debugging code into exp.py:

#!/usr/bin/env python3
# -*- coding:utf-8 -*-

from pwn import *
context.clear(arch='amd64', os='linux', log_level='debug')

attach_host = '192.168.1.8'
attach_port = 9545
def attach(script=''):
    tmp_sock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    gdb_script = re.sub(r'#.*', '', 
f'''
define pr
    x/16gx $rebase(0x0)
end

b *$rebase(0x0)
''' + '\n' + script)
    gdbinit = '/tmp/gdb_script_' + attach_host
    script_f = open(gdbinit, 'w')
    script_f.write(gdb_script)
    script_f.close()
    _attach_host = attach_host
    if attach_host.find(':') == -1: _attach_host = '::ffff:' + attach_host
    tmp_sock.sendto(struct.pack('BB', 0x02, len(gdbinit.encode())) + gdbinit.encode(), (_attach_host, attach_port))
    tmp_sock.recvfrom(4096)
    tmp_sock.close()
    print('attach successfully')
def strace():
    tmp_sock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP) # UDP
    _attach_host = attach_host
    if attach_host.find(':') == -1: _attach_host = '::ffff:' + attach_host
    tmp_sock.sendto(struct.pack('B', 0x03), (_attach_host, attach_port))
    tmp_sock.recvfrom(4096)
    tmp_sock.close()
    print('strace successfully')
def address(search:str)->int:
    tmp_sock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    _attach_host = attach_host
    if attach_host.find(':') == -1: _attach_host = '::ffff:' + attach_host
    tmp_sock.sendto(struct.pack('BB', 0x04, len(search.encode())) + search.encode(), (_attach_host, attach_port))
    tmp_recv = tmp_sock.recvfrom(4096)[0]
    tmp_sock.close()
    return struct.unpack('Q', tmp_recv[2:10])[0]
def run_service():
    tmp_sock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP) # UDP
    _attach_host = attach_host
    if attach_host.find(':') == -1: _attach_host = '::ffff:' + attach_host
    tmp_sock.sendto(struct.pack('B', 0x06), (_attach_host, attach_port))
    tmp_sock.recvfrom(4096)
    tmp_sock.close()
    print('run_service successfully')

'''
Your Code
'''

Your Code refers to the debugging code you need to write.

Simple Debugging Example

Here’s a debugging example. You just need to replace the Your Code section with the code below. This example demonstrates how to debug a program simply.

sh = remote(attach_host, 9541)
sh.recvuntil(b'Input: ')
attach()
sh.sendline(b'Hello world')
sh.interactive()

The output of exp.py locally is as follows:

$ python3 exp.py 
[+] Opening connection to 192.168.1.8 on port 9541: Done
[DEBUG] Received 0x7 bytes:
    b'Input: '
attach successfully
[DEBUG] Sent 0xc bytes:
    b'Hello world\n'
[*] Switching to interactive mode
$

The output of gdbpwn.py locally is as follows:

$ gdbpwn.py 192.168.1.8
2024-05-19 18:17:28,277 : INFO : Connecting to 192.168.1.8:9545
2024-05-19 18:17:28,282 : INFO : It has connected successfully
2024-05-19 18:17:28,282 : INFO : Start gdb client
2024-05-19 18:21:49,450 : INFO : Receive COMMAND_GDBSERVER_ATTACH
pwndbg: loaded 157 pwndbg commands and 46 shell commands. Type pwndbg [--shell | --all] [filter] for a list.
pwndbg: created $rebase, $base, $ida GDB functions (can be used with print/break)
Remote debugging using ::ffff:192.168.1.8:9549
...
Breakpoint 1 at 0xaaaab8580000
...
─────────────────────[ DISASM / aarch64 / set emulate on ]──────────────────────
 ► 0xffff9a219e64 <read+36>    svc    #0 <SYS_read>
        fd: 0 (socket:[3088])
        buf: 0xffffd4b72538 ◂— 0
        nbytes: 0xff
   0xffff9a219e68 <read+40>    mov    x19, x0
   0xffff9a219e6c <read+44>    cmn    x0, #1, lsl #12
   0xffff9a219e70 <read+48>    b.hi   #read+148                   <read+148>
 
   0xffff9a219e74 <read+52>    mov    x0, x19
   0xffff9a219e78 <read+56>    ldp    x19, x20, [sp, #0x10]
   0xffff9a219e7c <read+60>    ldp    x29, x30, [sp], #0x30
   0xffff9a219e80 <read+64>    ret    
 
   0xffff9a219e84 <read+68>    mov    x20, x2
   0xffff9a219e88 <read+72>    str    x21, [sp, #0x20]
   0xffff9a219e8c <read+76>    mov    x21, x1
───────────────────────────────────[ STACK ]────────────────────────────────────
...
pwndbg>

Automatic Breakpoints

Here is an example demonstrating how to automatically set breakpoints. First, let’s look at the disassembly of the echo program:

$ aarch64-linux-gnu-objdump -d ./echo 
...
0000000000000814 <main>:
 814:    d10483ff 	sub	sp, sp, #0x120
 818:    a9117bfd 	stp	x29, x30, [sp, #272]
 81c:    910443fd 	add	x29, sp, #0x110
 820:    f00000e0 	adrp	x0, 1f000 <__FRAME_END__+0x1e62c>
 824:    f947f400 	ldr	x0, [x0, #4072]
 828:    f9400001 	ldr	x1, [x0]
 82c:    f90087e1 	str	x1, [sp, #264]
 830:    d2800001 	mov	x1, #0x0                   	// #0
 834:    910023e0 	add	x0, sp, #0x8
 838:    4f000400 	movi	v0.4s, #0x0
 83c:    ad000000 	stp	q0, q0, [x0]
 840:    ad010000 	stp	q0, q0, [x0, #32]
 844:    ad020000 	stp	q0, q0, [x0, #64]
 848:    ad030000 	stp	q0, q0, [x0, #96]
 84c:    ad040000 	stp	q0, q0, [x0, #128]
 850:    ad050000 	stp	q0, q0, [x0, #160]
 854:    ad060000 	stp	q0, q0, [x0, #192]
 858:    ad070000 	stp	q0, q0, [x0, #224]
 85c:    b90007ff 	str	wzr, [sp, #4]
 860:    d28000e2 	mov	x2, #0x7                   	// #7
 864:    90000000 	adrp	x0, 0 <__abi_tag-0x278>
 868:    91238001 	add	x1, x0, #0x8e0
 86c:    52800020 	mov	w0, #0x1                   	// #1
 870:    97ffff98 	bl	6d0 <write@plt>
 874:    910023e0 	add	x0, sp, #0x8
 878:    d2801fe2 	mov	x2, #0xff                  	// #255
 87c:    aa0003e1 	mov	x1, x0
 880:    52800000 	mov	w0, #0x0                   	// #0
 884:    97ffff9b 	bl	6f0 <read@plt>
 888:    b90007e0 	str	w0, [sp, #4]
 88c:    d2800102 	mov	x2, #0x8                   	// #8
 890:    90000000 	adrp	x0, 0 <__abi_tag-0x278>
 894:    9123a001 	add	x1, x0, #0x8e8
 898:    52800020 	mov	w0, #0x1                   	// #1
 89c:    97ffff8d 	bl	6d0 <write@plt>
 8a0:    b98007e1 	ldrsw	x1, [sp, #4]
 8a4:    910023e0 	add	x0, sp, #0x8
 8a8:    aa0103e2 	mov	x2, x1
 8ac:    aa0003e1 	mov	x1, x0
 8b0:    52800020 	mov	w0, #0x1                   	// #1
 8b4:    97ffff87 	bl	6d0 <write@plt>
 8b8:    d503201f 	nop
 8bc:    17ffffde 	b	834 <main+0x20>

This time, our goal is to set a breakpoint at 89c. The corresponding debug code is as follows:

sh = remote(attach_host, 9541)
sh.recvuntil(b'Input: ')
attach(
f'''
b *{address("echo")+0x89c}
c
''')
sh.sendline(b'Hello world')
sh.interactive()

After setting the breakpoint at 89c, the c (continue) command is executed immediately. This allows the program to run directly to the 89c breakpoint without manual adjustments, significantly speeding up the debugging process.

The local output of gdbpwn.py is as follows:

Breakpoint 1 at 0xaaaae31d0000
Breakpoint 2 at 0xaaaae31d089c: file echo.c, line 12.

Breakpoint 2, 0x0000aaaae31d089c in main () at echo.c:12
12            write(STDOUT_FILENO, "Output: ", 8);
...
─────────────────────[ DISASM / aarch64 / set emulate on ]──────────────────────
 ► 0xaaaae31d089c <main+136>    bl     #write@plt                  <write@plt>
        fd: 1 (socket:[3150])
        buf: 0xaaaae31d08e8 ◂— adr x15, #0xaaaae32b9793 /* 'Output: ' */
        n: 8
 
   0xaaaae31d08a0 <main+140>    ldrsw  x1, [sp, #4]
   0xaaaae31d08a4 <main+144>    add    x0, sp, #8
   0xaaaae31d08a8 <main+148>    mov    x2, x1
   0xaaaae31d08ac <main+152>    mov    x1, x0
   0xaaaae31d08b0 <main+156>    mov    w0, #1
   0xaaaae31d08b4 <main+160>    bl     #write@plt                  <write@plt>
 
   0xaaaae31d08b8 <main+164>    nop    
   0xaaaae31d08bc <main+168>    b      #main+32                    <main+32>
 
   0xaaaae31d08c0 <_fini>       nop    
   0xaaaae31d08c4 <_fini+4>     stp    x29, x30, [sp, #-0x10]!
───────────────────────────────[ SOURCE (CODE) ]────────────────────────────────

Setting Breakpoints in lib Libraries

Typically, using gdb to set breakpoints for programs like PHP and Nginx, which require importing lib libraries, is challenging. However, this debugging mode can address this limitation.

For example, if you need to set a breakpoint at the 8th byte offset of the entry point of the write function, the disassembly of libc.so.6 is as follows:

$ $ aarch64-linux-gnu-objdump -d libc.so.6
...
00000000000d9f10 <__write@@GLIBC_2.17>:
   d9f10:    a9bd7bfd 	stp	x29, x30, [sp, #-48]!
   d9f14:    d0000663 	adrp	x3, 1a7000 <getdate_err@@GLIBC_2.17+0x338>
   d9f18:    910003fd 	mov	x29, sp
   d9f1c:    3967a063 	ldrb	w3, [x3, #2536]

The debug framework code for this would be:

sh = remote(attach_host, 9541)
sh.recvuntil(b'Input: ')
attach(
f'''
b *{address("libc.so.6")+0xd9f18}
c
''')
sh.sendline(b'Hello world')
sh.interactive()

First, the address of libc.so.6 is obtained, and then a breakpoint is set at the corresponding offset. This process is automated, facilitating debugging of lib libraries. This is especially useful for libraries without symbol functions, making the debugging process significantly more efficient.

The local output of gdbpwn.py is as follows:

Breakpoint 1 at 0xaaaab94f0000
Breakpoint 2 at 0xffffb6b09f18

Breakpoint 2, 0x0000ffffb6b09f18 in write () from target:/lib/libc.so.6
...
─────────────────────[ DISASM / aarch64 / set emulate on ]──────────────────────
 ► 0xffffb6b09f18 <write+8>     mov    x29, sp                   FP => 0xffffc4e3bd90
   0xffffb6b09f1c <write+12>    ldrb   w3, [x3, #0x9e8]
   0xffffb6b09f20 <write+16>    stp    x19, x20, [sp, #0x10]
   0xffffb6b09f24 <write+20>    sxtw   x19, w0
   0xffffb6b09f28 <write+24>    cbz    w3, #write+68               <write+68>
 
   0xffffb6b09f2c <write+28>    mov    x0, x19                   X0 => 1
   0xffffb6b09f30 <write+32>    mov    x8, #0x40                 X8 => 0x40
   0xffffb6b09f34 <write+36>    svc    #0
   0xffffb6b09f38 <write+40>    mov    x19, x0
   0xffffb6b09f3c <write+44>    cmn    x0, #1, lsl #12
   0xffffb6b09f40 <write+48>    b.hi   #write+148                  <write+148>
───────────────────────────────────[ STACK ]────────────────────────────────────

Debugging Network Applications

For network applications like Nginx and sshd, since they interact through sockets rather than standard input and output streams, the debugging process is often very cumbersome. Additionally, when problems arise, it usually requires restarting the service.

Here, we demonstrate using debug-server to debug such network programs.

On the remote environment, enter the following command to start debugging the target program:

./debug-server -e /usr/sbin/sshd -o 'pidof sshd'

The corresponding debugging code is as follows:

run_service()
time.sleep(1)
attach()
sh = remote(attach_host, 22)
sh.send(b'aaaa')
sh.interactive()

Before starting this debugging process, ensure that there are no already running sshd services on the current system.

The run_service() function will execute /usr/sbin/sshd, then time.sleep(1) pauses the debugging script for 1 second to ensure the sshd service starts successfully. The attach() function attaches to the target process, locating its PID by executing the popen function with the pidof sshd command. This corresponds to the -o 'pidof sshd' parameter.

This debugging method can greatly simplify the debugging process for network applications.

Disabling Randomization

To disable randomization for the target program when starting the remote debugging service, add the -n parameter.

./debug-server -n -e ./echo

This operation only affects the target program and does not change the system’s randomization rules, thus maintaining higher security.

Example of Using strace

For strace support, simply replace the attach() function with the strace() function.

The corresponding debugging code is as follows:

sh = remote(attach_host, 9541)
sh.recvuntil(b'Input: ')
strace()
sh.sendline(b'Hello world')
sh.interactive()

The corresponding remote output log is as follows:

2024-05-19 21:02:46 | INFO    | Strace start, pid=389
strace: Process 388 attached
read(0, "Hello world\n", 255)           = 12
write(1, "Output: ", 8)                 = 8
write(1, "Hello world\n", 12)           = 12
write(1, "Input: ", 7)                  = 7
read(0,

Using this method, you can observe the system calls of the specified code, making it easier for researchers to understand the program.

Pausing at the Entry Point

For some programs without IO or programs that need modifications before the entry function, debugging with pwntools can be very cumbersome.

The debug-server can use the -s parameter to pause the program at the entry point, making it convenient to perform specific initializations, which is especially helpful for reverse engineering programs without IO.

The command to start the remote debugging service is as follows:

./debug-server -s -e ./echo

The corresponding debugging script is as follows:

sh = remote(attach_host, 9541)
attach(
f'''
b main
c
c
''')
sh.recvuntil(b'Input: ')
sh.sendline(b'Hello world')
sh.interactive()

Since debug-server uses the SIGSTOP signal to pause the program, the first c (continue) command handles the SIGSTOP signal, and the second c command continues the program execution.

The local gdbpwn.py output is as follows:

Breakpoint 1 at 0xaaaaac560000
Breakpoint 2 at 0xaaaaac560820: file echo.c, line 5.

Program received signal SIGCONT, Continued.
0x0000ffff8a2d2980 in ?? () from target:/lib/ld-linux-aarch64.so.1
...
Breakpoint 2, main () at echo.c:5
...
─────────────────────[ DISASM / aarch64 / set emulate on ]──────────────────────
 ► 0xaaaaac560820 <main+12>    adrp   x0, #0xaaaaac57f000     X0 => 0xaaaaac57f000 ◂— 0
   0xaaaaac560824 <main+16>    ldr    x0, [x0, #0xfe8]        X0 => 0xffff8a2f7b88 (__stack_chk_guard) ◂— 0x657551da6d76f000
   0xaaaaac560828 <main+20>    ldr    x1, [x0]                X1 => 0x657551da6d76f000
   0xaaaaac56082c <main+24>    str    x1, [sp, #0x108]
   0xaaaaac560830 <main+28>    mov    x1, #0                  X1 => 0
   0xaaaaac560834 <main+32>    add    x0, sp, #8              X0 => 0xffffdf9036d8 —▸ 0xffff8a2f1f60 ◂— 0
   0xaaaaac560838 <main+36>    movi   v0.4s, #0
   0xaaaaac56083c <main+40>    stp    q0, q0, [x0]
   0xaaaaac560840 <main+44>    stp    q0, q0, [x0, #0x20]
   0xaaaaac560844 <main+48>    stp    q0, q0, [x0, #0x40]
   0xaaaaac560848 <main+52>    stp    q0, q0, [x0, #0x60]
───────────────────────────────[ SOURCE (CODE) ]────────────────────────────────

If the entry point pause function is not enabled, the program will not stop at the entry point. The reason is that the IO speed is too fast. If the program is not paused, the IO speed will always be faster than the debugging speed, making it impossible to debug the code before the IO process.

Open Source Support

The debug-server uses the MIT open-source license. All interested geeks are welcome to join in the maintenance.