Debugger System Design

This isn’t a very common question but I faced this in one of the interviews. Where the interviewer one line question was “If you have to design your own Debugger how can you do it?”.

Please provide your comments on it to improve it or if you have a better design.

Questions I asked:

• Can we use some external library? as I need some system call to control the target program. -Answer was yes….
• So here I will design a Debugger which will run a piece of code on file and will stop at a breakpoint that we were given by us.

So I suggested using Ptrace(). Though I didn't have much knowledge of it.

Ptrace(): Ptrace is a system call that is present in Linux/Unix Operating systems, it gets control over the target programs.

System Design:

We basically need a system call , which is a process( let me call it tracer process) that will observe and control another process (execution process) — that is PTRACE().

the execution process.

When you’re running an executable there’s a CPU register called the Program Counter (aka the Instruction Pointer) which contains the address of the next instruction to run. And also the value of the breakpoint to match.

How to get the break point?:

As each instruction runs, the Program Counter is updated to point to the next instruction. If the CPU finds the instruction code x8133, it will stop execution and issue a signal.

After getting the breakpoint:

Let's say “syscall ” is our process when we reach the breakpoint the process will get into the waiting stage.

Syscall.wait() — this wait() function can have two parameters, 1. Pid, Boolean signal (0= stop, 1= proceed). And initially, it will take 0 by default.

LLD:

PTRACE already has the function to execute a process, pick the breakpoint and get the line of the code, etc. But we are writing our own functions so something similar functions we need to create. So I suggested the below functions.

1. A function that allows it to run the next machine code instruction

func DebugeringleStep(pid int) (err error)

2. We need to get the process register (accessible location available to a computer’s processor)

DebugerGetRegs(pid int, regsout *PtraceRegs) (err error) 
DebugerSetRegs(pid int, regs *PtraceRegs) (err error)

3. To get/print the current memory of breakpoint/ the tracer process/pointer is on

DebugerPeekData(pid int, addr uintptr, out []byte) (count int, err error) – print target memory
DebugerPokeData(pid int, addr uintptr, data []byte) (count int, err error) – print the target memory

4. We can have more enhanced functions not just to get the memory but to get the text/variable or line of code.

DebugerPeekText(pid int, addr uintptr, out []byte) (count int, err error)
DebugerPokeText(pid int, addr uintptr, data []byte) (count int, err error)

5. A function to restart and keep running until the next system call, param= pid, and signal from hardware to proceed.

DebugerSyscall(pid int, signal int) (err error) 

This was all I explain as part of a system design of designing my own Debugger. Besides that, I felt like adding a few more enhanced features like skipping a function, skipping a block, and jumping back word and forward but the interviewer was happy with this. Adding these functionality can be explained by adding a few new functions under LLD.

Please let me know if this can be improved, if anything wrong with the approach or if you are expecting more on it.