System Calls vs. Libraries: The Real Story

strace doesn’t actually show you every libc call; it shows you every syscall. That’s a crucial distinction because libc is just a thin wrapper around the kernel’s actual system call interface.

Let’s see what’s really happening under the hood when you run a simple ls.

# Create a dummy file
touch /tmp/testfile

# Trace ls
strace ls /tmp/testfile

The output will be a deluge of system calls. You’ll see things like execve (the kernel loading the ls program), openat (opening the directory /tmp), newfstatat (getting file status), getdents64 (reading directory entries), write (printing the filename to your terminal), and exit_group (the program finishing).

Here’s how strace maps to libc, using openat as an example. When your C program calls openat(), it’s not a direct kernel instruction. Instead, the openat function in libc performs these steps:

Sets up arguments: It takes your filename, flags, and mode, and places them into specific CPU registers (like rdi, rsi, rdx on x86-64).
Issues syscall instruction: It executes the syscall instruction. This is a special CPU instruction that tells the processor to switch from user mode to kernel mode and jump to a specific handler in the kernel.
Kernel handler: The kernel uses the value in a register (like rax on x86-64) to identify which syscall you want. For openat, this number is 257. The kernel then reads the arguments from the other registers.
Performs action: The kernel executes the openat logic, which involves finding the file, checking permissions, and if successful, returning a file descriptor.
Returns to user space: The kernel places the result (e.g., the file descriptor or an error code) into the rax register and switches back to user mode, returning control to the libc openat function.
Libc cleanup: Libc checks the return value. If it’s an error (negative value), it often sets the global errno variable and might return -1 to your application. Otherwise, it returns the success value (like the file descriptor).

So, strace intercepts that syscall instruction before libc does its cleanup. It sees the raw syscall number and arguments the kernel is about to process, and the result the kernel returns before libc potentially modifies it.

What problem does this solve? Debugging. When a program behaves unexpectedly, especially with file I/O, network connections, or process management, strace is your primary tool. It bypasses the abstraction layers of libc and libraries, showing you exactly what the kernel is being asked to do and what it’s returning. This is invaluable for understanding why a file isn’t opening, why a network socket is failing, or why a process is exiting prematurely.

Consider this scenario: you’re trying to open a configuration file, and your application crashes with an ENOENT (No such file or directory) error, but you swear the file is there. strace will show you the exact path your application is passing to openat. You might discover it’s looking in /etc/myapp/config.conf when it should be /usr/local/etc/myapp/config.conf.

The most surprising thing about strace is how much of "normal" program execution is actually a series of requests to the kernel. Every time a program needs to do anything that interacts with the outside world – read a file, send data over the network, allocate memory, create a process – it has to ask the kernel. strace makes this explicit. It reveals that even simple operations are complex sequences of privileged kernel instructions.

The setuidgid system call, for instance, is what allows a process to change its effective user ID. When you see setuidgid in strace, it means the program is asking the kernel to drop or change its privileges. This is a critical security boundary.

The next concept you’ll likely encounter is ltrace, which traces library calls, showing you the libc functions themselves rather than the underlying syscalls.