strace / ltrace

Purpose

Guide agents through tracing system calls with strace and library calls with ltrace — the most effective tools for diagnosing incorrect binary behaviour without a crash or debugger.

Triggers

• "My program behaves incorrectly — how do I trace what it's doing?"

• "How do I find what files a binary is opening?"

• "strace shows ENOENT — how do I interpret it?"

• "How do I trace network calls with strace?"

• "What is ltrace and how does it differ from strace?"

• "How do I trace a running process?"

Workflow

1. Basic strace usage

Trace all syscalls of a command

strace ./myapp arg1 arg2

Attach to running process

strace -p 12345

Trace child processes too (-f = follow fork)

strace -f ./myapp

Save to file (raw output — not stdout)

strace ./myapp 2> trace.txt

Most useful: timestamps + summary

strace -t -f ./myapp 2>&1 | head -100

2. Filter by syscall category

Trace file operations only

strace -e trace=file ./myapp

Trace network syscalls

strace -e trace=network ./myapp

Trace specific syscalls

strace -e trace=open,openat,read,write ./myapp

Trace process management

strace -e trace=process ./myapp

Trace memory operations

strace -e trace=memory ./myapp

Trace signals

strace -e trace=signal ./myapp

Multiple categories

strace -e trace=file,network ./myapp

Category Syscalls included

file

open, openat, stat, access, unlink, rename, ...

network

socket, connect, bind, accept, send, recv, ...

process

fork, exec, wait, clone, exit, ...

memory

mmap, munmap, mprotect, brk, ...

signal

kill, sigaction, sigprocmask, ...

ipc

pipe, socket pair, shmget, ...

desc

close, dup, poll, select, epoll, ...

3. Interpreting common errors

See return values and errors

strace -e trace=file ./myapp 2>&1 | grep -E "ENOENT|EPERM|EACCES|ENOTSUP"

Error Meaning Common cause

ENOENT

No such file or directory Config file missing, wrong path

EACCES

Permission denied File permissions, SELinux

EPERM

Operation not permitted Missing capability, suid needed

EADDRINUSE

Address already in use Port already bound

ETIMEDOUT

Connection timed out Network unreachable, firewall

ECONNREFUSED

Connection refused Server not listening

EAGAIN

Resource temporarily unavailable Non-blocking I/O, try again

ENOMEM

Out of memory Allocation failed

EBADF

Bad file descriptor Using closed/invalid fd

ENOEXEC

Exec format error Wrong binary format for arch

Find what file is not found

strace ./myapp 2>&1 | grep 'ENOENT'

Example output:

openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file or directory)

→ Config file expected at /etc/myapp.conf

4. Useful strace flags

Show strings fully (default truncates at 32 chars)

strace -s 256 ./myapp

Timestamps

strace -t ./myapp # wall clock time strace -T ./myapp # time spent in each syscall strace -r ./myapp # relative timestamps

System call count summary

strace -c ./myapp

Shows count, time, errors per syscall — great for profiling

Trace with PIDs in output (for -f)

strace -f -p ./myapp

Output: [pid 12346] open("/etc/passwd", O_RDONLY) = 3

Decode numerical arguments

strace -e verbose=all ./myapp

Print instruction pointer at each syscall

strace -i ./myapp

5. ltrace — library call tracing

Trace all library calls

ltrace ./myapp

Trace specific library function

ltrace -e malloc,free,fopen ./myapp

Trace nested calls (lib → lib)

ltrace -n 2 ./myapp # indent nested calls

Trace with syscalls too

ltrace -S ./myapp

Attach to running process

ltrace -p 12345

Summary statistics

ltrace -c ./myapp

Typical ltrace output:

malloc(1024) = 0x55a1b2c3d000 fopen("/etc/myapp.conf", "r") = 0 free(0x55a1b2c3d000) = <void>

strace vs ltrace:

strace ltrace

Traces Kernel syscalls User-space library calls

Overhead Lower Higher (PLT hooking)

Shows open() , read() , write()

fopen() , malloc() , printf()

Use when Binary interacts with OS/files/network Binary calls library functions you can't see

6. Practical diagnosis workflows

Find missing config file

strace -e trace=openat,open ./myapp 2>&1 | grep ENOENT

Find what network connections are made

strace -e trace=network -f ./myapp 2>&1 | grep connect

Debug dynamic library loading failures

strace -e trace=openat ./myapp 2>&1 | grep ".so"

Find permission issues

strace -e trace=file ./myapp 2>&1 | grep -E "EACCES|EPERM"

Debug slow startup (find where time is spent)

strace -c ./myapp 2>&1

Look for high % time in unexpected syscalls

Watch IPC/shared memory

strace -e trace=ipc,shm ./myapp

Find what the binary exec's

strace -e trace=execve -f ./myapp

7. seccomp filter debugging

If a program is killed by a seccomp policy, strace reveals which syscall triggered it:

strace -e trace=all ./myapp 2>&1 | tail -5

Often shows the last syscall before SIGSYS

For strace output patterns and ltrace filtering examples, see references/strace-patterns.md.

Related skills

• Use skills/debuggers/gdb when strace shows the failing location and you need to inspect internals

• Use skills/binaries/elf-inspection to understand what libraries and symbols a binary uses

• Use skills/binaries/dynamic-linking for diagnosing LD_* and library loading issues

• Use skills/profilers/linux-perf for performance profiling (strace overhead is too high for perf)