strace is a powerful diagnostic, instructional, and debugging utility for Linux that allows users to monitor the interactions between a specific process and the operating system kernel. By intercepting and recording system calls (syscalls) and signals, it provides a transparent view of how an application requests resources, handles files, and manages network communications in real-time.
Whether you are a software developer trying to figure out why an application is crashing or a system administrator investigating a "Permission Denied" error on a file that seemingly has the correct attributes, strace (often phonetically searched as "stuff trace") is the definitive tool for uncovering what is happening "under the hood." In 2026, despite the rise of more complex observability frameworks like eBPF, strace remains the go-to standard for immediate, granular process inspection.
The Fundamentals of System Call Tracing
To understand how to use strace effectively, one must first understand the concept of a system call. In a modern operating system, there is a strict separation between "user space" (where your applications run) and "kernel space" (where the core of the OS manages hardware). When an application needs to perform an action—such as reading a file, sending data over a network, or creating a new process—it cannot do so directly. Instead, it must execute a system call, which is a formal request to the kernel to perform the action on its behalf.
strace works by using a specific kernel feature called ptrace (process trace). This allows one process (the tracer) to observe and control the execution of another process (the tracee). When strace is attached to a command, it pauses the execution of that command every time it enters and exits a system call, logging the name of the call, the arguments passed to it, and the return value.
Why use strace instead of logs?
Application logs are only as good as the developer who wrote them. If a developer didn't include a log entry for a specific failure point, the log is useless. Strace, however, does not rely on application-level logging. It captures the interaction at the boundary of the OS. If a program attempts to open /etc/config.json and fails because the file isn't there, strace will show the openat() syscall failing with ENOENT (No such file or directory), regardless of whether the application explicitly reports that error.
Prerequisites and Environment Setup
In 2026, strace is pre-installed on most "server" and "developer" variants of Linux distributions (like Ubuntu Server, Fedora, or Arch). However, if you are working with a minimal container image or a fresh installation, you may need to install it manually.
Installation Commands
- Debian/Ubuntu/Linux Mint:
sudo apt update && sudo apt install strace - RHEL/CentOS/Fedora/AlmaLinux:
sudo dnf install strace - Arch Linux:
sudo pacman -S strace
Security and Permissions
Because strace allows you to inspect the memory and behavior of processes, it is a sensitive tool. By default, you can trace any process that you own. However, to trace a process owned by another user (like a root-owned web server) or to attach to an already running process, you generally need sudo privileges or the CAP_SYS_PTRACE capability.
In recent years, many Linux distributions have enabled a security feature called yama ptrace_scope. If you receive an error like "attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted" even when using sudo, you may need to temporarily adjust this setting:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
Basic Syntax and Getting Started
The simplest way to use strace is to provide it with the command you want to run.
strace ls /etc
This command will output a massive stream of text to the terminal. Each line represents a system call. Let’s break down a typical line:
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
- openat: The name of the system call (used to open files).
- (...): The arguments. Here, it’s looking at
/etc/passwdwith read-only flags. - = 3: The return value. In this case, '3' is the File Descriptor (FD) assigned to the file. If it failed, it would show
-1followed by an error code likeEACCES.
Common Command Options
While a raw strace is informative, it is often overwhelming. Use these flags to refine the output:
- -o : Redirects the output to a file instead of stderr. This is crucial because many programs you trace also output to the terminal, making the raw output unreadable.
- -p : Attaches strace to an already running process. This is the primary method for debugging services that are already hanging or behaving oddly.
- -s : Increases the maximum string size printed for arguments. By default, strace truncates long strings (like long file paths or data buffers) at 32 characters. Using
-s 1024is often necessary to see full SQL queries or file contents. - -v: Verbose mode. Displays unabbreviated versions of environment variables, stat structures, and more.
Advanced Filtering Techniques
Filtering is the difference between a novice and an expert user. You rarely need to see every system call. You usually only care about specific categories, like file operations or network traffic.
Filtering by System Call Type
Use the -e trace= flag to narrow your focus:
- File-related calls:
strace -e trace=file ls(Shows open, stat, chmod, etc.) - Network-related calls:
strace -e trace=network curl https://google.com(Shows socket, connect, sendto, etc.) - Process management:
strace -e trace=process ./my_app(Shows fork, execve, wait4, etc.)
Tracing Child Processes
By default, strace only monitors the specific process you started. If that process "forks" (creates a child process), the child is not traced. Use the -f (follow) flag to trace all children as they are created. This is essential for debugging shell scripts or complex servers like Apache or Nginx that spawn multiple worker threads or processes.
Timing and Statistics
If you are investigating performance bottlenecks, strace can act as a rudimentary profiler:
- -T: Prints the time spent in each system call at the end of each line. This helps identify which specific OS request is causing a delay (e.g., a slow DNS lookup or a disk I/O bottleneck).
- -c: Instead of a line-by-line log, it provides a summary table at the end, showing how many times each syscall was called and the total time spent in each.
Real-World Use Cases and Troubleshooting
Case 1: The "Missing" Config File
An application fails to start, claiming "Configuration not found," but you see the file in the directory.
Action: strace -e trace=openat -f ./application
Discovery: You might find the application is actually looking in /usr/local/etc/ instead of /etc/, or it’s trying to append a .conf extension that you didn't know was required.
Case 2: Silent Failures in Scripts
A bash script exits prematurely without an error message.
Action: strace -f -o trace.log ./script.sh
Discovery: Search the log for exit_group. Just before that, you might see a write() call to a file descriptor that failed with ENOSPC (No space left on device), which the script failed to report to the user.
Case 3: Investigating Slow Network Connections
A command-line tool is taking 30 seconds to respond.
Action: strace -r -e trace=network
Discovery: The -r flag shows the relative time between syscalls. You might see a huge gap between a connect() call to a DNS server and the subsequent recvfrom() call, indicating a DNS timeout.
Comparing strace with Related Tools
While strace is powerful, it is part of a larger ecosystem of Linux diagnostic tools. Depending on the depth of your issue, other tools might be more appropriate.
| Tool | Focus Area | Best For | Impact on Performance |
|---|---|---|---|
| strace | System Calls (User-Kernel boundary) | Debugging file access, permissions, network sockets. | High (Significant slowdown) |
| ltrace | Library Calls (User-Library boundary) | Debugging calls to libc, openssl, or custom .so files. |
High |
| gdb | Code Logic / Memory | Step-by-step debugging of source code, inspecting variables. | Very High (Pauses execution) |
| tcpdump | Network Packets | Analyzing the actual data sent over the wire (TCP/UDP). | Low to Medium |
| bpftrace | Kernel/System-wide tracing | Production performance monitoring and low-level kernel hooks. | Very Low |
Performance Considerations and Limitations
It is critical to note that strace carries a heavy performance penalty. Because the kernel must pause the process at every syscall entry and exit to let strace record the data, the application can run 10x to 100x slower than normal.
Do not run strace -f on a high-traffic production database or web server unless absolutely necessary. If you must trace a production process, use it sparingly:
- Attach to a specific PID for only a few seconds.
- Use specific filters (
-e) to reduce the amount of data captured. - Output to a fast filesystem (like a RAM disk /dev/shm) to avoid adding I/O wait to the bottleneck.
For 2026-era high-performance environments, eBPF-based tools (like those found in the bcc or bpftrace suites) are preferred for production monitoring. eBPF programs run inside the kernel and do not require pausing the user-space process, making them safe for heavy loads. However, they are harder to write and interpret than a simple strace log.
Frequently Asked Questions
Can strace be used to debug a process that is already running?
Yes, using the -p flag followed by the Process ID (PID). For example, sudo strace -p 1234 will attach to process 1234. When you are finished, press Ctrl+C to detach. The original process will continue running normally.
Why does strace show "Permission Denied" when I am running as root?
Even root can be restricted by Linux Security Modules (LSMs) like SELinux or AppArmor. If strace shows an EACCES or EPERM error for a root-owned process, check the audit logs (/var/log/audit/audit.log or dmesg) to see if a security policy is blocking the action.
Is it possible to see the data being written to a file or socket?
Yes. Use the -e write=all or -e read=all flags, or specifically -e write=3 to see data written to file descriptor 3. To see the full content without truncation, remember to use the -s flag (e.g., -s 2000).
Can strace capture passwords or sensitive data?
Absolutely. If an application reads a password from a configuration file or from user input via a system call (like read), strace will capture that data in plain text in the log. Always handle strace logs with the same security precautions you would use for the sensitive data themselves.
Does strace work on macOS or Windows?
No, strace is specific to the Linux kernel and its syscall interface. macOS uses a similar tool called dtruss (based on DTrace). Windows has no direct equivalent, though the Sysinternals Process Monitor (ProcMon) provides similar functionality through a GUI by monitoring registry and file system activity.
Practical Recommendation
When faced with a mysterious application failure, follow this workflow:
- Check logs first: Look at
/var/log/syslogor the application's own logs. - Run a summary: Use
strace -cto see if a specific syscall is failing frequently or taking too much time. - Targeted Trace: If you see many errors in
openat, runstrace -e trace=file -o debug.txt. - Analyze and Search: Open the output file and search for the string "ERR". This usually leads directly to the system call that triggered the failure.
By mastering strace, you move from guessing why a program is failing to having factual evidence provided directly by the operating system.
