Magic-trace is a tool developed by Jane Street that allows for high-resolution, low-overhead tracing of program execution. It leverages Intel Processor Trace technology to capture detailed information about a program's behavior, including function calls, branching, and memory access patterns, with minimal impact on performance.
Pros
Provides extremely detailed program execution traces with low overhead
Supports both native and OCaml programs
Offers a user-friendly interface for analyzing and visualizing trace data
Can be used for performance analysis, debugging, and understanding complex codebases
Cons
Currently only supports Intel processors with Processor Trace capability
May require some setup and configuration to use effectively
Large trace files can be generated, potentially requiring significant storage space
Learning curve for interpreting and utilizing the trace data effectively
Getting Started
To get started with magic-trace:
Install the tool:
opam install magic-trace
Run magic-trace on your program:
magic-trace -- your_program [args]
Analyze the trace using the provided viewer:
magic-trace view trace.out
For more detailed instructions and advanced usage, refer to the project's README and documentation on GitHub.
FlameGraph is primarily written in Perl, while magic-trace is implemented in OCaml
FlameGraph focuses on generating flame graphs from collected data, whereas magic-trace provides a more comprehensive tracing solution
magic-trace offers real-time tracing capabilities, while FlameGraph typically works with pre-collected data
FlameGraph has a broader language and platform support, while magic-trace is more specialized for certain environments
Both tools serve valuable purposes in performance analysis, with FlameGraph being more established and versatile, and magic-trace offering more advanced features for specific use cases.
import"net/http/pprof"funcmain(){gofunc(){ log.Println(http.ListenAndServe("localhost:6060",nil))}()// ... rest of the program}
magic-trace focuses on capturing system-level events using hardware tracing, while pprof typically requires code instrumentation or runtime support for profiling. magic-trace provides a simpler command-line interface for capturing and viewing traces, whereas pprof offers a more comprehensive set of analysis tools and visualizations through its web interface.
Both tools serve different use cases: magic-trace is ideal for low-level performance analysis with minimal overhead, while pprof is better suited for general-purpose profiling across various languages and runtime environments.
Allows for subsecond offset selection for detailed analysis
Cons of Flamescope
Limited to CPU profiling data
Requires separate data collection and import process
Less detailed system-wide tracing compared to Magic-trace
Code Comparison
Magic-trace:
let trace_command ~output_file command =
let pid = Unix.fork () in
if pid = 0 then
Unix.execvp command.(0) command
else
trace_pid ~output_file pid
Flamescope:
defparse_perf_script(input_file):
stack_counts = collections.defaultdict(int)for line in input_file:
stack = line.strip().split(';')
stack_counts[tuple(stack)]+=1return stack_counts
Both projects aim to provide performance analysis tools, but they differ in their approach and implementation. Magic-trace focuses on system-wide tracing using Intel Processor Trace, while Flamescope specializes in visualizing CPU profiling data. Magic-trace is implemented in OCaml and provides a command-line interface, whereas Flamescope is built with Python and offers a web-based UI for interactive exploration of flame graphs.
Both tools generate flame graphs, but Magic-trace offers a built-in viewer, while Pyflame requires additional tools like flamegraph.pl for visualization.
Magic-trace provides a more comprehensive system-wide tracing capability, while Pyflame focuses specifically on Python profiling. Magic-trace is actively maintained and offers broader language support, making it more versatile for general-purpose profiling and debugging. However, Pyflame's Python-specific features may be advantageous for teams working primarily with Python applications.
Magic-trace focuses on low-overhead tracing with a command-line interface, while Hotspot provides a feature-rich GUI for performance analysis. Magic-trace offers simplicity and efficiency, whereas Hotspot excels in visualization and cross-platform support (excluding macOS). The choice between the two depends on the user's preference for CLI vs. GUI, the desired level of detail in performance analysis, and the target operating system.
#include<OrbitProfiler.h>ORBIT_SCOPE("MyFunction");// Your code here
Summary
Magic-trace focuses on lightweight, low-overhead tracing with easy setup, while Orbit offers a more comprehensive profiling suite with advanced visualization tools. Magic-trace is simpler to use but has limited features, whereas Orbit provides deeper insights at the cost of complexity and potential performance impact. The choice between them depends on the specific profiling needs and the level of detail required for the project.
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
traces every function call with ~40ns resolution, and
renders a timeline of call stacks going back (a configurable) ~10ms.
You use it like perf: point it to a process and off it goes. The key difference from perf is that instead of sampling call stacks throughout time, magic-trace uses Intel Processor Trace to snapshot a ring buffer of all control flow leading up to a chosen point in time1. Then, you can explore an interactive timeline of what happened.
You can point magic-trace at a function such that when your application calls it, magic-trace takes a snapshot. Alternatively, attach it to a running process and detach it with Ctrl+C, to see a trace of an arbitrary point in your program.
Testimonials
"Magic-trace is one of the simplest command-line debugging tools I have ever used."
Francis Ricci, Jane Street
"Magic-trace is not just for performance. The tool gives insight directly into what happens in your program, when, and why. Consider using it for all your introspective goals!"
Andrew Hunter, Jane Street
I use perf a ton, and I think that both perf and magic-trace give perspectives that the other doesn't. The benefit I got from magic-trace was entirely based on the fact that it works in slices at any zoom level, so I was able to see all the function calls that a 70ns function was performing, which was invisible in perf.
Make sure the system you want to trace is supported. The constraints that most commonly trip people up are: VMs are mostly not supported, Intel only (Skylake2 or later), Linux only.
If downloading the prebuilt binary (not package), chmod +x magic-trace3
If downloading the package, run sudo dpkg -i magic-trace*.deb
Then, test it by running magic-trace -help, which should bring up some help text.
Getting started
Here's a sample C program to try out. It's a slightly modified version of the example in man 3 dlopen. Download that, build it with gcc demo.c -ldl -o demo, then leave it running ./demo. We're going to use that program to learn how dlopen works.
Run magic-trace attach -pid $(pidof demo). When you see the message that it's successfully attached, wait a couple seconds and Ctrl+Cmagic-trace. It will output a file called trace.fxt.gz in your working directory.
Open magic-trace.org, click "Open trace file" in the top-left-hand and give it the trace file generated in the previous step.
That should have expanded into a trace. Zoom in until you can see an individual loop through dlopen/dlsym/cos/printf/dlclose.
W zooms into wherever your mouse cursor is pointed (you'll need to zoom in a bunch to see anything useful),
S zooms out,
A moves left,
D moves right, and
scroll wheel moves your viewport up and down the stack. You'll only need to scroll to see particularly deep stack traces, it's probably not useful for this example.
Click and drag on the white space around the call stacks to measure. Plant flags by clicking in the timeline along the top. Using the measurement tool, measure how long it takes to run cos. On my screen it takes ~5.7us.
Congratulations, you just magically traced your first program!
In contrast to traditional perf workflows, magic-trace excels at hypothesis generation. For example, you might notice that taking 6us to run cos is a really long time! If you zoom in even more, you'll see that there's actually five pink "[untraced]" cells in there. If you re-run magic-trace with root and pass it -trace-include-kernel, you'll see stacktraces for those. They're page fault handlers! The demo program actually calls cos twice. If you zoom in even more near the end of the 6us cos call, you'll see that the second call takes far less time and does not page fault.
How to use it
magic-trace continuously records control flow into a ring buffer. Upon some sort of trigger, it takes a snapshot of that buffer and reconstructs call stacks.
There are two ways to take a snapshot:
We just did this one: Ctrl+C magic-trace. If magic-trace terminates without already having taken a snapshot, it takes a snapshot of the end of the program.
You can also trigger snapshots when the application calls a function. To do so, pass magic-trace
the -trigger flag.
-trigger '?' brings up a fuzzy-finding selector that lets you choose from all
symbols in your executable,
-trigger SYMBOL selects a specific, fully mangled, symbol you know ahead of time, and
-trigger . selects the default symbol magic_trace_stop_indicator.
Stop indicators are powerful. Here are some ideas for where you might want to place one:
If you're using an asynchronous runtime, any time a scheduler cycle takes too long.
In a server, when a request takes a surprisingly long time.
After the garbage collector runs, to see what it's doing and what it interrupted.
After a compiler pass has completed.
You may leave the stop indicator in production code. It doesn't need to do anything in particular, magic-trace just needs the name. It is just an empty, but not inlined, function. It will cost ~10us to call, but only when magic-trace actually uses it to take a snapshot.
Tristan Hume is the original author of magic-trace. He wrote it while working at Jane Street, who currently maintains it.
Intel PT is the foundational technology upon which magic-trace rests. We'd like to thank the people at Intel for their years-long efforts to make it available, despite its slow uptake in the greater software community.
magic-trace would not be possible without perfs extensive support for Intel PT. perf does most of the work in interpreting Intel PT's output, and magic-trace likely wouldn't exist were it not for their efforts. Thank you, perf developers.
magic-trace.org is a fork of Perfetto, with minor modifications. We'd like to thank the people at Google responsible for it. It's a high quality codebase that solves a hard problem well.
The ideas behind magic-trace are in no way unique. We've written down a list of prior art that has influenced its design.
Footnotes
perf can do this too, but that's not how most people use it. In fact, if you peek under the hood you'll see that magic-trace uses perf to drive Intel PT. ↩
Strictly speaking, anything newer than Broadwell, but this is not a platform we regularly test on, and timing resolution is worse (~1us). ↩
magic-trace
designs to code with AI
