July 21st, 2024

rr – record and replay debugger for C/C++

RR is a C/C++ debugging tool for Linux, enhancing gdb by enabling deterministic recording and replay of failures, supporting various applications like Firefox, with low overhead and a chaos mode.

Read original articleLink Icon
DebuggingExperiencesQuestions
rr – record and replay debugger for C/C++

The tool RR aims to be a primary C/C++ debugging tool for Linux, enhancing gdb by allowing users to record failures once and then debug the recording deterministically as many times as needed. It offers efficient reverse execution under gdb, supports various applications, including Firefox and Chrome, and enables debugging of multiple-process workloads. RR features low overhead, durable traces, and a chaos mode to reproduce intermittent bugs. Developers can use RR to record, replay, and debug applications efficiently. The tool's deterministic replay preserves execution details, making debugging easier and more effective. RR's design goals include simplicity, deployability on stock Linux kernels, and low runtime overload. While RR has limitations like emulating a single-core machine and requiring knowledge of system calls, it provides a valuable solution for debugging complex applications like Firefox. Overall, RR lowers the cost of fixing bugs, improves software quality, and enhances the debugging experience for developers.

Related

Show HN: R2R V2 – A open source RAG engine with prod features

Show HN: R2R V2 – A open source RAG engine with prod features

The R2R GitHub repository offers an open-source RAG answer engine for scalable systems, featuring multimodal support, hybrid search, and a RESTful API. It includes installation guides, a dashboard, and community support. Developers benefit from configurable functionalities and resources for integration. Full documentation is available on the repository for exploration and contribution.

Show HN: Xcapture-BPF – like Linux top, but with Xray vision

Show HN: Xcapture-BPF – like Linux top, but with Xray vision

0x.tools simplifies Linux application performance analysis without requiring upgrades or heavy frameworks. It offers thread monitoring, CPU usage tracking, system call analysis, and kernel wait location identification. The xcapture-bpf tool enhances performance data visualization through eBPF. Installation guides are available for RHEL 8.1 and Ubuntu 24.04.

Boosting Compiler Testing by Injecting Real-World Code

Boosting Compiler Testing by Injecting Real-World Code

The research introduces a method to enhance compiler testing by using real-world code snippets to create diverse test programs. The approach, implemented in the Creal tool, identified and reported 132 bugs in GCC and LLVM, contributing to compiler testing practices.

Below: A time travelling resource monitoring tool

Below: A time travelling resource monitoring tool

Facebook's resource control team introduces "below," a new Linux system monitoring tool addressing existing tool deficiencies. It offers user-friendly features, historical data recording, process navigation, and compatibility across kernels.

Rye Language

Rye Language

Rye Language is a high-level, dynamic programming language emphasizing interactive use, backend development, and information processing. It supports various functionalities, runs on multiple platforms, and promotes user-friendly information manipulation with strict design principles.

AI: What people are saying
The comments on the article about the RR debugging tool for Linux highlight various experiences and insights:
  • Users find RR invaluable for debugging complex C/C++ issues, especially with features like deterministic replay and reverse execution.
  • There are discussions about RR's compatibility with other languages and environments, including Rust, OpenGL, Vulkan, and even Android.
  • Some users mention alternative tools and methods, such as GDB's built-in reverse debugging and WinDbg for Windows.
  • RR's limitations are noted, particularly with concurrency bugs and specific hardware like Ryzen CPUs.
  • There is interest in integrating RR more seamlessly into development workflows and exploring its potential for kernel debugging.
Link Icon 18 comments
By @whatsakandr - 4 months
I've used rr very sucessfully for reverse engineering a large code base using a break on variable change combined with reverse-continue. Took the time to extract critical logic way down.
By @suby - 4 months
Perhaps worth mentioning is that someone attempted to port this to Rust and got about 60,000 lines of code into it before archiving the project. I feel like comparing these two efforts would be an interesting case study on the impacts / benefits / limitations or difficulties, etc involved in rewriting from C++ to Rust.

https://github.com/sidkshatriya/rd/

By @laserbeam - 4 months
Is it truly only for C/C++?

My limited understanding says a debugger needs: a list of symbols (.pdb files on windows, can't remember what they are on linux), understanding of syscalls and a few other similar things. I thought they don't care too much what generated the binaries they are debugging (obviously as long as it's native code).

Doesn't rr work with other languages like rust, zig, odin, nim, and similar ones? Obviously, I wouldn't expect it to work for python, js, c# and other languages with managed memory.

By @rtpg - 4 months
rr is really cool, but almost every time I have decided to pull it out as one of the "big guns" it turns out that I have a concurrency bug and so rr is unable to reproduce it.

Despite that, it would be very, very, very cool if some languages built rr directly into their tooling. Obviously you can always "just" use rr/gdb, but imagine if rr invocations were as easy to set up and do as pdb is in Python!

By @modeless - 4 months
See also https://pernos.co/ which is based on rr but adds a queryable database of the whole program execution, which allows you to do things like this:

> [...] just click on the incorrect value. With full program history Pernosco can immediately explain where this value came from. The value is tracked backwards through events such as memcpys or moves in and out of registers until we reach a point where the value "originated" from, and each step in this process is displayed in the "Dataflow" panel that opens automatically. There is no need to read or understand the code and think about what might have happened, we can simply ask the debugger what did happen.

https://pernos.co/examples/race-condition

By @ho_schi - 4 months
GDBs built-in reverse debugging: https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial

I assume rr provides more features and flexibility. Anyway I want to mention that GDB itself can already reverse debug for some time now.

By @forrestthewoods - 4 months
On Windows you can use WinDbg for the same thing. It has better support for debugging multi-threaded issues.

https://www.forrestthewoods.com/blog/windbg-time-travelling-...

By @dmitrygr - 4 months
Long ago, VMWare workstation supported doing this, but not just for userspace programs but also for kernels and even drivers, in a VM. The feature shipped and existed for a few versions before it was killed by internal politics.
By @leoc - 4 months
Are rr’s problems with Ryzen CPUs now firmly in the past or not?
By @jfk13 - 4 months
By @teaearlgraycold - 4 months
I used this to help make my toy JIT compiler: https://github.com/danthedaniel/BF-JIT

Super useful, especially considering I know barely anything about x86-64.

By @MaskRay - 4 months
I almost use rr every day, along with a gdb frontend: cgdb.

rr record /tmp/Debug/bin/llvm-mc a.s && rr replay -d cgdb

I've have success story with some bugs only reproducible with LTO. Without rr it would be a significant challenge.

It would be nice if Linux kernel could be debugged with rr. Does anyone have success with kernel under rr+qemu ? :)

By @alfiedotwtf - 4 months
[off-topic] does anyone here who regularly uses a debugger (even just breakpoints and watchers in their IDE) use it for async execution? I've never tried, but I'm just trying to think how all that jumping around the executor and any runtime would work (if at all).
By @iamcreasy - 4 months
Is it possible to use this with C/C++ code compiled to dll/so and called by Python?
By @boguscoder - 4 months
Curious if anyone tried rr for/on Android? It seems possible to crosscompile it and it could be a good tool for native side debugging
By @whatsakandr - 4 months
Has anyone gotten rr to work with opengl or vulkan? It seems to always crash for me after making an opengl call.
By @throwaway2037 - 4 months
Can someone explain how it works?