How to build highly-debuggable C++ binaries
David Hashe's article offers guidance on configuring C++ toolchains for better debuggability, emphasizing practices like enabling sanitizers, using debug modes, and balancing performance with debuggability in large projects.
Read original article
The article by David Hashe provides actionable advice on configuring C++ toolchains to produce highly-debuggable binaries. It highlights the complexities of C++ compilation and the lack of standard build tools, which can complicate debugging. Hashe emphasizes the importance of understanding the compilation model and recommends using the project's preferred dependency management system. He discusses the merits of interactive debugging versus printf-style debugging, suggesting that interactive debugging is generally more beneficial for large legacy projects.
To create highly-debuggable binaries, Hashe outlines several key practices: enabling sanitizers, using debug modes for standard libraries, generating debugging information for macros, and compiling with frame pointers. He also advises enabling asynchronous unwind tables for precise stack unwinding and ensuring that static libraries are fully linked. The article categorizes the advice into general compilation changes, semi-specific changes, and specific source code modifications.
Hashe stresses the need to balance debuggability and performance, suggesting that developers should decide in advance which parts of the binary will be debuggable based on the specific bug being addressed. He concludes that while C++ debugging cannot match the ease of scripting languages, targeted changes and custom debugger extensions can significantly enhance the debugging experience. The guide is tailored for g++ and clang++ on x86_64 GNU/Linux with gdb, although some advice may apply to other platforms.
Related
Mix-testing: revealing a new class of compiler bugs
A new "mix testing" approach uncovers compiler bugs by compiling test fragments with different compilers. Examples show issues in x86 and Arm architectures, emphasizing the importance of maintaining instruction ordering. Luke Geeson developed a tool to explore compiler combinations, identifying bugs and highlighting the need for clearer guidelines.
An insect is sitting in your compiler and doesn't want to leave for 13 years
An insect in a compiler caused encoding issues for 13 years, leading to build failures and ODR violations. Despite bug reports and patches, the problem persists, highlighting the need for timely fixes.
Hash-Based Bisect Debugging in Compilers and Runtimes
Hash-Based Bisect Debugging uses binary search to locate code issues efficiently. It applies binary search to debug by bisecting data or version history, aiding in pinpointing bugs in code changes or optimizations.
Driving Compilers
The article outlines the author's journey learning C and C++, focusing on the compilation process often overlooked in programming literature. It introduces a series to clarify executable creation in a Linux environment.
Better Firmware with LLVM/Clang
LLVM and Clang are gaining traction in embedded software development, particularly for ARM Cortex-M devices. The article advocates integrating Clang for better static analysis, error detection, and dual compiler usage.
- Many commenters emphasize the importance of enabling frame pointers and the benefits they bring to debugging.
- There is a discussion about the use of sanitizers like ASAN in production, weighing their advantages against potential security risks.
- Several users share personal experiences and techniques for improving stack trace generation and debugging efficiency.
- Comments highlight the challenges of debugging with modern C++ features, particularly with template-heavy libraries.
- There is a consensus on the need for better tools and practices to enhance debuggability in C++ projects.
17 comments if (breakpoint_1 && (x_id == 153827)) {
__asm("int $3");
}
__asm(“int3\n\tnop”);
Setting the build for an old x64 machine (https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) for reversible / time travel debuggers seems unnecessarily restrictive to me. I'd expect a modern time travel debug tool (e.g. either rr or Undo - disclaimer, which I work on) to cope fine with most modern instructions (I believe GDB's built-in record / replay debugging tends to be further behind the curve on new CPU instructions - but if you're doing anything at scale it's not the right choice anyhow).
Regarding compilation (https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) - we generally advise customers to use -Og rather than -O0. As the article states, this will still optimise out some code but should be a good trade-off without being too slow. (NB. last I checked, clang currently uses -Og as an alias for -O1, so it may behave less satisfactorily than under GCC).
It's also not said enough but: you don't need a special debug build to be able to debug. It's less user-friendly to debug a fully-optimised release build but it's totally possible. You just need to retain the DWARF debug info (instead of throwing it away). This is really important to know if you're debugging on a customer system or analysing a bug that's only in release builds.
#define STACKTRACE GLOBAL_STACK_FILE[GLOBAL_STACK_IDX] = __FILE__; GLOBAL_STACK_LINE[GLOBAL_STACK_IDX++] = __LINE__; StackTraceCleaner stc;
So at any point in time I could inspect GLOBAL_STACK_FILE and GLOBAL_STACK_LINE and have a complete stack trace of the game.
Obviously this only worked because these games weren't performance-critical and because they were essentially single-threaded, but it did the job at the time. We're talking about a time when Visual Studio 6's support for templates was half-broken, and the STL wasn't exactly S, to the point that I had to roll out my own string, smart pointers, containers, etc -- made twice as hard because of the aforementioned broken template support in VS6 :(
I do miss these simpler, more innocent times, though.
Not a silver bullet but still, being able to collect and analyze user-mode crash dumps is sometimes the best way to investigate and fix bugs.
Thanks a lot!! I don't know how many times I've stepped into the C++ standard library and it gets really annoying..
Unlike the article we are using lldb rather than gdb ... and while I appreciate thats its possible _at all_ to script the debugger to do some pretty printing -- I found it quite a bit more frought to implement than initially expected ...
To take the Eigen example ... Eigen is a 'header only' library and offers Templated vector and matrix types. The types are template over (optionally), data type, number of rows, number of columns, and matrix row order (row major or column major). All that information is not actually even available at runtime -- just the type name (with the instantiated values for template arguments) ...
I ended up having to super hackily parse information out of the template type name in order to be able to pretty print the matrix appropriately in lldb ...
Problems of this nature abound when debugging c++ ... Very often with a header only library, there isn't even a symbol for methods you might want to call -- so you want to eg, call the size() method on some object within the debugger to see how big it is, you'll often be out of look due to an undefined symbol reference since the 0-overahead compilation models ensures the symbol doesn't even have to be created in the binary ...
Would be nice if there was some kind of way around that -- I guess I need to try the workaround mentioned in the article of explicitly instantiating template classes for common classes in 'debug' mode ... My fuzzy mental model derived from previous experience somehow doesn't think that will actually help the issue tho -- but I'd be happy to be wrong!
A nice trick with MSVC is you can turn off optimizations for TU or any block of code with:
#pragma optimize( "", off )
Once upon a time it was widely said that ASAN should not be used for production code. The authors advocated against it and from a general-purpose security perspective it gives attackers a very large writable memory region at a fixed offset to play with. But over time I see more and more ASAN code in production on the theory that ASAN may make a system easier to exploit, but a memory corruption will make it easier to exploit. And so it's better to have knowledge of the issue.
Also, I've personally found the glibc malloc tunables very useful for debugging.
If your call graph has more roots than your neighbors garden and the whole thing is a forest and not a tree you will have a hard time understanding, analyzing and ultimately debugging.
>Compile with frame-pointers.
It's good to see that enabling frame pointers are included in the recommendations for debugging purposes.
The discussions on the relevance and the usefulness of frame pointers earlier this year on HN [1]:
[1] The return of the frame pointers:
> If using libc++:
> Add this define to your CXXFLAGS: -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_DEBUG
... until the bikeshedding bastards change the define yet again in the next release.
Related
Mix-testing: revealing a new class of compiler bugs
A new "mix testing" approach uncovers compiler bugs by compiling test fragments with different compilers. Examples show issues in x86 and Arm architectures, emphasizing the importance of maintaining instruction ordering. Luke Geeson developed a tool to explore compiler combinations, identifying bugs and highlighting the need for clearer guidelines.
An insect is sitting in your compiler and doesn't want to leave for 13 years
An insect in a compiler caused encoding issues for 13 years, leading to build failures and ODR violations. Despite bug reports and patches, the problem persists, highlighting the need for timely fixes.
Hash-Based Bisect Debugging in Compilers and Runtimes
Hash-Based Bisect Debugging uses binary search to locate code issues efficiently. It applies binary search to debug by bisecting data or version history, aiding in pinpointing bugs in code changes or optimizations.
Driving Compilers
The article outlines the author's journey learning C and C++, focusing on the compilation process often overlooked in programming literature. It introduces a series to clarify executable creation in a Linux environment.
Better Firmware with LLVM/Clang
LLVM and Clang are gaining traction in embedded software development, particularly for ARM Cortex-M devices. The article advocates integrating Clang for better static analysis, error detection, and dual compiler usage.