Debugging an evil Go runtime bug: From heat guns to kernel compiler flags

> Over the course of 22 kernel builds, I managed to simplify the config so much that the kernel had no networking support, no filesystems, no block device core, and didn’t even support PCI (still works fine on a VM though!).

Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C/C++ experience but I'd helped out with Bitbake recipes and everyone else was busy or moved to other projects.

To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a hypervisor misconfiguration, overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered.

A month of frustration, two characters to fix, the highest ratio I've encountered so far.

My reward for struggling through a complex problem I was unqualified for? "Great, now we need to backport security patches from the main Linux kernel to the SoC vendor's custom fork..."

(2017) Previous discussion: https://news.ycombinator.com/item?id=15845118

This is honestly wild. 99% of devs would have found a work around and moved on. Going so far as to create a multi-kernel test bench to narrow down the source of the instability is a level of dedication I have not personally seen, and I respect it.

Related (for the hash based bisecting): https://research.swtch.com/bisect

You can follow Hector Martin @marcan at https://social.treehouse.systems/@marcan/

He works on Asahi Linux, a Linux port to arm64 Apple hardware.

I'm pretty confident in my computer abilities, but when I read stuff like this I feel like I have no skills at all compared to this guy. Like I'm still a high school athlete and he's an Olympian (also he was only 26 when he wrote the article).

This is very elegant. I’ve had my share of nasty system bugs ( compilers and kernels ) , but the dedication and the speed with which he went through it is quite remarkable.

The explanations are also very clear. Thanks for posting.

One thing I learned from that post back then is that you can instruct Grub to ignore some part of your physical memory. Really nice trick, not sure this is doable on Windows / Mac?

I feel like I still didnt fully understand what's going on here. Is the following correct? "Threads hava a 'canonical' stack that the OS auto-grows for you as you use more of it. But you can also create your own stack by putting any value you want in RSP. This is what the Go program did, and the vDSO, assuming it ran on an auto-growing stack, tried to probe it, which lead to corruption."

I guess Gentoo Hardened meant the difficulty level in this case.

Could someone explain why Google is obsessed with small stack sizes? The musl library also has an extremely small thread stack size, which makes many applications crash that are used to 8192K on Linux.

On Linux the 8192K aren't reserved unless the are actually used, so what is the point?

Ok, Golang will allocate green threads from its own allocator, but 104 bytes?!