October 15th, 2024

Try to fix it one level deeper

The author addressed a bug revealing systemic issues, improved error handling and code formatting, standardized parameters, uncovered dead code, and sought elegant solutions to ongoing coding challenges.

Read original articleLink Icon
AppreciationCuriosityInsight
Try to fix it one level deeper

The author reflects on a productive day focused on addressing a bug that revealed larger underlying issues. Initially, a colleague pointed out a substandard error message related to a database running out of disk space. Instead of merely fixing the message, the author recognized the need for a broader infrastructure to handle fatal errors, leading to the implementation of a solution. While reviewing a pull request (PR) that corrected typos, the author discovered that the formatting of Go code was not being enforced during continuous integration (CI), prompting a fix for that issue as well. Another PR highlighted a confusing log message due to similar compile-time parameters, which led to a deeper fix by standardizing the parameters and ultimately removing unnecessary complexity. The day’s work also included refactoring code that had been difficult to explain, which uncovered dead code that obscured logic. An attempt to add postcondition checks resulted in an unintended reentrancy issue, prompting further exploration for a more elegant solution. The author concludes with a sense of ongoing problem-solving and a desire to improve code quality.

- The author focused on fixing a bug that revealed larger systemic issues.

- Improvements were made to error handling and code formatting in CI.

- The author standardized compile-time parameters to reduce confusion.

- Refactoring efforts uncovered dead code that needed addressing.

- The day involved problem-solving and seeking elegant solutions to coding challenges.

AI: What people are saying
The comments reflect a deep engagement with the themes of debugging and software engineering practices.
  • Many commenters emphasize the importance of understanding the root causes of bugs rather than just fixing them.
  • There is a consensus on the need for effective error handling and improving engineering processes to prevent future issues.
  • Several comments highlight the balance between deep investigation and practical solutions in complex codebases.
  • Some contributors draw parallels between software debugging and broader problem-solving strategies in life.
  • There is a call for better methods to assess developers' abilities to diagnose and address underlying issues during the hiring process.
Link Icon 10 comments
By @niccl - about 12 hours
In the course of interviewing a bunch of developers, and employing a few of them, I've concluded that this ability/inclination/something to do this deeper digging is one of the things I prize most in a developer. They have to know when to go deep and when not to, though, and that's sometimes a hard balancing act.

I've never found a good way of screening for the ability, and more, for when not to go deep, because everyone will come up with some example if you ask, and it's not the sort of thing that I can see highlighting in a coding test (and _certainly_ not in a leet-code test!). If anyone has any suggestions on how to uncover it during the hiring process I'd be ecstatic!

By @andai - about 13 hours
I was reading about NASA's software engineering practices.

When they find a bug, they don't just fix the bug, they fix the engineering process that allowed the bug to occur in the first place.

By @brody_hamer - about 12 hours
I learned a similar mantra that I keep returning to: “there’s never just one problem.”

- How did this bug make it to production? Where’s the missing unit test? Code review?

- Could the error have been handled automatically? Or more gracefully?

By @raphlinus - about 8 hours
The title immediately brings to mind the Osterhout classic, "Always Measure One Level Deeper", [1], and I imagine was probably inspired by it. Also worth revisiting.

[1]: https://cacm.acm.org/research/always-measure-one-level-deepe...

By @Cpoll - about 11 hours
This kind of reminds me of https://en.m.wikipedia.org/wiki/Five_whys.
By @KaiserPro - about 3 hours
You need to choose your rabbit holes carefully.

In large and complex codebases, its often more pragmatic to build a guard in your local area against that bug, than following the bug all the way downthe stack.

Its not optimal, and doesn't make the system better as a whole. but its the only way to get things done.

That doesn't mean you should be silent though, you do need to contact the team that looks after that part of the system

By @Terr_ - about 8 hours
IMO it may be worth distinguishing between:

1. Diagnosing the "real causes" one level deeper

2. Implementing a "real fix" fix one level deeper

Sometimes they have huge overlap, but the first is much more consistently-desirable.

For example, it might be the most-practical fix is to add some "if this happens just retry" logic, but it would be beneficial to know--and leave a comment--that it occurs because of a race condition.

By @peter_d_sherman - about 8 hours
>"There’s a bug! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the bug is valuable, and it is pointing in the direction of a bigger related problem."

That is an absolutely stellar quote!

It's also more broadly applicable to life / problem solving / goal setting (if we replace the word 'bug' with 'problem' in the above quote):

"There’s a problem! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the problem is valuable, and it is pointing in the direction of a bigger related problem."

In other words, in life / problem solving / goal setting -- smaller problems can be really valuable, because they can be pointers/signs/omens/subcases/indicators of/to larger surrounding problems in larger surrounding contexts...

(Just like bugs can be, in Software Engineering!)

Now if only our political classes (on both sides!) could see the problems that they typically see as problems -- as effects not causes (because that's what they all are, effects), of as-of-yet unseen larger problems, of which those smaller problems are pointers to, "hints at", subcases of, "indicators of" (use whatever terminology you prefer...)

Phrased another way, in life/legislation/problem solving/Software Engineering -- you always have to nail down first causes -- otherwise you're always in "Effectsville"... :-)

You don't want to live in "Effectsville" -- because anything you change will be changed back to what it was previously in the shortest time possible, because everything is an effect in Effectsville! :-)

Legislating something that is seen that is the effect of another, greater, as-of-yet unseen problem -- will not fix the seen problem!

Finally, all problems are always valuable -- but if and only if their surrounding context is properly perceived...

So, an an excellent observation by the author, in the context of Software Engineering!