Be Suspicious of Success

Haven't seen it mentioned here in the comments so I'll throw in — this is one of the best uses for code coverage tooling. When I'm trying to make sure something really works, I'll start with a failing testcase, get it passing, and then also use coverage to make sure that the testcase is actually exercising the logic I expect. I'll also use the coverage measured when running the entire suite to make sure that I'm hitting all the corner cases or edges that I thought I was hitting.

I never measure coverage percentage as a goal, I don't even bother turning it on in CI, but I do use it locally as part of my regular debugging and hardening workflow. Strongly recommend doing this if you haven't before.

I'm spoiled in that the golang+vscode integration works really well and can highlight executed code in my editor in a fast cycle; if you're using different tools, it might be harder to try out and benefit from it.

> It's not enough for a program to work, it has to work for the right reasons. Code working for the wrong reasons is code that's going to break when you least expect it.

This reminds me of the recent discussion of gettiers[1]. That article focused on Gettier bugs, but this passage discusses what you might call Gettier features.

Something that's gotten me before is Python's willingness to interpret a comma as a tuple. So instead of:

    my_event.set()

I wrote:

    my_event,set()

Which was syntactically correct, equivalent to:

    _ = (my_event, set())

The auto formatter does insert a space though, which helps. Maybe it could be made to transform it as I did above, that would make it screamingly obvious.

[1a] https://jsomers.net/blog/gettiers

[1b] https://news.ycombinator.com/item?id=41840390

> How do I know whether my tests are passing because they're properly testing correct code or because they're failing to test incorrect code?

One mechanism to verify that is by running a mutation testing [0] tool. They are available for many languages; mutmut [1] is a great example for Python.

[0] https://en.wikipedia.org/wiki/Mutation_testing

[1] https://mutmut.readthedocs.io

One general way I like to think about this is that most software you use has passed through some filter – it needed to be complete enough for people to use it, people needed to find it somehow (eg through marketing), etc. If you have some fixed amount of resources to spend on making that software, there is a point where investing more of them in reducing bugs harms one’s chances of passing the filter more than it helps. In particularly competitive markets you are likely to find that the most popular software is relatively buggy (because it won by spending more on marketing or features) and you are often more likely to be using that software (for eg interoperability reasons) too.

As the Dijkstrian expression goes, testing shows the presence of bugs, not their absence. Unit tests can show that a bug exists, but it cannot show you that there are no bugs, save for the particular cases tested and even then, only in a behaviorist sort of way (meaning, a your buggy code may still produce the expected output for tested cases). For that, you need to be able to prove your code possesses certain properties.

Type systems and various forms of static analysis are going to increasingly shape the future of software development, I think. Large software systems especially become practically impossible to work with and impossible to verify and test without types.

In my regressions, i make sure i include an "always fail" test, to make sure the test infrastructure is capable of correctly flagging it.

I love unit tests, but I sometimes additionally manually step through code in the debugger, looking for anything out of place. If a variable does anything surprising then I know I don't understand what I just wrote.

Colleagues: If the code works, it’s good!

Me: Hmmm.

Managers, a week later: We’re starting everyone on a 50% on-call rotation because there’s so many bugs that the business is on fire.

Anyway, now I get upset and ask them to define “works”, which… they haven’t been able to do yet.

> If the code still works even after the change, my model of the code is wrong and it was succeeding for the wrong reasons.

If someone else wrote the code, your model of why it works being wrong doesn't mean anything is wrong other than your understanding.

Sometimes even if you wrote something that works and your own model is wrong, you don't necessarily have to fix anything: just learn the real reason the code works, go "oh", and leave it. :) (Revise some documentation and write some tests based on the new understanding.)

There are few things that terrify me more nowadays at this point in my career than spending a lot of time writing something and setting it up, only to turn it on for the first time and it works without any issues.

Is this not common practice? I'd expect a good engineer who cares about their work to be just as suspicious if something works _when it shouldn't_ as when something doesn't work when it should. Both indicate a mismatch between your mental model and what it's actually doing.

The author is simply talking about the most common testing types[0] but in a more philosophical way.

[0] https://www.perfecto.io/resources/types-of-testing

I had a customer complain once about how great Akamai WAF was, because it never had false positives. (My company's WAF solution had many)

Is that actually desirable? This article articulates my exact gut feeling.

> This is why test-driven development gurus tell people to write a failing test first.

To be precise, it’s one of the big reasons, but it’s far from the only reason to write the test first.

Website not quite loading.. HN hug of death?

Nvidea ?

Site is buggy