Jeremy Rowley resigns from DigiCert due to mass-revocation incident

I'm with amir in comment 23 and with Aaron in previous comments. Stuff happens. And when there are multiple moving pieces, the process and policies are the issue, not the individuals. Since individuals rarely have a complete overview of the entire system.

Also, as noted in the comments, it sets a bad precedent for people coming forward reporting issues.

I think this comment from the thread sums it up:

“When DigiCert has another incident (and while I have tremendous faith in Tim, it will happen), I would rather that they have Jeremy Rowley with his wisdom and scar tissue around to guide their response and subsequent improvement.”

> “The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems.”

This could happen to anyone, but imagine being the developer or development team that made this mistake.

For those of you who don't know who he is, he was the Chief Information Security Officer

https://www.digicert.com/blog/author/jeremy-rowley

This should probably be a direct link to the comment announcing the resignation: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17

> The ultimate root cause ended up being me. I have led the compliance team for the past several years. The fact this went unnoticed in our many reviews during that time shows that we need a different approach to both our internal investigations and compliance controls. I also dropped the ball on the certificate problem report by failing to escalate the issue to engineering and give it the proper attention it deserved. Although I did some investigation, I failed to treat the allegations with sufficient seriousness based on what could have been wrong. I assumed I knew the systems and what was happening in them rather than deeply investigating the report. Finally, I didn’t do enough to eliminate the silos between compliance and engineering.

Really does sound like he personally dropped the ball in the handling of the report. It would be interesting to hear the story from the researcher who will undoubtably have been frustrated beyond reason that they kept acting like there was no issue despite the repeated persistent attempts at getting them to take it serious.

I question why we accept someone resigning after making a big mistake.

Unless it's malice, or the fault truly is entirely on that person, what good would resigning do?

Rowley admitted he fucked up, badly, he admitted on several layers what must be changed. How he must change. How the org must change. How the way things are presently is not good enough. Made an extremely deep dive into what happened.

And now he's leaving??? Someone who royalty messes up, would not want to mess up on the same issue twice. So all that experience is now worthless and doesn't benefit Digicert in the slightest.

> We note that other customers have also initiated legal action against us to block revocation.

This seems crazy to me. In what world does suing your business partner make more sense than clicking some buttons in a UI or running some shell commands to renew your cert?

I imagine someone as articulate and humble as that guy is going to land on his feet. That was a really good write up.

One thing I find odd about this: the rules for CAs are long and detailed, but they don’t seem especially complicated. If I were implementing a CA, I would have the main code (their “service oriented architecture” or a monolith or whatever) produce not just an instruction to issue a certificate but a transcript of the entire exchange. Then a completely separate code path (plain old synchronous Rust or Python or Go or Haskell or ML — no microservices) would check the transcript for compliance with each clause of the requirements and block issuance if anything fails. And raise an alert that gets noticed.

One could even get fancy and use verifiable randomness for everything in the protocol that is supposed to be random.

And then one could refactor some other code with much less worry about messing up.

This might also reduce the blast radius from a bug in some other component. If the magic random string generator can be coerced into returning ‘www’, then a separate check would prevent this from compromising everything.

(I work in a different industry, and in my industry there is plenty of complex, evolving code, that needs to do the right thing. The more competent players have separate verification code as a double-check.)

> We note that other customers have also initiated legal action against us to block revocation.

How can it make economic sense to initiate a lawsuit rather than just get new certificates?

I have low expectations from C-level executives. But this incident and his response to it has changed my perspective of them just slightly.

It's a rare incident where a C-level executive actually takes accountability for their fuck up. Shit rolls down hill. He is very likely to end up taking the helm at another place or startup on his own. He is the exact opposite of the CrowdStrike CEO (George Kurtz) that caused an absolute shitstorm compared to DigiCert incident.

> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.

Interesting. What is the value of a microservice that generates random numbers over just using a language's SecureRandom equivalent?

He resigned with honor, grace and responsibility – and should be applauded.

This is what real accountability looks like, and doing so not only preserves the reputation and trustworthiness of his employer, but demonstrates that he is a valuable contributor and trustworthy individual. He will land on his feet as a result.

I don’t quite follow why a missing underscore results in a security problem. It seems like it must be somehow related to what’s valid for CNAME records?

Given that a revocation is simply a publication of additional data by a CA, and does not directly affect the customer’s systems, how is the TRO in this case not unconstitutional? I’m not a lawyer but it feels like prior restraint, no?

While I applaud his openness and willingness to take accountability, I agree with others that resigning shouldn't be necessary.

Resigning is what you do when you are clearly not fit for your post. Jeremy has demonstrated that he is anything but unfit. People that can see where things went wrong, who can communicate such, can come up with changes to fix those issues, and can implement them are exactly what is needed at such a high level of management. Most people would bury the story or claim ignorance, but Jeremy doesn't hide anything and takes full responsibility.

I wish Jeremy could have stayed and used this honesty and insight to make the necessary changes. Firing a C-level executive when things go wrong doesn't fix anything any more than finding a low level engineer to blame and fire. Experienced people learn lessons by making mistakes. It sucks that it happens, but unexpected circumstances can't be foreseen. Hindsight is 20/20. Now that they know, they know to look out for it and to change the system to prevent it next time.

Perhaps he did overlook it. Perhaps he didn't respond when he should have. It's easy to get complacent. This is a wake up call. I have no doubt that he would be much more attentive and responsive as a result of this, and as such, be exactly what's needed for his post.

Mistakes don't call for sacrifices; they call for systematic changes to prevent making the same mistakes again.

Thank you Jeremy for being as forthcoming as you have been. I only wish more C-level execs would do the same. I hope you find a good place to land where you can take this experience and do an even better job. And I hope that whoever replaces you can bring the same rigor and professionalism that your brought.

Sounds like inside baseball