Moxie Marlinspike: Agile is killing software innovation

I agree with Moxie. But I’d go even further: I think the real problem stems from the modern corporate structure.

We have this idea in modern management theory that responsibility and decision making must travel up the corporate hierarchy. That execs are best positioned to make product level decisions. Or decisions of any kind. And peons at the bottom - the “individual contributors” who do all the programming must know the least about the product.

But that’s completely backwards. The people on the ground in any organisation know the most about what’s really going on. If we disempower them, and try to turn software engineering into an assembly line process (jira tickets go in, code comes out) then of course programmers stop innovating. They’re explicitly, and intentionally distanced from their capacity to improve the product in creative ways.

To be clear, I also don’t think the answer is to pretend we’re all equal and have an entirely flat management hierarchy. Different people have different skills, and a healthy organisation needs lots of talents to thrive. But there’s lots of ways to do that which don’t disempower the people on the ground.

The book Reinventing Organisations is a gem of a book. It totally blew my mind on this stuff. It talks about all sorts of different ways that innovative companies have upended their corporate structures, and along the way have been able to get the best from their people. If / when I start a company, I’m tempted to make it required reading.

The industry took the good ideas of Agile (don't overplan, work in small steps, reassess regularly, oh and write tests and do code reviews) and absorbed them into normal software engineering. So now whenever someone describes themselves as an "agile" programmer it suggests a strict devotion to stand-up meetings, kanban boards, and various other cultic rituals.

But I'm not convinced that Agile is really to blame for the compartmentalization of knowledge and down-skilling of software engineering, as it has happened in every other industry too. It seems like a tendency of mass production. Car companies may be founded by small teams of master generalist mechanics, but they end up employing hundreds of thousands of ordinary technicians who know only what they need to know. Same for furniture factories and carpenters. Why not for software too?

Agile, scrum, and OKRs have always bothered me (I know, how profound!)

My issue with them, though, is one that I don't usually see spelled out very often. All three promise, whether directly or implied, freedoms and responsibility being pushed further down the org chart. In reality all three are implemented in a way that further centralizes control towards the top of the org, leaving independent contributors at the bottom feeling almost entirely powerless.

I've always wanted to see a company flip OKRs on their head. Every employee should define key results for their little corner of the company. Managers should be responsible for taking those as fact and working to build a direction for the team that aligns as many of those results as possible. By the time you get to the top you have leadership setting the company's direction based on what the employees know to be most important rather than leadership seeming to energy on a hilltop to force key results down on everyone else.

Bottom up, not top down. Hire well, train well, and trust your people. It could fail spectacularly but at least your people would be respected and enjoy the ride, and we'd all learn something about the pros and cons of running a company without a dictatorship.

We had a backlog grooming meeting yesterday. The boss asked me to estimate a task that was a bug fix for some code I am completely unfamiliar with. If I had a dollar for every time that has happened I think I could buy a used jet ski in good condition. So I did the usual thing. I told her that I didn't know how long it would take. Then she said ok, then this is a task to estimate the size of the bug fix task, how long will that take. Rather than continue to struggle I said "I'm about to lie to you" and then pulled a (Fibonacci) number out of my butt. That was enough to make her happy and we moved on. I'll give it my best effort and there will never be any accountability for my stinky estimate.

This has been my general experience with agile as practiced at three different outfits.

I think this touches on an interesting problem. I don't see a reason to praise security experts though.

Here's a theory on why (some form of) Agile is problematic: Subdividing work into smaller parts is typically beneficial. However, in computer programming, this approach often fails because even the smaller tasks require some creativity, and unexpected challenges are likely to arise.

The person who did the subdividing gains a lot of insight while breaking down the larger problem. However, when transferring a portion of the work to a developer, much of this knowledge is inherently lost. The developer must then devise a creative solution, and lacking the necessary information, may either come up with a suboptimal solution or need further communication with the original architect.

There is no clear-cut solution. Some might argue for more experienced developers who have all the necessary knowledge readily available in their heads. Others might advocate for better design diagrams and documentation to capture all the relevant information. Ultimately, it requires careful consideration, or luck, to strike the right balance. But the dogmas of Agile certainly aren't helping much.

Software has gotten so bad over the past couple of decades. It's not as responsive as it was even in the 80-90s, yet we have orders of magnitude more powerful machines. Even simple website forms are often broken, due to everything from poor implementations to bit rot. It's perhaps just a coincidence that this time period has also been the rise of agile.

Had to do it at a company and thank god we fought against daily standups, but the constant retrospectives, the useless breaking down of tasks into subtasks and ranking how long it’ll take just for the benefit of the non technical manager and nobody else… I admit it was so bad I never wanted to work for a company after that.

Just let an engineer “own” a section of the code. That was the era of my own career when I felt that the software our team worked on was at its best.

My impression is larger or more synthetic orgs don't want devs to lead things anymore, they want a very top down development of vision -> product -> ux -> project management, and then the developers scurry around and do whatever it is they do using cloud something something, hopefully in as commodity a way as possible. The article clearly states the problems with this; devs can no longer understand the whole picture or make important top level suggestions, and the cloud dictates what can be done.

> the Signal founder stole the show with an opening chat laying out a case for reclaiming the "magic" of software development that's been lost after 20 years

How to tell you’ve been in an industry for more than 20 years.

Spend time around excited young programmers, and you’ll find the magic is still there. It’s just different than it was 20 years ago. I remember the people with 20 years of experience back then telling me how awful everything was, but I was having way too much fun making things to give them much attention.

The problem is not Agile, the problem is that people have no clue what Agile is. Here's a refresher:

Value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Principles:

1. Customer satisfaction by early and continuous delivery of valuable software.

2. Welcome changing requirements, even in late development.

3. Deliver working software frequently (weeks rather than months).

4. Close, daily cooperation between business people and developers.

5. Projects are built around motivated individuals, who should be trusted.

6. Face-to-face conversation is the best form of communication (co-location).

7. Working software is the primary measure of progress.

8. Sustainable development, able to maintain a constant pace.

9. Continuous attention to technical excellence and good design.

10. Simplicity—the art of maximizing the amount of work not done—is essential.

11. Best architectures, requirements, and designs emerge from self-organizing teams.

12. Regularly, the team reflects on how to become more effective, and adjusts accordingly.

https://en.m.wikipedia.org/wiki/Agile_software_development

For most of the critics, I would say "become more Agile" as a solution.

The people who originally created the agile principles devised it for consulting agencies, not product development. A lot of the ideas are relevant still, but there's a bit of chaff to sort through to get at the wheat.

Not to mention the now capital-A Agile practices, courses, certifications(!!) and processes (!!?!?) which end up being exactly against the spirit of the original manifesto.

This meme that "Agile is evil", needs to die. No, people, you've just had really bad managers who don't know how to manage software engineering teams.

SCRUM isn't agile, it's SCRUM. Kanban isn't agile, it's Kanban. Doing something that has lots of process but that isn't waterfall isn't agile just because it isn't waterfall - it's a thing with lots of process.

Agile is about responding to the context of the team and the customer. You probably need a way to keep track of what needs to be done, and have regular updates about blockers and progress, but that's it.

The problem is if you ask people who have never been software engineers to manage a software engineering project, they're going to struggle to communicate, because they have no natural intuition of what the hell is going on. The project management industry is quite happy to serve these people by inventing big complex processes to keep them busy and sell them books, courses, certifications and conferences.

Most of big tech won't allow someone to manage an engineering team or project manage those projects without a good amount of hands-on engineering experience. The rest of industry needs to pick up on this.

People without that experience can learn. But they should adapt to software being what software is, not try and bring processes derived from those invented for other domains to software.

Agile done well is absolutely amazing, when its in the hands of teams who have all built software, understand their customer problem and the milestones they need to hit (yes, deadlines are unavoidable in any modern business environment, sorry), and each other.

Trust me, I remember when waterfall was the way. Waterfall is so bad, that even when you do 2-week long waterfall sprints (waves at SCRUM), it feels awful.

Go back to the Agile Manifesto [1]. It's all you need. Keep track of what was decided or discovered (bugs, features that are needed, and so on). If you have regular bottlenecks due to skill/knowledge imbalances, steal pairing from XP. Maybe explore Theory of Constraints but resist Kanban unless you understand exactly what it is doing and why. Have regular conversations as a group, and one-on-ones, not just between managers and reports, but between each other. Hold short but regular retros to understand systemic issues and then make sure the actions are actioned.

It's not that hard. It's just Agile.

[1] http://agilemanifesto.org

Can't spell "fragile" without "agile"!

Great article. If only we could make the people holding the bags of money understand...

The same management and teams that turn agile into the hellscape many folks experience, would make waterfall or whatever the same mess. I've seen pre and post agile, it's all due to attitude, skills, and experience.

The big issue to me seems to be this idea that software should do everything under the sun, instead of doing one thing, and doing it well.

The smaller software is, the less it needs to be designed by committee.

I get that this isn't ALWAYS doable. And that when you're designing things for end users that are allergic to reading any documentation, in-line or otherwise, it's an uphill battle. But it's an ideal to strive toward where possible to avoid monstrosities like systemd and pulseaudio.

Some of the best code is the code written by one person.

Agile isn't killing inovation. Corporations are. Agile is just a tool to do so.

Want to stay creative as a programmer, be a lone wolf... Do what you like and whenever you like, follow the creative spark...

You know what kills software progress and bug fixing?

Closing and locking thousands of Signal issues automatically and locking all conversations for "spring cleaning"...

... which feels exactly like something somebody oversubscribed to agile methods in the wrong way would do.

Where can I view their entire talk?

He's right.

My take on this is that the developers are costly. A working hour of single developer costs a lot of money. The organization will always try to get the maximum out of that one hour. Spending more money so that the developers are 'better' is not efficient. They just need to be good enough.

Seems unlikely. A more plausible countertheory is that software is a very young industry and we're leaving the era where there were a lot of easy unknowns. The industry as a whole has figured out social networks, got a pretty good bead on the economics of software (especially open source). Databases are well developed, networking is well developed, new basic algorithms seem to have stopped appearing with regularity.

We might not have killed software engineering as much as we are exhausting the list of things that very smart programmers will just figure out on their own.