When did estimates turn into deadlines?

So when those times have occurred I've (we've more accurately) adopted what I refer to the "deer in the headlights" response to just about anything non-trivial. "Hoo boy, that could be doozy. I think someone on the team needs to take an hour or so and figure out what this is really going to take." Then you'll get asked to "ballpark it" because that's what managers do, and they get a number that makes them rise up in their chair, and yes, that is the number they remember. And then you do your hour of due diligence, and try your best not to actually give any other number than the ballpark at any time, and then you get it done "ahead of time" and look good.

Now, I've had good managers who totally didn't need this strategy, and I loved 'em to death. But for the other numbnuts who can't be bothered to learn their career skills, they get the whites of my eyes.

Also, just made meetings a lot more fun.

Conversely, if you're trying to launch a space probe and the planets are no longer in the right positions for the required gravity assist, your spacecraft will not get where it needs to go. Or if you're a little $100M/yr toolmaker, and Ford asks you for a die for the 2026 F150 production line, to be delivered by March, and the contract states you owe a $20,000 per MINUTE penalty if you're late...you don't wait until February to say something surprising happened and it's not going to be ready. You don't sign on that dotted line unless you know for certain that you can do it.

Ford or NASA won't bat an eye when you tell them that a quote is going to cost $XX,XXX. They won't be surprised when they give you an ECO and you say that it's going to take 3 weeks and $8,000 to deliver a part that everyone knows you can probably make by hand in 30 minutes, they know that you're hedging against those deadlines, and pricing in the acceptance phase and inspection phase and contingency plans and everything else that makes their deadline-heavy industry function.

But if you tell someone at OP's modernization group that due to incomplete information you think that the 30-minute task to change the text of that button will take "no more than 3 weeks and $8,000" they'll laugh you out the door. Optimistic estimates get rewarded, pessimistic estimates get discouraged, accurate estimates are irrelevant, and in the end you're constantly behind schedule and no one's really surprised.

Due to small side projects like the painting of the Sistine Chapel ceiling it took around 40.

Failure to meet the deadline informed by the estimate meant that the scale of the project was massively reduced because: Pope Julius II had died prior to completion, there were changes requested by the customer (both Julius and his heirs), supply chain issues, contract renegotiations, labor disputes, shortages of qualified workers, and money running out due to the long duration of the project.

So, since 1505 at least?

The funny thing is that the pope isn't even interred there.

Unfortunately it’s often true. People keep saying: "but didn’t you initially say X?"

"Sure I did, but I have new knowledge" won't always work.

A nasty side-effect is that people who are aware of this shy away from giving you numbers.

That happens all the time with insurance. I'm surprised at the confident tone in "reality does not operate like this". Not just car/home insurance either...health insurance also. They do often negotiate to a reasonable place, but not always.

You should not provide an estimate for "feature X implemented", but rather for "feature X engine". If you discover additional work to be done, then you need to add "existing code refactor", "feature X+Y integration", etc. as discovered milestones.

Unfortunately, you need that nomenclature and understanding to go up the chain for this to work. If someone turns your "feature X engine" milestone into "feature X complete" with the same estimate, you are screwed.

------

There is a related problem that I've seen in my career: leadership thinks that deadlines are "motivating".

These are the same people that want to heat their home to a temperature of 72F, but set the thermostat to 80F "so it will do it faster".

I was once in a leadership meeting, where the other participants forgot that I, lowly engineer, was invited to this meeting. Someone asked if we should accept that deadline X was very unlikely to be met, and substitute a more realistic deadline. To which the senior PM responded that "we never move deadlines! Engineering will just take any time given to them!"

Engineering, in that case, gave the time back when I left that team.

There is often understandable resistance to this at first. To address concerns, I find it helpful to share with stakeholders that a reasonably accurate estimate, one which could be legitimately used in planning concerns, is really only possible in one of two situations:

  A) the outstanding work is a carbon-copy of a previous
     effort, such as the second time provisioning a data
     center for the same system.

  B) the remaining new functional work is determined
     by the team to be in the last quartile and is
     well-defined, including remaining risks to
     successful completion.

Micro-estimates are the enabler of micro-management. A healthy team identifies the highest priority tasks to perform and does so in descending order, where priority is defined as risk to project success.

0 - https://www.youtube.com/watch?v=MhbT7EvYN0c

1 - https://www.goodreads.com/book/show/30650836-noestimates

https://news.ycombinator.com/item?id=37965582

  My estimate math:
  
  R = t × [1.1^ln(n+p) + 1.3^X]
  
  R - time it really takes.
  
  t - shortest possible time it would take without need to communicate.
  
  n - number of people working and involved during the process, both customers and developing organization.
  
  p - longest communication distance network involved in the project (typically from the lowest level developer to the end user)
  
  X - number of new tools, libraries, techniques, used in the process.
  
  Example. Project involving one developing writing code. Project would take 2 weeks (t=2), but it has 5 people (n=5) involved total, only 1 new tool (X=1) and longest communication distance is 4.
  
  2×(1.1^ln(5+4) + 1.3^1) = 4.5 weeks.

Those dates were mostly informed guesses of what would actually happen or go wrong. Importantly this was between friends. Needless to say, they turned out very accurate.

More often, the focus on estimating comes from management layers where incentives are not structured to reward anyone for accurate estimates, merely to punish them for missed deadlines.

Time to finished is only one dimension of estimation. With any unit of engineering work there may be code debt added or removed, complexity increased or decreased, morale increased or decreased, etc.

Focusing only on time, especially in a punitive way surely negatively impacts the others.

It feels like as a community, it would be useful to get more articles seeing things from the other side and exploring functional approaches beyond provide-a-worst-case-scenario-estimate.

There's a reason this dynamic is so pervasive. In order for everyone in an organization to do their job well, people do often need a realistic set of estimates. Can sales promise the prospect this integration? Can marketing plan a launch for the new feature? Can the CPO make a reasonable bet on a new line of work?

In my experience, the nuance here is more about handling the mis-estimates. How do we discuss the risks up front? How much work should we put into contingency planning? How do we handle the weeks / months before a deadline when it is clear that we need to limit scope?

- Realistic e.g. tech managers and people who favors agile/lean/XP/etc.

- Optimistic e.g. sales managers and people who want to promote.

- Pessimistic e.g. risk managers and people who need firm deadlines.

- Equilabristic e.g. project managers and people doing critical chain

The abbreviation is ROPE, and it turns out to work really well in practice to cover all four bases. My notes are below. Constructive criticism welcome.

https://github.com/SixArm/project-management-rope-estimate

I eventually learned, when asked for a "quick" estimate, I would give something drastically longer than I knew would be accepted. I said "But I can give you a better estimate if you give me a few days to do a better plan." This always got me the extra time to provide an estimate.

Tell me a story I want to believe, even if it isn't true. Then you can make it the teams' fault because they said they would and didn't.

In my personal experience: the first time I gave what could be construed as an official estimate.

There's the cases where it's used to roughly schedule work, or to prioritize features. My boss wants to know roughly how much is on our plates, so he can plan for known upcoming work.

Then there's the cases where it's more of a XY situation, where the boss is asking for estimates because in reality they've got a customer on the hook but they won't sign unless we can implement some functionality before go-live, or something along those lines. Typically that'll be a hard deadline, as customer will either have to switch to us or pay another year of licensing, and the boss wants to know if I can deliver.

I try to suss out if it's the latter, and if I'm unsure I will simply ask why they want the estimate.

If that's the case and it'll be a struggle to make the deadline, I'll try to help figure out if we can perhaps solve the core issue some other way. Perhaps a temporary solution that the client can live with for a week or two extra while we finish the proper solution, or perhaps we just simplify our proposed solution, enabling us to leverage existing infrastructure, and that turns out to be good enough for the customer.

I've found that most management just want to be involved in the process and have definite times set for updates and can handle timeline changes as long as information is coming at regular intervals.

With constant feedback, the whole team is participating in the emergent complexity, instead of being passive and just annoying you with "is it done yet"?

The issue isn't around tasks that are predictable in nature and therefore easy to estimate with a small margin of error, it's around complexity in software, unforeseen things, bugs, etc, which can compound for larger long term projects.

If engineers give estimates close to what it would be if everything goes right, then they risk overpromising and underdelivering if something goes wrong (hofstader's law). They might've just wanted to do the right thing by saving the company money and time, but in the end they footgunned themselves.

Or engineers intentionally over-estimate in order to manage the complexity, but then you end up with a lot of padding and parkinson's law. Because as soon as the engineer starts underpromising and overdelivering consistently, management will pressure them to lower their estimates because they have a track record of doing that, so instead they're incentivized to pad and then fill up the entire time they estimated even if it took less time.

Sprints were probably invented in order to deal with some of these issues, so that people just work with a bunch of smaller tickets that are much easier to estimate, with the more complex long term estimates going to management, which are incentivized to get it right because they're shareholders. That often leads to micromanagement and burnout, and it doesn't fix the padding/overestimation issue either, it might even amplify it in a lot of cases.

People here mention giving ranges or probability distributions, and have also equally mentioned that they don't work because management wants a single number, or management just assumes the best-case or middle-case of the range as the actual estimate, and then they still get in trouble for giving ranges and it didn't solve anything. It also doesn't solve the problem of unanticipated setbacks, the whole you don't know what you don't know thing, which can only really be solved culturally in some way.

While there are certainly bad managers that want to squeeze their workers, a lot of the time management is probably also pressured to give estimates and that's why they want and need that accuracy, because they're pressured by investors and clients that want to know how much time and money something will cost.

Overall the entire problem is a system cultural issue around managing complexity.

In this case don't estimate the time to build a poorly defined $something - invert the problem to estimate the value of $somethig.

It's amazing how many of those managers asking for estimates push back when they have to put one out. With all the same reasoning that engineers have when estimating.

A good manager should start with the Value first and allocate time-budget that makes that Value payoff.

And that's because their entire existence is based upon money, not results. I've only ever had one good manager, and that was because he knew what he didn't know and accepted that we do and are trying our best.

This is a problem people and we're not impressed.

estimates = contract amount = project budget ...

it is hard for a project team to negotiate later

often the estimates need to be competitive or bottom most for you to win the contract

no one acknowledges the unknown and risks at the beginning of the project

writing down more than 4-5 risks along with the bid amount is taken as a sign of a team that will not be easy to work with and fight over every bullet point whether something is in scope or a change request

and often the one who estimates and wins the project may not be on the actual project team with delivery accountability

At the very least, you need to do a bit of legwork to gather data prior to giving an estimate. Call it design, call it architecture, call it research, call it proof-of-concept, I don't care. Just stop insisting that decisions be made in a vacuum of data. Real results from running code trumps everything.

To be clear, you can produce software without using the scientific method. You can build anything without a data-driven process. But you get what you pay for. The head-in-the-sand approach ignores valuable information and yields poor quality as a result - it doesn't fit the definition of engineering.

"What's your estimate?" "I'm not sure." "Just ballpark it."

"Well, when would you want it by?"

This is the trap that new managers will fall into every time. If they give you an answer? Bingo. You give them "The Wince":

You suck air between your teeth, and with a frown say, "Oh, that's completely unrealistic. Where in the world did you get that number from??" Then you provide a number that's many multiples higher, or offer a reduced amount of work: "Oh, gosh. We'd only be able to get X feature done in that amount of time, and only if we got lucky and cut feature Y from the other project."

Regardless, whatever you do, don't be the first to provide a number. It takes a little bit to get the feel for it, like poker or buying a car.

Time is money, even in a big corporation. Treat it as such: It's a zero sum game.

Well yeah, because there's not an inherent power imbalance like there is in employment.

Part of this imbalance results in the ability for managers to employ Taylorization upon their directs. The majority of the time Taylorization hinders workers but management loves it because they can have more control in outcomes. What ends up happening though, is that an shadow work plan ends up getting established that management has less control over unless they want to drive out top talent by employing technocratic solutions to social problems.

The solution is simple: get better at estimating.

Software engineers act as if they're the only ones in the world who are asked to estimate and then held accountable.

It's a skill issue.

Feel the Bern.

No, actually, this sounds quite realistic and in many cases even reasonable. Medical insurance does this literally all the time. Insurance companies of many different kinds in general have to do this to prevent fraud and keep costs (and by extension premiums) low.

> There is no fixed path when modernizing a complex legacy system. There is no rulebook to follow.

Of course not. But as an engineer tasked with this kind of project your estimate should reflect some kind of plan for accomplishing the project, where the plan has more concrete and actionable steps that make estimation easier. And if you are good at your job, your estimate should account for known-unknowns (eg this part is contingent on something partially out of our control and may be delayed by X months) and give enough slack to handle inevitable unknown-unknowns without excessive padding. And uncertainty regarding any particular risks or variance in how long somethign takes should be explicitly noted and communicated.

My $0.02:

The unfixable estimate vs deadline problem ultimately boils down to money. A particular project might be worth it if it ties up 10 people for 3 months, but not worth it if it ties up 10 people for 12 months. And relatedly, businesses oftentimes find themselves needing something finished by a certain time to close a sale/keep a customer happy/catch up to a competitor, and the stakeholders on the other end (customers, investors, partners) need some kind of date for their own planning.

Good estimation is really about identifying the probability distribution of the completion date, rather than a single "expected" completion date, and identifying/communicating the related risks. For a big project this itself probably will take a few days to months to complete and communicate. If you do this properly, there should never be any ambiguity as to what is an estimate and what is a deadline. If you don't do this, you are basically assuming that your manager (and their managers and so on) can read your mind and have the exact same understanding as you do regarding how firm the estimate is + what the risks and possibly "actual completion times" are. You're also denying them the opportunity to help you derisk or accelerate the project by eg adding more people to work on it, reach out to some external stakeholders, or communicating the end date to eg investors/customers in a way that reflects risks. You also need to update the people that care about the completion semi regularly to tell them about any new risks or known unknowns for the same reasons.

If you are wildly off with your estimate even after spending the time to breakdown the project with a more actionable plan, identify risks, and come up with ranges/a distribution of outcomes - let's say you blow past your p99 time to completion - you are probably just bad at your job (or trying to operate above your skill level) and it would probably be very reasonable for your manager to hold you to a deadline with the threat of firing you/canceling the project. Yes, the sunk cost fallacy dictates that even projects that run over should probably be completed in many cases (though since overall budget/personnel are usually constrained I think the opportunity cost of doing other things becomes pretty important). But if people can't trust you to estimate properly they probably can't trust you to lead a project or to have estimated the remaining time to finish the project.

[1] https://users.ece.utexas.edu/~perry/education/SE-Intro/fakei...

# You need to align the rest of your team on things: maybe you have marketing tied to real world events, you're hiring and training to a sales cycle. If you do it wrong you waste money

# You need to fight scope creep which happens with unbounded timelines

# You need to fight a minimal product that's not viable because it isn't done enough for your market

and more. The ideal is for the decision maker to constantly have full information on progress and make instantaneous minute adjustments to keep things going at full steam. But there's no way to have full information and there's no way to make instantaneous minute adjustments if you're not a one-man shop. So everything else comes from the organizational effort of trying to work with that.