Removing stuff is never obvious yet often better

I don't know if this calculator was good or bad, but the rationale sounds superficially ridiculous.

> Visitors who didn't see the calculator were 16% more likely to sign up and 90% more likely to contact us than those who saw it. There was no increase in support tickets about pricing, which suggests users are overall less confused and happier.

Of course if you hide the fact that your product might cost a lot of money from your users, more of them will sign up. Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. That's not something you will figure out from a short-term A/B test on the signup page. So this seems like totally useless evidence to me.

I see this dynamic frequently with A/B tests. For example, one of my coworkers implemented a change that removed information from search result snippets. They then ran an A/B test that showed that after removing the information, people clicked through to the search result page more often. Well, obviously, it makes sense that they might click through more often, if information they wanted which was previously in the snippet, now requires them to click through. The question of which is actually better seemed to have been totally forgotten.

Upvoted to spread the immense wisdom in this post. But golly I must say the line can get blurry quickly.

> Would anything of value be lost if this or that chunk of it was removed?

In early stage projects I’ve seen this mentality backfire occasionally because it’s tough to estimate future value, especially for code and data.

For example, one time in a greenfield project I created the initial SQL schema which had some extra metadata columns, essentially storing tags for posts. The next week, a more senior engineer removed all those columns and associated code, citing the YAGNI principle (“you aren’t gonna need it”). He was technically correct, there was no requirement for it on our roadmap yet.

But the original work had taken me maybe an hour. And the cost of keeping the data around was approximately zero. It seemed he didn’t consider that.

Guess who needed the columns to build a feature a year later? Yeah me, so I found myself repeating the work, with the additional overhead of prod DB migrations etc now that the product had users.

I guess my point is, sometimes it’s also wise to consider the opposite:

Would anything of value be gained if this or that chunk of it was removed?

In this article the evidence is clearly affirmative, but in my case, well it wasn’t so clear cut.

> In an internal poll, 7 of every 10 people in the company thought the version with the calculator would do better.

An interesting and pretty classic dynamic - I liked the article overall but I think this point didn't get the highlighting it deserved. If 30% of the people involved think that the calculator is a bad idea that signals a potentially huge problem even if the majority think it is fine.

Be alert to the politics here. Although it seems otherwise, people generally don't like to criticise other teams in the business unless there is a political gain to be had. By extension, if I polled the company and asked "is this thing my team did a net positive?" I'd expect the default position to be "yes" as people wouldn't want to stir the pot for no reason. 30% of people signalling that it might be value-destructive is much more significant than it seems because of that. It should trigger some fairly thoughtful consideration of why exactly they thought that.

In this case they seem to have indeed been alert to all that and the story has a happy ending, which is nice. But this poll result was always evidence of a serious problem.

The general message is interesting.

This specific bit kinda gives pause though:

> One slight misinterpretation and wrong input and you'd get an estimate that's overstated by as much as 1,000x.

Does it also mean that in real world usage, one slight misinterpretation or misevaluation of your metrics and you're liable to 1000x more than you planned to ?

I totally see this as a reality of online billing systems. I've misconfigured GCP prototypes and ended with 100+ bills where I though it would be 2 or 3 at most and didn't care to watch for a few days.

But I'd understand a client bailing out when they realize slight changes to the sliders result in wild increases in the planned pricing. And removing the tool would sure help for registration, but not help the customer if they hit these kind of issues down the line.

Here’s a followup for the author: follow your own advice. Remove the “Psst... Get the next post in your inbox” interruption midway through your post. Dare to remove the stupid button that follows you around as you scroll.

I counted five obvious ways to subscribe on that page alone. Five! Do you really need to shove them in our faces and in the middle of the freaking content? Do you think you get more subscribers by interrupting people and being annoying? Are those the kind of subscribers you want?

Removing stuff is often obvious, you just have to take your mind out of the “more more more, make money, attract customers, more more” gutter mindset and think “what is respectful to users, how can I help them while simultaneously treating them like human beings instead of wallets to be milked”.

> Before long, a dedicated Slack channel was created, which accrued over 550+ messages representing opinions from every corner of the company. Another few thousand words and dozens of hours were spent in meetings discussing what we should add to the calculator to fix it.

This is a symptom of over hiring. Too many people removes agency.

When people lose sight of what's actually important and feel that they must reach consensus by committee then there are too many people.

Perhaps removing a pricing scheme so complicated that it literally can't be modelled usefully by the customer would be even better?

My employer has something like 250 products. Five of them are responsible for 80% of the revenue.

Dev teams of those five are stretched thin, barely able to keep up with bug fixes, and struggling to add new, important features -- so much so that getting anything on the roadmap is an impossible battle, no matter who is asking.

There are thousands of devs at the company, most of them attached to the products that contribute relatively nothing to the revenue stream.

I don't know if it's not obvious -- it must be obvious -- that to move forward the right thing is to cut most of the products and refactor the teams to drive the remaining money-makers. Yet, this has not happened, and I see no indications, no murmurs of it happening. Corp politics are wild.

I experienced something similar. We had a website with several products that seemed fairly similar, so we were concerned that people might have trouble deciding which one to get, and not buy as a result.

So we made a product advisor applet where the user would answer a few questions and it would suggest one or two products that would be best for them. Getting the applet right took a bit of work, but once it was done it worked very well.

We put it live on the site and.... conversions dropped precipitously. We A/B tested it and yep, it definitely hurt conversions. I still don't know why it hurt conversions, but it did. So we moved it from the homepage to a FAQ section of the site, and hardly anyone ever used it at all.

Interesting case study, but I'm skeptical of the broader implications. Pinecone is notoriously expensive compared to other vector database services. A horizontal price comparison reveals several better options in the market. Removing the calculator doesn't solve the core issue - it just obfuscates costs and makes it harder for users to compare options upfront. In my view, this approach merely reduces the comparison stage, potentially leading more uninformed users to upload data without fully understanding the pricing implications. While simplification can be valuable, in this case it seems to benefit the company more than the users. A better approach might be to improve the calculator's accuracy and usability rather than removing it entirely. Transparency in pricing is crucial, especially for B2B services where costs can scale quickly.

Not only is it often better but it can literally enable you to get to market an order of magnitude faster (at with higher probability of success).

I'm working on a tool that makes running wire inside residential homes easier. It requires carving a channel on the back side of the baseboard.

My original idea required a mechanical tool with a motor [2]. We prototyped a working version of this but always felt that the manufacturing challenge would be large.

We ended up adapting the system to existing routers. That meant our product was just a series of extruded parts with almost no moving parts [1].

[1] https://trywireshark.com

[2] https://www.youtube.com/watch?v=DTzsU9gMF3I

I don’t get it. Presumably the pricing model didn’t change, so all you’ve done is push the burden of doing the math onto the user (or more realistically, hope they just don’t even bother?) If users are frequently estimating costs that are off by orders of magnitude, surely the correct response is to change the pricing model so it’s easier for customers to understand?

Your biggest next TODO is to rethink how you price the usage and make it 1000% simpler.

We, as humans, always carry a few types of things with us on a journey.

Things we like, things we don't like, things that are totally useful, things that are somewhat useful etc.

There comes a point where we need to let go of the thing(s) we like - I called them the precious payload where we are most reluctant - and in this case the 'calculator' was the precious payload so many people in the company were unwilling to remove this feature except for one person.

In business, adding a completely new feature or subtracting an age-old feature is extremely difficult but oftentimes, this is where growth comes from.

> We assume that if something exists then it exists for a good reason

I suspect that this often exists because people have tried this, and been summarily burnt by either politics, or some form of Chestertons-Fence.

Which leads to the question of: how and when do we discern the difference? How do we design our teams and engineering to do things like,

- ensuring the psychological or political safety to suggest slimming down systems without having every other team come down on your head.

- incentivise “svelte” systems at a product level, without falling into Google-levels-of-“lmao this is going away now thanks bye”

- engineer for slimmer systems. There’s lots of ink spilled on the topic of making things extensible, or able to have stuff added to it, but seemingly little about the opposite. Is it the same engineering practices, or are there other concerns you’d need to take into account, if so, what are they?

- would you as a customer pay for something that was better at its purpose but didn’t have every single feature under the sun? I sure would, how many other people/orgs would, if given the option? I semi-controversially think that too many tools providing too much flexibility mostly encourages orgs to paint themselves into wacky processes, just because they can. I doubt this entirely goes away, but if given less feature/flexibility bloat, how much “fat” (process/overhead/friction/etc) could be trimmed out?

May be the problem is just having so many people (the real cost).

Pinecone is just generally thought as something extremely overpriced:

https://www.timescale.com/blog/pgvector-vs-pinecone/

Until they can solve the feeling (or cut down on their own overhead), I don't see that great future for the company.

Is not just true within product design, but is even more of an issue with regards to the rules that our societies live under.

One could argue this is exactly the prescription that the Gnome project embraced. Remove, simplify, excise. To mixed reviews.

Also see Muntzing: https://en.m.wikipedia.org/wiki/Muntzing

Reminds me of the advice that, if you need to join two mismatched pieces together, you have two options.

1)Add an adapter that’s customized on both ends, or 2) subtract the differences so they mesh directly.

Always look for opportunities to subtract, rather than add. Your system gets easier to understand over time as it becomes the purest representation of itself, instead of a collection of gap-fillers.

i think of this lesson often (i don't remember who told me) elon building something at spacex and ruthlessly cutting out pieces they initially thought were needed. less complexity meant cheaper and faster construction so = more tests.

i use this in day to day to life: making plans, buying things, managing others - reducing has led to more accomplished.

Dubious claims with really terrible arguments.

Simply removing a feature without addressing the root cause of confusion will lead to other problems down the line, such as increased customer support demands or user dissatisfaction when actual costs differ from expectations.

I love how this lesson applies to AI software products: it might not be obvious at first, but removing that dedicated vector database (which is only there because everybody's using one in the first place) often improves things :^)

Is it standard to use percents of percents in conversion tracking? Going from 20 to 23% conversion rate is not a 15% increase in conversions, it is 3%. If that is the kind of shenanigans being played, there is something else to remove

I never feel more productive than on the days when I can delete a whole lot of code.

I can see how this can be so useful across the board. Well explained with something as tiny as a calculator.

And, yes, removing things should be rewarded. It's a cognitive bias to reward just for adding things.

How common is dropping a feature because of A/B testing results?

I feel like it should have a lower priority on such business decisions. My guess is that calculator removal was approved before A/B testing have started.

I just finished Elon Musk’s biography by Walter Isaacson and I was struck by how often this actually works. Whether you’re actually sending humans to the ISS, or building cars removing requirements and complexity is a surprisingly effective tactic.

I love his “algorithm” for getting shit done. The final step is, “If you don’t put back at least 10% of what you take out, you aren’t removing enough.”

> If there's a big backlash from the team then you're on the right path.

People hate the proposal to remove the microwave ovens and free coffee from the kitchen, so it must be the right idea!

Why stop there? Why not just remove all pricing completely, and let your clients contact sales for the shake down? That model seems to work great for many SaaS companies.

The last approximation of this that really stuck with me was The Bitter Lesson.

It would seem a common thread is our inclination to embed part of our ego into everything we do.

Great post and Elon Musk has similar rule in his thinking.

For anyone who liked the trick in the post consider checking out TRIZ: https://en.m.wikipedia.org/wiki/TRIZ

There are too many interesting ideas in this framework, but one of the first steps in the algorithm of solving a problem is to "Formulate ideal final result", according to TRIZ.

Now the "Ideal final result" has a specific definition: The part of the system doesn't exist but the function is still performed.

I'm having a lot of fun with this and other tools coming from TRIZ when solving problems every day. You might like it as well!

As for A/B testing and getting unexpected results: inside TRIZ there is explanation why it works - it is called "Psychological innertion". i.e. when engineer is getting a problem it is usually already formulated in a certain way and engineer has all kinds of assumptions before he even starts solving a problem. This leads to him thinking along specific "rails" not getting out of box. Once you have algorithm like TRIZ, it allows to break through psychological innertion and look at the problem with clear eyes.

Some other trick one might use to find interesting solutions to the problem from the post: "Make problem more difficult". I.e. instead of how to make calculator simple and unrestandable, formulate it in a different way: "how to make calculator simple and unrestandable, visual, fun to use and interact with, wanting to share with your collegues?"

"Turn harm into benefit". calculator in the post is treated as a necessary evil. Flip it. Now we have a calculator, but we could show some extra info next to prices, which our competitors can't do. We can compare with competitors and show that our prices are better, calculator can serve as a demo of how customer is always in control of their spending as the same interface is available after they become customer to control their spend etc.

Formulating this way already gave me some ideas what could be added to calculator to make it work.

Hope it helps someone.

> Except for one person, it never occurred to this very smart group of people that removing the source of confusion could be a good option.

Reminds me of the quote: "It is difficult to get a man to understand something when his salary depends on his not understanding it."

That applies to us as software engineers too, our salary depends on having projects to write code for so it's not so surprising that a very smart group of people don't often consider that doing nothing or removing something is a valid solution. I like the author's observation on this too: it would be nice if removing things were rewarded. I wonder of the employee who questioned the calculator's necessity got any reward.

YAGNI: You aren't gonna need it.

“Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away”

“Less is more”

“The value of a creative product doesn’t lie in how much is there, but in how much has been discarded.”

rttm: Reduce to the max

"Kill your darlings"

...

This is how you end up with cars with no stalk and no shifters (cough, Tesla).

I hope the wages of sales people are lower than that of a calculator.

The sad truth is, nobody ever got promoted for removing stuff.

Software is finished when there is nothing else to remove.

This is a beautifully designed blog interface. It’s one of the best I’ve seen, making it a pleasure to read.

removing unobvious bugs makes things better

> The calculator also gave users a false sense of confidence, which meant they were unlikely to double-check the estimate by reading the docs, contacting the team, or trying it for themselves.

How dare they think that a thing called calculator is accurate and not double check!

no doubt about it.

To be a sociopath for a second:

Up-front pricing is only good for the buyer and never good for the seller. And while being an advocate for the user is a good thing in general, it's not a good thing if the result is less revenue. And if you want to present a pricing estimate your incentive is always to lowball and underestimate.

Businesses that are successful tend to exploit information asymmetry. This is frustrating as a consumer but if you can guarantee that the user doesn't know their final cost you can always sell them a sweeter picture than reality and play off the cost of switching to a competitor, if one exists.

Total aside but this is why housing prices are insane, at the most micro level no buyer has any idea over the final cost until the last possible moment at which point the cost of undoing the transaction probably outweighs the risk of continuing - psychologically, at least.

(I may or may not have been the victim of this recently and thought about it from the seller's side)

Listened to a Lex Fridman podcast and wrote an article on it. Nice.