Why software only moves forward

My perspective is that your typical business does not incentivize, or care to allow their engineers to do "proper" software engineering, which is effectively a proper abstraction of the requirements and its implementation from scratch (to a great extent), in order to maximize the extensibility and control of the software. Most orgs love to use some kind of existing framework or library that comes with heavy constraints on how you need to implement requirements (typically everything above the database layer).

If business domains are properly abstracted, most of the time extending the software with changes is something close to adding some configuration or using existing functions with different parameters. But the hesitation to create things from scratch comes at the cost of training and also transparency, two of the highest priorities for managers and stakeholders who not only want to hire people with minimal overhead to begin contributing, but also want to understand as much as possible what the engineer is doing and how they are doing it.

Software is incredibly maleable and has unlimited potential, yet somehow most places i've worked at get constantly bottle-necked by recurring problems that are very easy to solve with proper abstractions.

Marginally related. I wish I knew the context any more, but I encountered something like this.

App keeps updating. Version N understands version N-1's saved data, maybe N-2, N-3, but not indefinitely. Still you're lulled into a sense of security; your data will always be retrievable. Goes without saying that the app modifies the data store to the latest version.

But then you don't use it for a while and you it's now at N+10 and you still have data from N, and it's slid out of the "backwards compatibility window" and is no longer readable. And the intermediate versions of the App are not available. Now you're screwed. One such scenario, for sure, is updating from version N to, say, N+5 of Android and trying to migrate your data, but I think it was something else.

A better title is "Once you start keeping financial state, there are no rollbacks". The current title might as well be click bait.

Most software hardly moves at all. Consumer hardware moves backwards though, I can no longer get a phone with all the features I had in 2012, removable battery, headphone jack, and an sd-card slot.

I take a different lesson from this - there are parts of software that you can change easily and parts that are 100x more difficult, and you should design your system so that you know which regime you're in at all times.

my framing of this is "Data outlasts Code, yet Code keeps winning" - i think data/schema design should be much more emphasized in bootcamps and college classes but sadly it seems most of us have to learn these painful lessons on the job.

https://www.swyx.io/data-outlasts-code-but

Software only moves forward in the sense that time only moves forward. Not sure there's much more to it than that.

This highlights why a design process is so important. It feels great to start with a clean slate and “just code”, and many seem to get rewarded for that in a “move fast and break things” culture.

However there always needs to be a plan on how to migrate data that uses the old way to the new way. It’s not as sexy, but it needs to be done and takes time away from building new features. Skipping the design process is tempting, but it’s a direct path to this type of technical debt that’s easy to foresee.

Moving software "backwards" is hard, but not impossible. There are ways to structure both data and code that make it easier. A change being "rollback-safe" can be an important question in designs and code reviews.

This article brings up some very simple software engineering problems and the most obvious/widely used approach to dealing with them, but this is by no means the only approach that works best at scale. Also not necessarily the best approach at scale because sometimes you do need to roll back.

If you're serious about writing robust and reliable systems, think about V-1 and V+1 not just the version you are writing and deploying. The point about your code interacting with multiple versions in production is a good one, but also upgrades and downgrades should be considered.

Intro:

> aside: This email is adapted from a book chapter I spent the last 3 weeks writing. It's a complex idea that took 10 years to grok. I tried squeezing it into a digestible length. Would love to hear your thoughts.

Okay...

Next line:

> Data lives forever

It doesn't. Now I'm unwilling to overlook the title, which is clickbait, so I'm done reading.

This is why we try to keep state as small and isolated as possible.

Event store architecture seems to solve this rather eloquently.

I thought _everything_ only moved forward... ?!