Why Your Data Stack Won't Last – and How to Build Data Infrastructure That Will

I used to think I could make a wonderful work of art which everyone will appreciate for the ages, crafted so that every contingency is planned for, every need met... But nobody predicts future needs that well. Someday whatever I make is going to be That Stupid Thing to somebody, and they're going to be justified demolishing the whole mess, no matter how proud I may feel about it now.

So instead, put effort into making it easy to remove. This often ends up reducing coupling, but--crucially--it's not the same as some enthusiastic young developer trying to decouple all the things through a meta-configurable framework. Sometimes a tight coupling is better when it's easier to reason about.

The question isn't whether You Ain't Gonna Need It, the question is whether when you do need it will so much have changed that other design-aspects won't be valid anymore. It also means a level of trust towards (or helpless acceptance of) a future steward of your code.

If this is a core conceit of the thinking then my answer is who cares?

Why do you want to try and influence a situation you're not even involed in?

Taking it back to the best lesson I was ever given in software engineering "don't code for every future".

Do what you're asked to and not get caught up in projecting your own biases into trying to make a "solid base" for the future when you can't know the concerns of said future.

> the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system

I put a lot of blame here on "the modern data stack". Hundreds[0] of point-solution data tools and very few of them achieve "business outcomes" on their own. You need to stitch together 5 of them to get dashboards, 7 of them to get real time analytics, etc.

We're going to see more products that achieve an outcome end-to-end. A lot of companies just want a few dashboards that give a 360 degree view of their data. They want all their data in one spot, an easy way to access it and don't want to spend fortune on it. That's what we're focused on at Definite[1].

We're built on the best open source data projects (e.g. DuckDB, Iceberge, Cube, etc.). If you decide to self host, you can use the same components, but it's generally cheaper to use us than manage all this stuff yourself.

0 - https://mattturck.com/landscape/mad2024.pdf

1 - https://www.definite.app/

2 - https://youtu.be/7FAJLc3k2Fo

Even at Google, which has some truly amazing homegrown technical infrastructure, you see what I called Promotion Driven Development ("PDD"). I didn't see this when it came to core technical infrastructure (eg storage, networking) but at a higher level I saw many examples of something being replaced solely (ultimately) because someobody wanted to get promoted and you don't get promoted for maintaining the thing. You get promoted for replacing the thing.

The most egregious example was someone getting promoted to Principal Engineer (T8) for being the TL of something that was meant to replace existing core infrastructure before it had even shipped. In the end it didn't ship. The original thing is still there. But wait, "we learned a lot".

So this happens because the organization rewards the new thing.

So why is your data infrastructure being replaced? Probably because of an organizational failure and it'll have almost nothing to do with technical aspects of that infrastructure. This is true at least 90% of the time (IME).

Data infrastructure is particularly bad for this because in any sufficiently large organization you will completely underestimate the impact of changing data dependencies for metrics, dashboards, monitoring, ML training and so on. Those things can be hard to find and map out and generally you only find them when they break. Sometimes they can break for years before anyone notices even when the thing is used by live production systems.

I tend to favor tools that have been around for a long time. A lot of the sites I have built have deployment scripts written in bash with dependencies on apt packages and repos, git to pull application code, and rsync to copy over files. It would probably be okay to update to zsh at this point. ;) I'm constantly shocked by the complexity of deployment infrastructures when I get into new projects: I've spent plenty of time working with Docker and Kubernetes and I have yet to see a case where these simplified things. As a rule, I don't throw out existing infrastructure, but if I'm doing greenfield development I never introduce containers--they simply don't do anything that can't be done more explicitly in a few lines of Bash.

One of the sites I still maintain has been running for 15 years. I ported it from Fedora (not my choice) to Debian about 8 years ago, and the only thing that changed in the deployment scripts was the package manager. I switched to DigitalOcean 5 years ago and the deployment script didn't change, period. The deployment script is 81 lines of bash. git blame shows 64 of those lines are from the original commit of the file. The changes are primarily to add new packages and to change the firewall to UFW.

And critically: I didn't write this script. This was written by a guy before me who just happened to have a similar deploy philosophy to me. That's a much better maintainability story than having to hire someone with your specific deploy tools on their resume.

- S3 for storage

- Glue catalog to describe / define source data shapes

- Athena to query the above

- dbt for business data modelling (has Athena and glue adapter)

The only difficult part I always struggle with is getting partitioning right.

- "Legacy" trading system in place (old but battle tested)

- New head of technology for the business comes in

- They get the bright idea to retire the old system and roll out a new system

- They promise ridiculous deadlines

- Roll out a half baked new system and retire AT MOST 60% of the old system

- They bounce to the next role at another firm b/c they have "retired legacy trading system / rolled out new system" on their resume

Meanwhile, the new system ALWAYS has some giant outage or near miss due to the rushed deadlines.

I've been there. Companies can be cheap about training and I was given none before building some sophisticated data stuff that surprisingly worked, but probably could have been simpler.

I got a much better job soon after, and hopefully my replacement got some training.

After that comes various kinds of caches, maybe postgres for frontend? Or something streaming?

But once everything is stored as files you gain the freedom to experiment or refactor.

It seems to me that the key is to make useful product based on sound technical decisions; entropy (a thing you can't control) will handle the rest.

> They quit and no one knows how it works

Either the infrastructure wasn't "amazing" in the first place or clueless management is looking for a scapegoat.

"Amazing" is an interesting word choice because a non-technical manager will be amazed by any blob of code. "Amazing" doesn't mean a straightforward, robust solution to a difficult problem.

Choosing third party, well known, FLOSS infra/open formats it's a thing, not developing their own infra with such tools in house is another.

Resume-driven engineering etc is the result of engineers with no stake in the business future. Any solution must involve incentives against "bad things" and favour "good things".

I feel like this is a major reason things don't get documentation. We don't get judged on it, nobody cares how good the docs are, but they DO care that we're shipping features.