Why I self host my servers and what I've recently learned

I self-host a lot of things myself. There is one scary downside I've learned in a painful way.

A friend and I figured all this out together since we met in college in the 1980s. He hosted his stuff and I hosted mine. For example, starting in 1994, we had our own domain names and hosted our own email. Sometimes we used each other for backup (e.g., when we used to host our own DNS for our domains at home as well as for SMTP relays). We also hosted for family and some friends at the same time.

Four years ago he was diagnosed with cancer and a year later we lost him. It was hard enough to lose one of the closest friends I ever had. In his last weeks, he asked if I could figure out how to support his family and friends in migrating off the servers in his home rack and onto providers that made more sense for his family's level of technical understanding. This was not simple because I had moved 150 miles away, but of course I said yes.

Years later, that migration is close to complete, but it has been far more difficult than any of us imagined. Not because of anything technical, but because every step of it is a reminder of the loss of a dear friend. And that takes me out of the rational mindset I need to be in to migrate things smoothly and safely.

But, he did have me as a succession plan. With him gone, I don't have someone who thinks enough like me to be the same for my extended family. I'm used to thinking about things like succession plans at work, but it's an entirely new level to do it at home.

So, I still host a lot, but the requirements are much more thoroughly thought through. For example, we use Paperless-ngx to manage our documents. Now there's a cron job that rsync's the collection of PDFs to my wife's laptop every hour so that she will have our important papers if something happens to me.

Thinking carefully enough to come up with reliable backups like this makes things noticeably harder because not all solutions are as obvious and simple. And it's not something that ever occurred to us in our 20s and 30s, but our families were one tragedy away from not knowing how to access things that are important soon after we were gone (as soon as the server had trouble). There is more responsibility to this than we previously realized.

This is tangentally-related, but I feel it is very wrong that so many smaller governments (e.g. smaller US cities) host "public information" on private servers (e.g. links to PDFs from a Google Drive)... or even worse inside some walled-garden (e.g. Facebook).

My own personal DNS does not resolve to any Google/Facebook products, reducing profiling; but by denying their ad-revenue, I also deny myself access to information which IMHO should be truly available to the public (without using a private company's infrastructure).

I absolutely understand that many people will just say "don't block them, then." My argument is that governments should not host public items on private servers.

I think we'll see some stratification in the self hosting community over the next few years. The current community, centered around /r/selfhosted and /r/homelab, is all about articles like this. The complexity and learning are sources of fun and an end in themselves. That's awesome.

But I think there's a large untapped market for people who would love the benefits of self hosting, without needing to learn much if any of it.

I think of it similar to kit car builders vs someone who just wants to buy a car to use. Right now, self hosting is dominated by kit cars.

If self hosting is ever going to be as turnkey as driving a car, I think we're going to need a new term. I've been leaning towards "indie hosting" personally.

I agree with Christian about pretty much everything here. We self-host for multiple reasons, and we don't necessarily need others to necessarily understand our rationale, although that'd be nice.

For me, one thing that stands out as something driving the desire to self-host everything is that large corporations, given enough time, invariably let us down. Christian's experience with Contabo illustrates the one game that I will do any amount of work to avoid: people who pretend to know what they're talking about but who really only waste our time in hopes to put off dealing with an issue until someone else actually fixes it.

The one place where I can't avoid this truly stupid game is with getting and maintaining Internet for my clients. You're not paying for "enterprise", with "enterprise" pricing of $750 a month for 200 Mbps? Then tough cookies - you'll get the same junk we force on our residential customers, and you'll never, ever be able to talk to a human who has any clue what you're talking about, but you'll be able to talk to plenty who'll pretend to know and will waste hours of your time.

The more time they waste of mine, the more energy I'll expend looking for ways to subvert or replace them, until I eventually rely on corporations for the absolute minimum possible.

Self hosting makes you realize how insanely expensive cloud providers are. AWS charging for IP4 addresses was the last straw for me.

Just today I had to sign up for a service and went to bitwarden app on my phone to generate password (linked to self hosted vaultwarden server) but the new password entry couldn’t be saved into the app because the server was unreachable.

Then I had to go restart my VM and reconnect my VPN. I am now thinking about switching to bitwarden premium and opt-out of self hosting for password managers.

I self host too.

a couple points

- proxmox hits an SSD pretty hard, continuously. I think with zfs, it probably hits even harder. A lot of it is every second keeping state for a cluster, even if you have only one machine.

- I bought mikrotik routers for openwrt. I tried out routeros, but it seemed to phone home. So I got openwrt going and didn't look back. I am switching to zyxel since you can have an openwrt switch with up to 48-ports.

- I used to run small things on a pi, but after getting proficient at proxmox, they've been moved to a vm or container.

- the most wonderful milestone in self-hosting was when I got vlans set up. Having vlans that stayed 100% in the house was huge.

- next good milestone was setting up privoxy. Basically a proxy with a whitelist. All the internal vlan machines could update, but no nonsense.

- it is also nice to browse the web with a browser pointing at privoxy. You'd be surprised at the connections your browser will make. Firefox internally phones home all. the. time.

Has anyone tried those Lithium Ion UPSes? ~5 years ago we removed the UPS from our dev/stg stack because in the previous 5 years we had more outages caused by the UPS than issues with the utility power. A better battery technology sounds compelling.

For production, of course, it's all dual feed, generator, UPS with 10 year batteries, N+1.

Home labs are great. They are a good learning tool to understand systems in _isolation_.

They're terrible for understanding emergent properties of production systems and how to defend yourself against active and passive attacks. Critically you also need to know how to unwind an attack after you have been bitten by one. These are the most important parts of "self hosting."

Otherwise, you might be getting in the habit of building big rube goldberg machines that are never going to be possible to deploy in any real production scenario.

Make it real once in a while.

One^W two things what makes self-hosting a bit more attractive:

a) besides the some bootstrapping nuances you are not forced to have a working phone number to be able to use some resource. It's usually not a problem until... well until it became a problem. Just like for me yesterday when for whatever I tried but I couldn't register a new Google account. There is just no other option than SMS confirmation.

b) there is way less things to change 'for your own convenience', like a quiet removal of any option to pre-pay for Fastmail.

PS oh and Dynadot (which I was happy using for more than 10 years) decided (for my convenience, of course) to change the security mechanism they used for years. Of course I don't remember the answer for the security question and now I forced to never ever migrate from them, because I literally can't.

A small suggestion about resources: try using NixOS/Guix System instead of containers to deploy home services, you'll discover that in a fraction of resources you get much more, stability, documentation and easy replication included.

Containers now, like full-stack virtualization on x86 are and was advertisement stuff pushed because proprietary software vendors and cloud providers need them, other do not need them at all and devs who works for themselves and generic users should learn that: if you sell VPS et al. obviously you need them, if you made your own infra from bare metal adding them it's just wasting resources and add dependencies instead of simplify life.

> My VPS, hosted by Contabo, randomly went down for almost 4 days.

Don't use Contabo! I have had this issue with different servers almost monthly, servers going down for 1 or 2 days, without any announcement or communication. They never say if anything is wrong, never apologies, have regular "unplanned maintenances", contacting support is almost impossible (or takes 3-4 days for a reply). As OP did, I am also migrating from Contabo to Hetzner.

I used to self-host a lot of things:

1. My blog

2. My friends' blogs

3. BIND for all this

4. A mail-server on this

5. A MySQL database on this

All this was on a Hetzner server that was nominally set up to be correct on restart. But I was always scared of that because I built this up from when I was a teenager onwards and didn't trust my younger self and couldn't find the time to audit. 10 years afterwards, with 10 years uptime, and no consequences of data loss or theft (it might have occurred, just that nothing affected me or my friends) Hetzner actually warned me they were going to decomm the underlying instance and no longer supported that VPS.

I backed everything up, copied it, and for the last 8 years have faithfully moved from home to home carefully transporting these hard-drives and doing nothing with them.

When I finally set up everything again, I did it much more manageably this time, with backups to Cloudflare R2 for the database and resources, and Dockerfiles for the code. I restarted the machine and brought everything up.

And now I use GSuite instead of my own mail. I use Cloudflare instead of my own DNS. There's a lot I outsource despite "self-hosting". It's just far more convenient.

So the answer is that I had no BCDR on the old thing. Maybe I'll train my kids and have them be my BCDR for the new thing.

I went back and read some previous blog posts. He was part of the great 2023 layoff. I'm curious where such a talented guy landed. Did he find a position?

We used GCS (like S3 for GCP) for storing prometheus backups (via thanos) at $previous_job. That thing cost an arm and a leg for what it did (presumably, thanos was partially at fault but I didn't dig that much).

I moved to minio (in a GCP VM) and reduced our overall GCP bill by 70%. Yes, using cloud storage was 2/3 of the cost of our cloud infrastructure.

But overall, going too far in the self-hosted route has its costs. Hardware depreciation is one (and the author mentions UPS which seems huge in addition to being critical), cooling/powering, and of course the time for maintenance. If you are going this route you are doing this because you want to learn that stuff, not because you want to save the subscription. Otherwise, just use less services and keep price comparison lists updated.

I love this but I'd like to know more about the hardware.

As an aside, I find it amusing that commenters here say that they "self host" in the cloud. It ain't self hosting unless the server is under the same roof as the family!

I do not understand this. Why use a three-node Proxmox cluster to self-host services that would (generously) need one N100 (with straightforward LiIon battery backup)? Why all this complexity for a relatively barebone setup?

They aren’t even self-hosting their own files, relying on Wasabi instead, and I do not understand why. Surely there is an HDD somewhere between those three nodes in the Proxmox cluster?

The author mentions they use Komga for comics and calibre-web for ebooks. However I personally find the Komga is a much better eBook reader than calibre-web. I use it for all my books and works incredibly well on mobile. The only thing I don't like about Komga is the logo, which I simply replaced on my private instance.

No mention of how many terrible routes there are to residential ISPs? The packet loss you get as a home enthusiast service provider to points of presence around the country is abysmal. VPNs become a requirement for accessing these non-cloud services. Always a troll toll.

Another great self-hosted app...

https://github.com/dani-garcia/vaultwarden

Also, if looking for a cheap/low power pi alternative. Check out used Wyse 5070s on eBay. They run DietPi easily and have M2 slots.

https://www.dell.com/en-us/shop/cloud-client/5070/spd/wyse-5...

There are very few people or projects talk about actually backup and restore application data.

This is especially true the open source systems like truenas scale. Any turnkey selfhosting software that's not implement a robust backup restore system is essentially holding your data hostage.

This is a very interesting blog post tbh.