Iron Mountain: It's Time to Talk About Hard Drives

Since storage constantly gets cheaper, 100GB first stored in 2001 can be stored on updated media for a fraction of that original cost in 2024.

I look forward to the first time logs from a few decades ago are required, and the media is absolutely dead.

EDIT: they weren’t even Azo dye, they were phthalocyanine. A decade was probably generous.

1. Incomplete copies with missing dependencies. 2. Old software and their file formats with a poor virtualization story. 3. Poor cataloging. 4. Obsolete physical interfaces, file systems, etc. 5. Long-term cold storage on media neither proven nor marketed for the task.

Managing archives is just a cost center until it isn't, and it's hard to predict what will have value. The worst part of this is that TFA discusses mostly music industry materials. Outside parties and the public would have a huge interest in preserving all this, but of course it's impossible. All private, proprietary, copyrighted, and likely doomed to be lost one way or another.

Oh well.

https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations

LTO-1 started in 2000 and the current LTO-9 spec is from 2021. But it only has backwards compatibility for 1 to 2 generations. You can't read an LTO-6 tape in an LTO-9 drive.

https://en.wikipedia.org/wiki/Sticky-shed_syndrome

> Sticky-shed syndrome is a condition created by the deterioration of the binders in a magnetic tape, which hold the ferric oxide magnetizable coating to its plastic carrier, or which hold the thinner back-coating on the outside of the tape.[1] This deterioration renders the tape unusable.

Stiction Reversal Treatment for Magnetic Tape Media

https://katalystdm.com/digital-transformation/tape-transcrip...

> Stiction can, in many cases, be reversed to a sufficient degree, allowing data to be recovered from previously unreadable tapes. This stiction reversal method involves heating tapes over a period of 24 or more hours at specific temperatures (depending on the brand of tape involved). This process hardens the binder and will provide a window of opportunity during which data recovery can be performed. The process is by no means a permanent cure nor is it effective on all brands of tape. Certain brands of tape (eg. Memorex Green- see picture below) respond very well to this treatment. Others such as Mira 1000 appear to be largely unaffected by it.

Data migration and periodic verification is the answer but it requires more money to hire people to actually do it.

I've got files from 1992 but I didn't just leave them on a 3.5" floppy disk. They have migrated from floppy disk -> hard drive -> PD phase change optical disk -> CD-R -> DVD-R -> back to hard drive

I verify all checksums twice a year and have 2 independent backups.

Yes, it's a bit of a PITA. OTOH, modern HD's are huge, so a relative few are needed. And we've lost 0 bits of our off-site data in our >25 years of using that system.

So what's actually wrong with hard drives for archival? Do they deteriorate? Do they "rot" like DVDs/blurays/etc have been known to do? Or is this just an ad for their archival service?

I'm struggling to understand why these miles of shelves filled with essentially hardware junk haven't been digitized at the time when this media worked and didn't experience read issues.

The article doesn't really provide an explanation for this other than incompetence and the business biting off more than it can reasonably chew. I'd be furious if I paid for a service that promised to archive my data, and 10-15 years later told me 25% of it was unreadable. I mean it's not like it was a surprise either. These workflows became digital 2-3 decades ago. There was plenty of time to prepare and convert this.

That's kind of what I'm paying you for.

As always, seems like the simple folk of /r/datahoarder and other archivist communities are more competent than a legacy industry behemoth.

Your entire article sounds like a sales pitch. Your solution is, well, it's bad, but trust us, we can maybe recover it anyways. Otherwise your article fails to convey anything meaningful.

We’re about to start a project to build an LTO-9 based in-house backup system. Any suggestions for DIY Linux based operation doing it “correctly” would be appreciated. Preliminary planning is to have one drive system on in our primary data center and another offsite at an office center where tapes are verified before storage in locked fireproof storage cabinet. Tips on good small business suppliers and gear models would be great help.

So it may sound like a sales pitch but I consider it more a warning notice

That aside, this sounds extremely old-fashioned, but it seems to me that the only media that is acceptable for long-term storage is going to be punched paper tape. How long does paper last? How long do the holes in it remain readable? Can it be spliced and repaired?

Why is this surprising?

It's been known for decades that magnetic media loses remanence at several percent a year. It's why old sound tape recordings sound noisy or why one's family videotapes of say a wedding are either very noisy or unreadable 20 or so years later.

Given that and the fact that hard disks are already on the margin of noise when working properly it's hardly surprising.

The designers of hard disks go to inordinate lengths to design efficient data separators. These circuits just manage to separate the hardly-recognizable data signal from the noise when the drive is new and working well so the margin for deterioration is very small.

The solution is simple, as the data is digital it should be regenerated every few years.

Frankly I'm amazed that such a lax situation can exist in a professional storage facility.

Edit: has this situation developed because the digital world doesn't know or has forgotten that storing data on magnetic media is an analog process and such signals are deteriorated by analog mechanisms?

We can't preserve bits like books.

Some 25 years ago, the hardest part in booting some Apollo workstations, was to make hard drives spin.

I think for the average person, the best thing to do, for long term archives is to take advantage of sturgeons law, "90 percent of everything is crap". triage the things you want to archive to a minimum, then print them out, at human scale, on paper. have physical copies of the photos you want to keep, listings of the code you are proud of, correspondence that is dear to you.

This will last, with no intervention, a very long time. Because as is increasingly becoming obvious, once the format drifts below human scale the best way to preserve data is to manage the data separate from the medium it is stored on with a constant effort to move it to a current medium. where it easily evaporates once vigilance drifts.

Ah, this stupid industry.

Or I have to mount them in another OS that isn't Windows. It's more than just adjusting DAW settings and updating plugins at this point you need to know that around 2000 the filesystems completely changed with NTFS and added security that wasn't present before.

By the time Vista/7 FAT hard drive support is gone from Microsoft land. There are of course add-ons and such but you still need to _know_ this happened and FAT drives look unformatted in modern Windows.

Any time you're physically warehousing old hard drives and whatnot, they're going to be turning into bricks.

Whereas with cloud providers, they're keeping highly redundant copies and every time a hard drive fails, data gets copied to another one. And you can achieve extreme redundancy and guard against engineering errors by archiving data simultaneously with two cloud providers.

Is there any situation where it makes sense to be physically hosting backups yourself, for long-term archival purposes? Purely from the perspective of preserving data, it seems worse in every way.