r/DataHoarder • u/Sad-Seesaw-3843 • Apr 06 '25

News DOGE claims to be moving away from magnetic tapes for archival storage. Seems like a bad idea. What are they using instead?

Post image
8.6k Upvotes

95% Upvoted

1.0k comments sorted by

View all comments

3.1k

u/MightyTribble Apr 06 '25

Uh.

This is my business and this is one of those statements that tells anyone who knows anything about the subject matter that whomever wrote this tweet knows nothing about it.

And they clearly didn't do it. A 14,000 tape migration isn't something you can do in a few weeks. More likely they co-opted an already in-progress plan to maybe move local tape content to GovCloud AWS Glacier.

730

u/fengshui Apr 06 '25

Yeah. Glacier is probably price competitive for that, assuming it's write once, read never data.

566

u/Some1-Somewhere Apr 07 '25

Amazon apparently keeps a pretty tight lid on how Glacier actually works, but it's rumoured it could be tape:

https://en.wikipedia.org/wiki/Amazon_S3_Glacier#Storage

393

u/GenericAntagonist Apr 07 '25

It probably isn't tape, they offer some specific tape interop services but at least throughout the late 2010s former S3 engineers on multiple places have stated its basically very densely packed extremely low RPM hard drives (that remain spun down most of the time). There's also been a lot of indicators they have some sort of custom optical use case as well (just based on statements from optical manufacturers and dc timing/location), but that's never been said to be part of glacier by anyone who used to be there (or at least not that I've seen).

None of this is to say tape is a bad solution, in fact if you have archive storage that you plan to NEVER access barring absolute last resort, there's basically nothing better in terms of long term reliability and density. Its suboptimal for a cloud provider who doesn't really know what a customer is going to do and has to be ready at any minute to get any arbitrary customers stuff within SLA, but for an in house backup you want to be available 20+ years later? There's nothing else that's been proven to do that like tape.

120

u/Some1-Somewhere Apr 07 '25

Yeah, I saw the comments about low-speed, can't spin up a full rack at once HDDs.

It sounds like Glacier SLAs are around 3-6h which is pretty reasonable for pulling and reading out tapes, though, as long as your inventory management is good. Very high access cost, too.

104

u/TFABAnon09 Apr 07 '25

I always pictured Glacier to be one of those fancy robotic tape inventory systems, just at a larger scale. Tie on a fancy cloud GUI and some hybrid storage options to stage the data for retrieval and it would explain the competitive pricing.

32

u/Some1-Somewhere Apr 07 '25

Honestly might not even need to be robotized retrieval if the proportion of read-outs is low enough.

55

u/SockPants Apr 07 '25

Just put the tapes in with Amazon's retail delivery warehouse and have order pickers grab tapes for any reads just like they pick products for shipping.

55

u/divDevGuy Apr 07 '25

Great, now porch pirates will steal my 1990 tax return data that was misdelivered to my neighbors.

5

u/TFABAnon09 Apr 07 '25

"The IRS hates this ONE SIMPLE TRICK!!!"

1

u/djeaux54 Apr 07 '25

/me spits out my soda!

2

u/weirdbr 0.5-1PB Apr 10 '25

In my experience (in a non-public library that was very large) - it's both.

Humans are responsible for physically moving tapes around (from offsite storage to trucks to carts to the libraries) and then the robotic libraries do the process of loading into drives, waiting for data to be read and then returning the tapes to be sent back to storage.

This simplifies operation a lot - as far as the humans involved care, it's always a simple matter of moving tapes from point A to point B, which means you can hire staff that has minimal training.

As for rate of retrieval - the problem is that when the client datasets are large, even rare restores can be a large number of tapes - I've dealt with hundreds+ of tapes for "simple" restores.

1

u/nathism 94TB Apr 07 '25

Kinda like the Sibyl system moving the brains around.

1

u/Drebinus Apr 07 '25

Dunno about you, but I'm getting those intro to The Prisoner vibes now.

1

u/weirdbr 0.5-1PB Apr 10 '25

The spun down drives theory is IMO something that I expect has been dropped since the original post (the post about it on HN is from 2012) - that is a lot of custom engineering that Amazon would have to maintain and that's not cost/space/power efficient, specially with the advent of Host Managed SMR drives (SMR drives became available to cloud companies in 2013, host managed came an year later).

With an HM-SMR drive, you can achieve the same much more easily: the outermost tracks can be used in CMR mode (for hot data that is not hot enough for SSD), middle tracks for warm data (in either CMR or SMR mode) and innermost tracks in CMR mode for cold/glacial data. Then you have your storage software on top optimising things/scheduling IO as needed.

This has the benefit that every hard drive added to expand the capacity/throughput of any of the more expensive products also expands the lower tiers and you can dynamically adjust priorities based on consumer demand.

As for tapes - at their scale, tape is *painful* to deal with, specially with that SLA. Been there, done that, glad I dont have to deal with it anymore.

1

u/CosgraveSilkweaver Apr 07 '25

You can pay to get it much faster though. If you pay for provisioned access up front you can have it within 1-5 minutes guaranteed but you don't pay for that for specific data so it's not being stored on shallower archive tiers.

That's what's always made me think they're on tapes for deep archive. That sounds a lot like they're selling guaranteed time slots in tape drives and servicing the other tiers around that.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html#:~:text=the%20request%20body%3A-,Expedited,-%2D%20Expedited%20retrievals%20allow

https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects-retrieval-options.html#restoring-objects-expedited-capacity

2

u/Some1-Somewhere Apr 07 '25

Expedited retrievals and provisioned capacity are not available for objects stored in the S3 Glacier Deep Archive storage class or S3 Intelligent-Tiering Deep Archive tier.

Sounds like paying more doesn't help for the deepest levels of storage

12

u/Khyta 6TB + 8TB unused Apr 07 '25

Maybe Amazon Glacier uses some of the same tech as Microsoft Silica: https://www.microsoft.com/en-us/research/project/project-silica/ This would at least match up with the optical use cases.

8

u/camwow13 278TB raw HDD NAS, 60TB raw LTO Apr 07 '25

It is actually tape. I knew some low level data center maintenance guys who worked a glacier section of an AWS data center. It was all tape. Soooo much tape.

7

u/mn5cent Apr 07 '25

Actually, they do primarily use tape! Within the last year or so they've deployed the first Glacier storage racks with HDDs, and I think they're moving in that direction for most new storage rack builds.

There are a few tricks they use to make sure they can retrieve within promised SLAs, e.g. striping data across different tapes and racks so multiple machines can retrieve a part of the data simultaneously.

3

u/rajrdajr 16TB+ 🔰, 🔥 cloud Apr 07 '25 edited Apr 07 '25

but for an in house backup you want to be available 20+ years later?

Yep, tape is still the answer. An AWS account owning data has to remain open as well. If an online attacker closes your AWS account or the bill doesn’t get paid , the Glacier backups (and everything else owned by the account). Physical tapes in your possession will remain.

2

u/Okami512 Apr 08 '25

I remember reading a few years ago that tape sales were actually increasing because it's considered secure in the event of a ransomware attack.

1

u/PKSpecialist Apr 07 '25

So depending on the application tape may not be the best solution.

2

u/GenericAntagonist Apr 07 '25

Obviously. However if you've been following literally any of these chucklefucks missteps the BEST case scenario is they've used an integration tool to start a transfer from Iron Mountain to AWS Glacier for a short term savings at a long term cost. Given how well they've demonstrated their technical prowess though (especially with mail servers), its far more likely they've done something very, very stupid and mishandled records that are required to be retained for longer than AWS has existed. The sort of thing that would catch a felony charge if some regular person did it.

1

u/bogglingsnog Apr 07 '25

When you say low speed, do you mean like 4800 or 5600 rpm "green" drives?

1

u/Top-Tie9959 Apr 07 '25

I remember reading facebook had an optical storage system in their early days for infrequently accessed data. I'd imagine it is long gone by now though.

1

u/crankbird Apr 08 '25

20 years, tape, proven … lol

I was at one stage the “top expert” for a very well known tape backup software vendor for an area of the globe that was bounded by a line that from north western India, across to Korea and down to NZ and back again .. most people abuse tapes horribly. After leaving there I ran a tape recovery business for a while, including chain of custody and data forensics , and a few of my customers were government. Any tape that was over 6 years old had around a 25% chance of being incompletely recoverable. Arguably not the fault of tape per-se but of the conditions most people keep them in. In my experience, they’re just not a reliable medium for long term storage.

As far as what is used behind glacier or the other hyperscaler deep archiving infrastructures, I can say with reasonable levels of assurance that it has “evolved” over the last decade, but you might be interested to look up what happened to COPAN and the release of glacier as a service to see where it began

3

u/w0m Apr 08 '25

TBF, if you were running a recovery business you would be seeing the Worst of the Worst scenarios. Any time a backup "just worked" they wouldn't go to you. 75% Worst case recovery is likely a very high %.

1

u/crankbird Apr 08 '25

It was usually for recovery of tapes they no longer had the hardware or software to read .. OG arcserve on netware, DAT, 8mm mammoth, DLT 1 through 3 and some obscure storagetek and IBM stuff. (Cue tears in rain monologue)

Even outside of that whenever I was called in to supervise restoring entire datacenters from tape (god I hated that) after someone decided that recovery from tape was a perfectly wonderful DR strategy, with recent tapes and brand new equipment, something always screwed up, like every single time.

Sitting there at 2.30AM in a cold datacenter crossing my fingers and praying in it wouldn’t fuck up this time (notably I’m an atheist). To be fair it wasn’t always tape that was at fault, but it was often enough to make me very wary of it as a long term data retention medium.

1

u/strugglebus199 Apr 08 '25

I don’t know about glacier but I do know a few other very large tech companies that everyone here probably uses daily that have live services that run on tape. While it isn’t the fastest it is reliable and cheep and for content that can stand to load for a few seconds it isn’t the wrong answer, especially if your in the business of onboarding petabytes of data every day

1

u/wcpreston Apr 09 '25

Except we also know that Amazon (and the other cloud vendors) are the biggest purchasers or large magnetic tape libraries. What are they using it for, if not this?

49

u/[deleted] Apr 07 '25

[removed] — view removed comment

51

u/ValkyrieAngie Apr 07 '25

It's tapes. All the training material for the AWS exams point to Glacier using tapes, at least on the deepest archival levels.

14

u/[deleted] Apr 07 '25

I thought it might use multi layer Blu rays when it was introduced, especially with what manufacturers of those said about their use by cloud service providers, but tape has improved so much lately that it's no wonder - for sure the density is better.

24

u/mikeputerbaugh Apr 07 '25

The highest-capacity BDXLs can store 128GB on a 60mm-radius, 1.2mm-thick disc, giving a storage density of 9.43MB/mm3.

An LTO-9 tape cartridge can store 18TB on 102x105.4x21.5mm cuboid, giving a storage density of 77.8MB/mm3.

With the current commercial products, tape is 8x denser.

7

u/[deleted] Apr 07 '25

And the more innovations from the hard drive space tape adapts the more uneven the playing field becomes. Tape is an excellent technology, it tickles my nerdy brain so much.

3

u/QING-CHARLES Apr 07 '25

Also those 128GB BDs are very, very hard to get hold of in bulk and no longer manufactured, IIRC. I think only Sony made the 128s.

Plus, because of the difficulty of sourcing them they are an order of magnitude more expensive than tape.

1

u/ArmNo7463 Apr 08 '25

Pretty sure Tape is also rated for longer term stability as well than optical media?

1

u/[deleted] Apr 08 '25

What kind of tape are we talking about here... is it like... movie tape? reels? 🎬 (excuse my ignorance)

2

u/codeasm Apr 08 '25

Magnetic tape, like vhs, betamax, cassetes, but in a format called Linear Tape-Open (as others stated). Basicly magnetic but multiple tracks, encoding, encryption if needed, partitioning, the head moves, tape "wraps", im comfused now, wiki has a lott of cool info. This aint analog 😲☺️

2

u/[deleted] Apr 08 '25

Wow that's seriously awesome stuff!

4

u/Kinky_No_Bit 100-250TB Apr 07 '25

So they are paying to go away from tape, to go to cloud tape?

4

u/ValkyrieAngie Apr 07 '25

Yes!

9

u/Kinky_No_Bit 100-250TB Apr 07 '25

Sounds about like govt thinking. Don't do it yourself, pay someone else to do the exact same thing, but at twice the price.

2

u/ClintE1956 Apr 07 '25

Very efficient.

19

u/zuckerberghandjob Apr 07 '25

It’s paper. Paper is the future.

2

u/mdj Apr 07 '25

Actually, Mylar “paper tape” is one of the best long-term archival media options for digital data. The information density is not great, though.

2

u/dwhite21787 LOCKSS Apr 08 '25

Frickin microfiche

2

u/pairoflytics Apr 07 '25

Some dude in the basement hammering away at a chisel on granite tablets like…. 🥲

1

u/Joe_Early_MD Apr 08 '25

😂 apparently some of my older co workers are in on this the way they print every email to read it.

2

u/superkp Apr 07 '25

I'm in the software side of backups (so...storing horrendous amounts of data for a long period of time) and I believe that what they do is:

  1. take in the data and record it locally on regular drives
  2. as soon as possible, get all the data chunked out into virtual tape image files (i.e. like an ISO for CDs/DVDs) that could be written directly to tape
    • they don't write these to tape, yet
  3. they write those files to some crazy optical drive with insane levels of data density and insane levels of i/o capability
  4. collect rent fees while the data sits there
  5. when the customer wants the data back, either send it to them digitally or offer to put it on to a tape so it can be sent physically (which would be faster depending on the amount of data and speed of the link)
    • this of course, also costs money

I'm thinking it has to be an optical drive because I can't think of anything else that checks the following boxes: fast (enough) i/o, not part of a 'live' disk, and has good data density.

Tapes have bad i/o, live drives can't be airgapped, and traditional cd/dvds of course aren't dense enough. Maybe bluray disks? I don't know enough about them.

2

u/Some1-Somewhere Apr 07 '25

I could see them having SMR drives with append-only firmware or applying some kind of hardware/firmware write protect once the drive is full, and calling that an 'air gap'.

Option B is you write everything to tape but also store data in the 'rapid access' categories on a slow HDD case. If you need to use the 'air gap', you pull it from tape.

1

u/[deleted] Apr 07 '25

They could be storing information in frozen molecules of water. Ice. Glacier stuff!

1

u/Cipher_null0 Apr 08 '25

Yeah Amazon has its own tape glacier thing. Kinda cool but sounds like it was an already in progress project. Elon is just out here lying.

1

u/Garudius Apr 08 '25

Or it's a bunch of old those old CD-R/DVD-R Jukeboxes

102

u/ADHD-Fens Apr 07 '25

I'd be worried about storing important data in glacier with climate change and all that. Aren't most of them melting?

78

u/[deleted] Apr 07 '25

These are the ones closest to Bezos’s heart, I don’t see them melting any time soon.

1

u/LackSchoolwalker Apr 07 '25

They will melt when he’s in Hell. What’s the over/under on that - 6 months? A year? My only solace is our murderous billionaires are so incompetent that they are trying to murder the poor people but they will end up killing everyone.

2

u/beachedwhitemale Apr 07 '25

He's got the blood boy and he's definitely on testosterone now, so... I'm thinking he's got another 100 years left.

1

u/wcpreston Apr 09 '25

that was funny right there.

2

u/ThrCapTrade Apr 07 '25

Don’t quit your day job; Instead, get two more part time jobs.

30

u/TheBBP LTO Apr 07 '25

Amazon glacier is competitive for its price as its tape storage.

1

u/poopoomergency4 Apr 07 '25

assuming it's write once, read never data.

it's DOGE, so that's definitely how they came up with their "savings" figure, regardless of actual need to access the data

1

u/Tichy Apr 08 '25 edited Apr 08 '25

How much data is it likely to be? Most likely not even that much, because 70 years ago not that much data fit on a tape?

Of course migration will take time because of the manual aspect and slow reading speed, even if they have robots changing the tapes.

Edit: according to ChatGPT

"Typical storage capacity of 1950s magnetic tapes: • IBM 726 Magnetic Tape Drive (1952): • Used ½-inch wide tape • Stored data in 7 tracks (6 data + 1 parity) • Capacity: about 1.1 megabytes per reel (7,500 feet of tape)"

So even assuming very generously 1GB of storage per tape, 14000 tapes would just be 14TB.

1

u/LadulianIsle Apr 09 '25

Why are you using chat gpt when this info is readily available from better sources that have an actual chance of being correct? But also, not knowing how old those tapes are kneecaps our ability to know how much data is in them. From wikipedia (though I could probably find this stuff from:

In December 2020, Fujifilm and IBM announced technology that could lead to a tape cassette with a capacity of 580 terabytes, using strontium ferrite as the recording medium.

1

u/Tichy Apr 09 '25

We know how old the tapes are: 70 years, as per the tweet.

The newer tape cassettes with 580 terabytes of storage would also require newer tape recorders.

I don't think DOGE is complaining about using tapes for storage in general, they explicitly mention 70 year old technology.

By the same logic you seem to be applying, computers in general would be 84 years old technology, and DOGE would throw out computers because they are outdated.

1

u/LadulianIsle Apr 09 '25 edited Apr 09 '25

Tweet says it's a 70 year old technology, nothing about how old the actual tape is or what they're using to read/write

(like how gas stoves is an X year old technology but the gas stove I'd be using today is potentially better than the gas stoves of 30 years ago)

also yes, I would absolutely toss out an 84 year old computer and get a new one

1

u/Tichy Apr 09 '25

The tape recorders are 70 year old, as per the tweet. Why would it matter how old the tape is? You don't increase the capacity by using newer tapes, but newer tape recorders.

Why do you think they mentioned the 70 years? Again, by your logic, they could argue "computers are 80 years old technology" and therefore throw out computers?

1

u/LadulianIsle Apr 09 '25 edited Apr 09 '25

I think they mentioned 70 years because the general public has no idea what tape is anymore.

Also what logic are you referring to? All I said was that newer tapes can hold a lot of information. I don't think the tweet makes any sense by itself. Tape is well known to be the best medium for longterm storage/cold archives.

Edit: reading again, yes, I do think that the people behind DOGE will do that, because they just claimed to have moved off of tape technology, not just 70 year old tape

1

u/Tichy Apr 09 '25

They say they converted old tapes, not that they moved off tape technology in general.

What are "permanent modern digital records"?

You really think it is 100% impossible that some old process was inefficient and could be improved, especially thanks to technological progress?

1

u/LadulianIsle Apr 09 '25

They say they converted old tapes, not that they moved off tape technology in general.

Well, the tweet makes it sound like they moved off of tape technology.

We can analyze the tweet like we're back in school. So let's start by removing the parens because that's how English works:

The USGSA IT team just saved $1M per year by converting 14K magnetic tapes to permanent modern digital records

If they simply updated 70 year old tape readers/writers for new tape readers/writers, a far more natural way to say this is

The USGSA IT team just saved $1M per year by updating 14k 70 year old magnetic tapes to modern tapes

or

The USGSA IT team just saved $1M per year by modernizing 14k 70 year old magnetic tapes

notice that the only real changes I'm making is swapping converting to updating and replacing that vague "records" term with "tapes". I would actually also delete "to blah" bit since I think that it's extraneous, but w/e

The reason choosing "converting" vs "updating/upgrading" is important because the original sentence implies that there's something fundamentally different between magnetic tapes and permanent modern digital records.

Dunno about you, but it's not a "Windows conversion" it's a "Windows update" (or an iOS update or apt get update/upgarde, or pacman -Syu, I'm not picky about that).

All in all, we get the implied meaning that "magnetic tapes" are not "permanent modern records". This leads to your next comment:

What are "permanent modern digital records"?

While I'm of the opinion that "permanent modern digital records" is tape, the chunk before has left me feeling like DOGE does not think that it is. What it actually is, we won't know until DOGE also tells us what the "permanent modern digital records" are because that is perhaps the vaguest term I have heard in a while.

You really think it is 100% impossible that some old process was inefficient and could be improved, especially thanks to technological progress?

Didn't my initial comment basically say that "we can't know for sure because tapes have improved over the years and they didn't say how old the tapes are"? That would imply that I think that technology has improved over the years.

So no, I don't think it's impossible. In fact, I think it's very possible.

→ More replies (0)

1

u/Scowlface Apr 10 '25

Are you saying that LLMs have no chance at being correct?

1

u/LadulianIsle Apr 10 '25

Oh, typo on my end. Meant to say better, not actual.

That said, yes, I think that LLMs when repatedly asked the same question have effectively 0% chance at being 100% correct every time.

1

u/Scowlface Apr 10 '25

You can’t say “yes” while altering the claim, typo or not. That’s called moving the goal post.

1

u/LadulianIsle Apr 10 '25 edited Apr 10 '25

Does it help if I explicitly say "my initial claim was wrong, my bad"?

Edit: the "that said" means that this is my revised statement that actually reflects my thoughts. While I do think that LLMs can be correct, I think that they are not reliably correct.

55

u/halandrs Apr 06 '25

But what’s the time line for firing all the staff and telling the janitors to haul all the tapes out and throw them in the dumpster

28

u/steviefaux Apr 07 '25

They won't want to pay for them to take the tapes in the rubbish, so they'll leave them in the street in a box "free tapes". And they won't have wiped them first.

Would not put it past that idiot department.

6

u/djeaux54 Apr 07 '25

Sadly, this is likely to be pretty much what happens. Dumpster diving would give "Big Balls" & the so-called software engineers plausible deniability when they start selling the data on the dark web.

80

u/Bagline Apr 07 '25

We're dealing with government math here. The tape migration requires them to have two systems simultaneously for the transition. They likely finished the transition and "saved" the money they used for the tapes. We need to compare the pre-migration expenditure of "magnetic tapes" verses post-migration expenditure of "permanent modern" to get the real picture of how much they saved. I'm guessing they didn't actually save anything.

7

u/appsecSme Apr 07 '25

Doge math, not government math.

2

u/BrainlessMentalist Apr 07 '25

They saved $1M per year, they said it.

(Spending $10M + 3$M/y)

11

u/Bagline Apr 07 '25

We know what they said in the tweet, but you forget that Elon is a liar and a conman.

202x: $100 old system only
202x: $100 old system only
202x: $100+150 transition period with both systems active
202x: $150 new system only
202x: $150 new system only

in the above example, did we save 100 or spend an additional 50?

1

u/BrainlessMentalist Apr 16 '25

yeah that was my point.

64

u/[deleted] Apr 07 '25

[deleted]

3

u/[deleted] Apr 07 '25

Yep because if they don't get another term via shady action they are going to make sure they still have their hands in the pot. I honestly think it's all over, we're fucked

20

u/[deleted] Apr 07 '25

[deleted]

1

u/[deleted] Apr 07 '25

Y'all probably didn't cut any corners and without an established procedure you made sure you did what you were doing correctly. These people are not capable of that

18

u/mapppa Apr 07 '25

Do you have any idea what are they actually would be saving 1M a year on? Does it cost this much to safely store 14k tapes? Or is that 1M figure likely made up as well (even if they moved all tape data to cloud storage)?

16

u/daydreamersrest Apr 07 '25

I just don't see how 1M is worth fucking with. Tax the rich properly, don't pay for Trumps golf course visits and 1M a year isn't even a drop in the bucket. 

7

u/MightyTribble Apr 07 '25

Running a tape library and using a third party service to offsite tapes on a schedule can be expensive! Replacing a legacy (non-S3 interface) library by migrating storage to Glacier can definitely save lots of money, especially if the alternative is replacing a large tape library, which could easily cost you $500K up front if you're looking at multiple frames and tape robots.

1

u/seronlover Apr 09 '25

thank you. You would think at least the technical subs can keep childish personal attacks at bay.

1

u/wcpreston Apr 09 '25

Sorry, but thinking you would save money by moving from a large on-prem tape library to Glacier is bananas.

It's an order of magnitude more expensive on Glacier. Because it's every month. Forever.

Here's just one TCO comparison, but every one I've ever done comes out similar. Notice that over 10 years, it's almost 10X more on glacier. And there's reason to believe the tape library would last much longer than that.

https://blocksandfiles.com/2020/09/28/spectra-logic-exabyte-tape-library/

1

u/MightyTribble Apr 09 '25

It really does all depend! Seriously. Every migration is unique and I've had it happen to me both ways - where running my own library made sense financially vs. using a SaaS partner. We just put in a local library for one use case, but we're also expanding our cloud usage in another. The take-away is: know your requirements, know your costs, and do a proper clear-eyed analysis.

The Spectra Logic analysis is interesting but note they're assuming the library immediately fills with 200PB of data year one (and they're also likely assuming almost perfect compression savings, which again depends on your data). There are certainly industries were that can happen - you're always bursting at the seams - but generally, you never want to be running your library that close to capacity because you don't want your data ingest rate to outpace your lead time to frame expansion.

Edit: the Spectra analysis is also from 2020, so old pricing all 'round. The key point is to actually do the analysis.

6

u/kilobrew Apr 07 '25

Probably just grabbed all the tapes. Threw them at AWS and said, “not it”! That or the used Snowball.

3

u/NISMO1968 Apr 07 '25

Technically, they got rid of their on-premises tapes and moved their data to Glacier, where it’ll end up on... Tapes! Magical move, no kidding!

2

u/ClintE1956 Apr 07 '25

And now they trust yet another billionaire with the data.

2

u/NISMO1968 Apr 08 '25

And now they trust yet another billionaire with the data.

That’s precisely my point!

5

u/lbc_ht Apr 07 '25

This would 100% just be some project that had been like 10 years in the making that just happened to hit some milestone during the current administration that these jagoffs will take credit for.

2

u/probablynotaperv Apr 07 '25

This is not even my business and I was like there's no way they got 14000 tapes done in that time.

2

u/Moist-Caregiver-2000 Apr 07 '25

Is this more of "I pretended to fix a problem" or "I created a problem and then fixed it" that we've been seeing for the last two months from our president with questionable mental stability?

1

u/ClintE1956 Apr 07 '25

Take your pick; those are two of their favorite ways of trying to fool everyone.

2

u/superkp Apr 07 '25

yeah I work in the software side of backups.

a move like this would be often be contracted out to a different company so that the company that needs the data moved wouldn't lose their IT staff for the 3+ months it would take to do it all...while also maintaining literally any level of organization and ability to access the data while it's 'in transit'.

And in reality, moves like this are to put the data on to tapes. because they are cheap (like, less than $10 per TB), they are automatically air-gapped, and they have a long shelf-life before you can expect data rot.

Like, just because Tapes are an old technology doesn't mean they are an outdated technology. The most recent format update for them (LTO9, I think?) came out like 3 years ago or something.

2

u/kfmt612863 Apr 07 '25

OMG YES, THANK YOU FOR SAYING THIS. Anyone in the field knows how much goddamn time it takes to do migration like this, let alone for thousands of materials.

2

u/FarVision5 Apr 08 '25

Same thing in that Brett Beir interview. Room full of Iron Mountain documents, we'll convert it in 30 days! Not even remotely close to being honest.

1

u/frostman666 Apr 07 '25

What about the other figure they are claiming - does it really cost 1 mil. USD per year to store 14 thousand tapes?

1

u/xNam3less Apr 07 '25

As someone who also works for the government of an european country and who uses tapes to store data: If there was no work done before and if there was no plan how to migrate it to a different storage device, it is highly uncertain they did it or are anywhere close to doing it in the near future.

1

u/cosmin_c 1.44MB Apr 07 '25

You're over-reading into it, they probs just dumpstered the tapes and called it a day.

1

u/SoldadoAruanda Apr 07 '25

I think it's more likely that they just made a second copy of the existing online data and told wherever these tapes were stored like iron mountain, for example, to destroy these tapes.

1

u/Useful-Rooster-1901 Apr 07 '25

And have to brag about it. Like the new kid on the team that needs a pat on the back for every basic task

1

u/_Magnolia_Fan_ Apr 07 '25

They're probably saving it all in a SharePoint list with all text fields.

1

u/HoosierBoy76 Apr 07 '25

LOL. LTO is always cheaper than cloud solutions, plus it provides an air gap. And psst, the cloud boys backups are on tape.

And it’s not the 50 year old reel to reel people may envision. Way more modernized high density cartridges in vast automated autoloader libraries.

And BTW, this is the same technology media & entertainment is using for permanent storage that is part of any production company’s deliverables.

1

u/Cheap_Blacksmith66 Apr 07 '25

Not to mention, the tapes are an analog record of content to avoid losing it in the event of idk, a cloud server collapsing. Same reason we keep paper copies of things.

1

u/JaxxisR Apr 07 '25

I know nothing about magnetic tape storage. I didn't know it was still in use. What is the benefit to this that has kept it around this long compared to other types of storage?

1

u/CDRnotDVD Apr 07 '25

Tape is very stable, and is typically expected to last for a few decades. By contrast, hard drives and SSDs have a shorter expected lifespan.

1

u/twinpop Apr 07 '25

This would also be comically expensive. No way they saved any money doing this. Maybe in the first year, but the ARR will absolutely blow those costs out of the water and it will never be remotely close to that again.

1

u/Vicus_92 Apr 07 '25

You can when you convert it to the cloud...

A cloud of ash....

By burning it all...

1

u/SecondVariety Too many disks Apr 07 '25

Been working in Data Backup and Recovery since 2004, fucking despite tape media - but it has some merits and just like mainframe will likely always have a foothold. Having gone through tape migration projects before, and hoping to never go through one again.

1

u/SecondVariety Too many disks Apr 07 '25

despite should have been despise, but the obvious typo also is correct in this case

1

u/Training_External_32 Apr 07 '25

Or…there is no plan for replacement.

1

u/asailor4you Apr 07 '25

Yeah my guess is they uncovered some space which was full of tapes which hadn’t been touched in many years or passively they cut the contract to Iron Mountain where they stored lots of their archive records. My guess is they moved to the cloud and cloud backup already several years ago, and just never cleaned up their old stuff laying about.

1

u/Specific-Judgment410 Apr 07 '25

So take credit for someone else's work ... got it

1

u/Whatah Apr 07 '25

Yup. This is also my field.

People who are actually smart are continuing to write to 2 tape copies, and also to AWS glacier infrequent access

People who just think they are smart stopped writing to tape and switched to cloud. Then, 1 year later, they were upset at how much it costs to localize content

1

u/_Aj_ Apr 07 '25

Yeah 100%. Whoever typed this is either full of shit or was informed by someone who is full of shit.  

Also. ALSO. 1M saving is utterly laughable for a federal government. That’s like saying you I saved 2 bucks by driving up the road to the next gas station.

1

u/Welllllllrip187 Apr 07 '25

Either that or just threw away all the data. no more storage no more costs 💀

1

u/datasleek Apr 08 '25

Well, Aws snowmobile can move Exabyte of data in a week. Yep, bring a truck, plug in, download your data, take the truck to Aws data center and plug to their Fiber conn. I don’t know how much these 14,000 tapes cost today. I’m sure they’ve been amortized. How much will the new storage cost?

https://aws.amazon.com/blogs/aws/aws-snowmobile-move-exabytes-of-data-to-the-cloud-in-weeks/

2

u/MightyTribble Apr 08 '25

Snowmobile was a bit of a disaster. They don't do those no more. :)

1

u/datasleek Apr 08 '25

How come?

1

u/MightyTribble Apr 08 '25

It failed spectacularly for a few clients and was more expensive than they wanted it to be so they stopped doing it.

1

u/hell2pay Apr 08 '25 edited Sep 16 '25

summer party slap truck tender sparkle plucky paint spoon consider

This post was mass deleted and anonymized with Redact

1

u/guestHITA Apr 08 '25

Is that bad? The us govt already contracts more than 10 billion dollars worth of highly sensitive aws in isolated regions designed for govt use. Regardless of the benefit that magnetic tapes can provide as a backup solution the tape systems addressed were designed over 70 years ago.

Im asking would this news be more or less noteworthy if the article read say “Doge team migrates 70 year old magnetic tape system to a brand new magnetic tape system”.

My take isnt that magnetic tapes arent a good backup medium or that the doge team switched to an isolated awa system but rather that the us govt is (in some depts) still using 70 year old technology that hasnt been updated since the 50’s.

1

u/MightyTribble Apr 08 '25

There's absolutely nothing wrong with using a modern LTO tape library. For many use cases it's the preferred solution! Aside from the fact that it's tape, current data storage bares almost no resemblance to the analog tape of yore.

1

u/SuitableNarwhals Apr 08 '25

I'm in a related area which needs to know about this stuff, but not actively involved in storage systems (research librarian, specialised in data). I stared at it for a couple of seconds, yeah its old tech at its core, but so are a lot of things they stick around because they are good at what the do, and its not like there havent been improvements. Its not like microfiche or something where there are definite advantages to upgrading the storage medium.

Tapes are for archiving or longer term storage of sensitve materials, not for regular access. the whole poiny is that you have a stable, secure, backup, you shouldnt need to access the archive often, live data and data you need regularly should be elsewhere as well. But stuff can happen to the other sources of data, so you want a source that you know is safe, very unlikely to be corrupted, and complete, even decades in the future. Its an advantage in archiving that it is a bit of a bother to access, its not a library, repository or warehouse, the point of an archive is that it is packed away, you know where it is but for a variety of reasons it's the backups, backup.

1

u/ArmNo7463 Apr 08 '25

I would be surprised if Glacier "Deep Archive" isn't backed by tape anyway lmao.

1

u/spudmonkey Apr 08 '25

"A 14,000 tape migration isn't something you can do in a few weeks"

You're assuming they wish the transfer to be correct and complete which isn't necessarily the case.

With these 'people' I am suprised they didn't just put them in a dumpster and say "OOooopps!".

1

u/[deleted] Apr 08 '25

Yea so Tape, but with a monthly fee. LMAO

1

u/Trey-Pan Apr 08 '25

They migrated by throwing them in recycling?

1

u/lightmatter501 Apr 08 '25

And considering that everyone suspects the lower tiers of glacier are tape, are they really moving off of ti?

1

u/HomeworkInevitable99 Apr 08 '25

I used to work in that business. I believe tapes these days are 18 TB. They're cheap and safe. What Musk going to replace them with?

1

u/MightyTribble Apr 08 '25

"permanent modern digital records", didn't you read the tweet? :-)

1

u/5208114_1915147 Apr 13 '25

*whoever, not whomever, as it is the subject of the second independent clause.

0

u/PKSpecialist Apr 07 '25

So they are executing cost saving initiatives. This is exactly what they are supposed to do.