this post was submitted on 05 Oct 2024
145 points (88.0% liked)
Technology
58531 readers
4461 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
That's honestly intense. I would be terrified of having that much data in one place
While not hard drives, at $dayjob we bought a new server out with 16 x 64TB nvme drives. We don't even need the speed of nvme for this machines roll. It was the density that was most appealing.
It feels crazy having a petabytes of storage (albeit with some lost to raid redundancy). Is this what it was like working in tech up till the mid 00s with significant jumps just turning up?
This is exactly what it was like, except you didn't need it as much.
Storage used to cover how much a person needed and maybe 2-8x more, then datasets shot upwards with audio/mp3, then video, then again with Ai.
The size increase in hard drives around that time was insane. Compared to the mid-90's which was just a decade ago, hard drives capacities increased around 100 times. On average, drive capacities were doubling every year.
Then things slowed down. In the past 20 years, we've maybe increased the capacities 30-40 times for hard drives.
Flash memory, on the other hand, is a different story. Sometime around 2002-3 or so I paid something like $45 for my first USB flash drive - a whole 128MB of storage. Today I can buy one that's literally 1000 times larger, for around a third of that price. (I still have that drive, and it still works too!)
Well hell, it's not like it's your money.
a petabye of ssds is probably cheaper than a petabye of hdds when you account for rack costs, electricity costs, and maintenance.
Not a problem I've ever faced before, admittedly
I guess you're expected to set those up in a RAID 5 or 6 (or similar) setup to have redundancy in case of failure.
Rebuilding after a failure would be a few days of squeaky bum time though.
Absolutely not. At those densities, the write speed isn't high enough to trust to RAID 5 or 6, particularly on a new system with drives from the same manufacturing batch (which may fail around the same time). You'd be looking at a RAID 10 or even a variant with more than two drives per mirror. Regardless of RAID level, at least a couple should be reserved as hot spares as well.
EDIT: RAID 10 doesn't necessarily rebuild any faster than RAID 5/6, but the write speed is relevant because it determines the total time to rebuild. That determines the likelihood that another drive in the array fails (more likely during a rebuild due to added drive stress). with RAID 10, it's less likely the drive will be in the same span. Regardless, it's always worth restating that RAID is no substitute for your 3-2-1 backups.
Yeah I have 6 14tb drives in raid 10, I'll get 2 more if i need it.
At raid6, rebuilds are 4.2 roentgens, not great but they're not horrible. Keep old backups.but the data isn't irreplaceable.
Raid5 is suicide if you care about your data.