this post was submitted on 14 Aug 2024
57 points (98.3% liked)

Selfhosted

40329 readers
624 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

I'm planning to upgrade my home server and need some advice on storage options. I already researched quite a bit and heard so many conflicting opinions and tips.

Sadly, even after asking all those questions to GPT and browsing countless forums, I'm really not sure what I should go with, and need some personal recommendations, experience and tips.

What I want:

  • More storage: Right now, I only have 1 TB, which is just the internal SSD of my thin client. This amount of storage will not be sufficient for personal data anymore in the near future, and it already isn't for my movies.
  • Splitting the data: I want to use the internal drive just for stuff that actively runs, like the host OS, configs and Docker container data. Those are in one single directory and will be backed up manually from time to time. It wouldn't matter that much if they get lost, since I didn't customize a lot and mostly used defaults for everything. The personal data (documents, photos, logs), backups and movies should each get their own partition (or subvolume).
  • Encryption at rest: The personal data are right now unencrypted, and I feel very unwell with that. They definitely have to get encrypted at rest, so that somebody with physical access can't just plug it in and see all my sensitive data in plain text. Backups are already encrypted as is. And for the rest, like movies, astrophotography projects (huge files!), and the host, I absolutely don't care.
  • Extendability: If I notice one day that my storage gets insufficient, I want to just plug in another drive and extend my current space.
  • Redundancy: At least for the most important data, a hard drive failure shouldn't be a mess. I back them up regularly on an external drive (with Borg) and sometimes manually by just copying the files plainly. Right now, the problem is, if the single drive fails, which it might do, it would be very annoying. I wouldn't loose many data, since they all get synced to my devices and I then can just copy them, and I have two offline backups available just in case, but it would still cause quite some headache.

So, here are my questions:

Best option for adding storage

My Mini-PC sadly has no additional ports for more SATA drives. The only option I see is using the 4 USB 3.0 ports on the backside. And there are a few possibilities how I can do that.

  • Option 1: just using "classic" external drives. With that, I could add up to 4 drives. One major drawback of that is the price. Disks with more than 1 TB are very expensive, so I would hit my limit with 4 TB if I don't want to spend a fortune. Also, I'm not sure about the energy supply and stability of the connection. If one drive fails, a big portion of my data is lost too. I can also transform them into a RAID setup, which would half my already limited storage space even more, and then the space wouldn't be enough or extendable anymore. And of course, it would just look very janky too...
  • Option 2: The same as above, but with USB hubs. That way, I theoretically could add up to 20 drives, when I have a hub with 5 slots. That would of course be a very suboptimal thing, because I highly doubt that the single USB port can handle the power demand and information speed/ integrity with that huge amount of drives. In reality, I of course wouldn't add that many. Maybe only two per hub, and then set them up as RAID. That would make 4x2 drives.
  • And, option 3: Buy a specialized hard drive bay, like this simpler one with two slots or this more expensive one for 4 drives and active cooling. With those, I can just plug in up to 4 drives per bay, and then connect those via USB. The drives get their power not from the USB port, but from their own power supply. Also, they get cooled (either passively via the case if I choose one that fits only two drives, or actively with a cooling fan) and there are options to enable different storage modes, for example a built in RAID. That would make the setup quite a bit simpler, but I'm not sure if I would loose control of formatting the drives how I want them to be if they get managed by the bay.

What would you recommend?

File system

File system type

I will probably choose BTRFS if that is possible. I thought about ZFS too, but since it isn't included by default, and BTRFS does everything I want, I will probably go with BTRFS. It would give me the option for subvolumes, some of which are encrypted, compression, deduplication, RAID or merged drives, and seems to be future proof without any disadvantages. My host OS (Debian) is installed with Ext4, because it came like that by default, and is fine for me. But for storage, something else than Ext4 seems to be the superior choice.

Encryption

Encrypting drives with LUKS is relatively straight forward. Are there simple ways to do that, other than via CLI? Do Cockpit, CasaOS or other web interface tools support that? Something similar to Gnomes' Disk Utility for example, where setting that up is just a few clicks.

How can I unlock the drives automatically when certain conditions are met, e.g. when the server is connected to the home network, or by adding a TPM chip onto the mainboard? Unlocking the volume every time the server reboots would be very annoying.

That of course would compromize the security aspect quite a bit, but it doesn't have to be super secure. Just secure enough, that if a malicious actor (e.g. angry Ex-GF, police raid, someone breaking in, etc.) can't see all my photos by just plugging the drive in. For my threat model, everything that takes more than 15 minutes of guessing unlock options is more than enough. I could even choose "Password123" as password, and that would be fine.

I just want the files to be accessible after unlocking, so the "Encrypt after upload"-option that Nextcloud has or Cryptomator for example isn't an option.

RAID?

From what I've read, RAID is a quite controversial topic. Some people say it's not necessary, and some say that one should never live without. I know that it is NOT a backup solution and does not replace proper 3-2-1-backups.

Thing is, I can't assess how often drives fail, and I would loose half of my available storage, which is limited, especially by $$$. For now, I would only add 1 or max 2 TB, and then upgrade later when I really need it. And for that, having to pay 150€ or 400€ is a huge difference.

you are viewing a single comment's thread
view the rest of the comments
[–] InnerScientist@lemmy.world 5 points 3 months ago

Drives connected to usb have an unstable connection in my experience, this is very annoying and gets worse with hubs.


RAIDs reduce the time a system is offline and reduce data loss, if a drive fails and you can afford to wait for the new disk and the backup to restore, and have regular backups that ensure no important data gets lost (though remember the data added between backups may be lost) then you don't need a RAID.

I don't use RAIDs cause if my disk fails then I can stomach the 2-4 days it takes to buy a new one and restore the backup

Very important: use S.M.A.R.T and a filesystem with checksums to make sure you're not backing up corrupted data and know to get a new one


For encryption at rest you may want to look at clevis and tang, though you need a server in your home network for this to work. The client (with clevis) then decrypts the disk at boot if it can reach the server (tang). The server can't decrypt the data without the client secret and the client can't decrypt it without the server public key.

Don't know what your server could be though, maybe a router with custom firmware?


You should also look into cloud storage/rclone, that way you can automate your backups more and reduce the need for manual intervention.

I use rclone and restic to automatically backup my servers daily which takes a few seconds most of the time due to them being incremental backups.