this post was submitted on 21 Oct 2024
20 points (95.5% liked)

Selfhosted

40394 readers
303 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
20
NFS mount disappearing (lemmy.world)
submitted 1 month ago by rehydrate5503@lemmy.world to c/selfhosted@lemmy.world
20 comments fedilink hide all child comments
 

Hi all,

I’m having an issue with an NFS mount that I use for serving podcasts through audibookshelf. The issue has been ongoing for months, and I’m not sure where the problem is and how to start debugging.

My setup:

  • Unraid with NFS share “podcasts” set up
  • Proxmox on another machine, with VM running Fedora Server 40.
  • Storage set up in Fedora to mount the “podcasts” share on boot, works fine
  • docker container on the same Fedora VM has Audiobookshelf configured with the “podcasts” mount passed through in the docker-compose file.

The issue:

NFS mount randomly drops. When it does, I need to manually mount it again, then restart the Audiobookshelf container (or reboot the VM, but I have other services).

There doesn’t seem to be any rhyme or reason to the unmount. It doesn’t coincide to any scheduled updates or spikes in activity. No issue on the Unraid side that I can see. Sometimes it drops over night, sometimes mid day. Sometimes it’s fine for a week, other times I’m remounting twice a day. What has finally forced me to seek help is the other day I was listening to a podcast, paused for 10-15 mins and couldn’t restart the episode until I went through the manual mount procedure. I checked and it was not due to the disk sinning down.

I’ve tried updating everything I could, issue persists. I only just updated to Fedora 40. It was on 38 previously and initially worked for many months without issue, then randomly started dropping the NFS mounts (I tried setting up other share mounts and same problem). Update to 39, then 40 and issue persists.

I’m not great with logs but I’m trying to learn. Nothing sticks out so far.

Does anyone have any ideas how I can debug and hopefully fix this?

all 21 comments
sorted by: hot top controversial new old
[–] SpeakinTelnet@programming.dev 8 points 1 month ago* (last edited 1 month ago) (1 children)

First thing I'd do is to look at the client (fedora) journal for anything funky happening.

'sudo systemctl status nfs-client'

Since it's random I assume you won't have any timeout in your /etc/fstab but it might be worth taking a look anyway.

Be aware that if the network drops the NFS will be disconnected and won't auto-reconnect so this could also be the issue.

I don't know if it plays well with container mounted volume, but looking at autofs could be a solution to auto-remount the share. I use it profusely for network mounted home directories.

[–] rehydrate5503@lemmy.world 1 points 1 month ago

Thanks for the detailed reply.

So the command gives me an error that nfs-client cannot be found.

The fstab just has basic default config. No timeout set.

I considered network issues, though it seems to be quite stable for other services. Not ruling it out just yet. I have a new switch coming in the next week, so will test if the issue persists when I put that in.

I will also give autofs a shot.

Thanks!

[–] schizo@forum.uncomfortable.business 3 points 1 month ago (3 children)

I'm going to have to cut up my nerd card here, but I had similar issues with NFS exports from my roll-your-own build.

After a month of troubleshooting I decided that working is better than purity so I just mounted the SMB shares instead and everything just worked going forward.

Best I can tell, NFS is just very very finnicky when it comes to hardware accessibility (drive spun down, etc.), network reliability, and is just a lot less robust than other options. I never was able to trace why NFS was the one and only thing that never seemed to work right, but at least there's other options as a workaround?

[–] phanto@lemmy.ca 4 points 1 month ago (1 children)

I did the hackiest, lamest thing back in the day... I had my client write the current date and time to a file on the share every two minutes as a Cron job... Kept it working for months! I saw it on a forum somewhere, tried it, and... Shocked Pikachu face I don't know if I ever disabled that Cron job! Haha!

[–] walden 1 points 1 month ago (1 children)

And as a bonus, presumably you have a nice file filled with historic dates and times!

[–] phanto@lemmy.ca 3 points 1 month ago

I checked, it's still there! (It doesn't append, it overwrites, so no, I just have a file with the current date and time accurate to within two minutes.)

[–] jbloggs777@discuss.tchncs.de 4 points 1 month ago (1 children)

NFSv3 (udp, stateless) was always as reliable as the network infra under Linux, I found. NFSv4 made things a bit more complicated.

You don't want any NAT / stateful connection tracking in the network path (anything that could hiccup and forget), and wired connections only for permanent storage mounts, of course.

[–] schizo@forum.uncomfortable.business 3 points 1 month ago

Yeah it was NAS -> DAC -> Switch -> endpoints and for whatever reason, for some use cases, it would just randomly hiccup and break shit.

I could never figure out what the problem was and as far as I could tell there was nothing in the network path that stopped working or flapped or whatever unless it did it so fast it didn't trigger any monitoring stuff, yet somehow still broke NFS (and only NFS).

Figured after a bit that since everything else seemed fine, and the data was being exported via like 6 other methods, that meh, I'll just use something else.

[–] rehydrate5503@lemmy.world 1 points 1 month ago

Haha don’t cut it up just yet! I’ll try some of the other options suggested here, as I’d like to learn what the issue is. The worst case I’ll try smb.

[–] AbidanYre@lemmy.world 3 points 1 month ago (1 children)

Try autofs

[–] rehydrate5503@lemmy.world 1 points 1 month ago

I’ll give that a shot, thanks!

[–] possiblylinux127@lemmy.zip 2 points 1 month ago (1 children)

How is it mounted?

[–] rehydrate5503@lemmy.world 1 points 1 month ago (1 children)

Through the Cockpit gui, which just puts it into fstab. I checked the fstab config and it just has the basic default settings.

[–] 2xsaiko@discuss.tchncs.de 2 points 1 month ago (1 children)

Never seen this before, but you can enable NFS debugging with ‘rpcdebug -m nfs -s all’ (or nfsd on the server, or rpc for the underlying protocol). It prints to dmesg.

[–] rehydrate5503@lemmy.world 2 points 1 month ago

Thank you, will try this when I have time later this week.

[–] saywhatisabigw@lemmy.world 2 points 1 month ago (1 children)

Could it be suspended due to power saving settings?

[–] rehydrate5503@lemmy.world 1 points 1 month ago

Doesn’t seem like it as far as I can tell.

[–] jonno@discuss.tchncs.de 1 points 1 month ago (1 children)

Are you loosing the mounts after a reboot? As in, are you mounting via /etc/fstab?

[–] rehydrate5503@lemmy.world 1 points 1 month ago* (last edited 1 month ago)

They are mounted via the gui, but it just puts the mount into fstab. I checked the config there and it is just the standard default options for an nfs mount.

Edit: and no, I don’t lose it on reboot. Reboot re-mounts the share correctly.

[–] Evoliddaw@lemmy.ca 1 points 1 month ago

After spending quite some time troubleshooting years ago, I chopped up the odd NFS disconnection once every month or two to random network errors. Autofs looks like it'll work but my bandaid solution was just to add a mount command to the NFS share in cron every 5 minutes. Took me 15 seconds and haven't had to look at it since.