this post was submitted on 30 Dec 2023
56 points (100.0% liked)

Lemmy

12568 readers
17 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 4 years ago
MODERATORS
 

I've found the following work-around works pretty well. If you host an instance that's currently on 0.19.0 or 0.19.1, consider implementing this.

There are two bugs that this helps with:

Work-around:
Create cronjobs that restart the Lemmy container every 6 hours (but not at midnight). The following example is used for a Debian system running Lemmy in Docker.

Type crontab -e into the terminal Add something like the following:

~~0 1 * * * docker container restart lemmy-lemmy-1
0 7 * * * docker container restart lemmy-lemmy-1
0 13 * * * docker container restart lemmy-lemmy-1
0 19 * * * docker container restart lemmy-lemmy-1~~

3 1-23/6 * * * docker container restart lemmy-postgres-1 && sleep 60 && docker container restart lemmy-lemmy-1

By restarting the container every 6 hours, outbound federation continues to work. There may still be some delays, but everything gets cleared up regularly.

By telling it what time to restart (0100, 0700, 1300, and 1900 as opposed to "every 6 hours"), it avoids restarting at midnight. This avoids the second bug.

My instance has been doing this for enough days where I'm confident that it's working. You can check your federation status here. Note that it's normal for there to be 0 up-to-date instances and a lot of lagging instances. As long as they sometimes turn "up to date", then everything is getting caught up.

all 18 comments
sorted by: hot top controversial new old
[–] RedKrieg@lemmy.redkrieg.com 16 points 10 months ago* (last edited 10 months ago) (1 children)

I like to do things just off the top of the hour, since top of the hour is when many maintenance crons run. If you're running a modern cron daemon, you can rewrite that as:

3 1-23/6 * * * docker container restart lemmy-lemmy-1

https://crontab.guru/#3_1/6___*

[–] walden 6 points 10 months ago* (last edited 10 months ago)

I figured someone more familiar with crontab would chime in. Thanks for the recommendation!

Edit: I've updated my cronjob to https://crontab.guru/#7_1-23/6___* and I'll keep an eye out to make sure it works on Debian 12.

[–] BlueEther@no.lastname.nz 4 points 10 months ago

The only thing that doesn't seem to federate on my 0.19.1 instance is new subscriptions, but in saying that it looks like the BE restarted an hour ago

[–] rikudou@lemmings.world 2 points 10 months ago (1 children)

Is "testing" a swear word to Lemmy "devs"? Such bugs after so many RCs? (Though it seems that for Lemmy "devs" RC actually means "random crap").

[–] walden 1 points 10 months ago (1 children)

Extra frustrating for you, I imagine, since you write a lot of API stuff for Lemmy.

[–] rikudou@lemmings.world 2 points 10 months ago (1 children)

Yeah, still have to fix some stuff, though I've taken a break during the holidays.

[–] walden 1 points 10 months ago (1 children)

That's great, hope you're having some fun with family and friends. Happy New Year.

[–] rikudou@lemmings.world 2 points 10 months ago

Happy New Year as well!

[–] SeeJayEmm@lemmy.procrastinati.org 2 points 10 months ago (1 children)

I didn't know how this tool is supposed to work but it always fails for me but my instance appears to be working ok.

https://phiresky.github.io/lemmy-federation-state

[–] walden 2 points 10 months ago

Interesting. Yeah I tried too and it doesn't work for lemmy.procrastinati.org

[–] bjoern_tantau@swg-empire.de 1 points 10 months ago* (last edited 10 months ago) (1 children)

Funnily enough, since I've started to regularly check if my comments made it out I haven't had any problems. Schrödinger's bug.

Edit: Then again, after checking with this post, although my comment made it out just fine, some comments haven't arrived on my instance. I'll try to restart.

[–] walden 2 points 10 months ago

Comments from other instances should arrive to yours no problem, as the bug only seems to affect outgoing stuff. If the other instance is running 0.19.0+, then of course those comments might be stuck.