For those two specifically I'd suggest to host one yourself. You then are 100% in control over blocking users/instances and the latency is super short because you're the only one on the server. That's what I'm doing.
Lemmy
Everything about Lemmy; bugs, gripes, praises, and advocacy.
For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.
Have you considered hosting your own instance instead? Seems like that would solve the issue.
That's what I do. I'm basically immune to any defederation drama, as I can choose who to federate or defederate with myself, and I'm immune to any downtime (other than the time I accidentally nuked my MBR the morning before I left on an all-day trip and couldn't access my instance all day because of it, but let's not talk about that).
Most people pick an instance by collecting a list of instances that are physically close to them, and then reviewing the /instances page to see how big the block list is. Lemmyverse.net can tell you how many users are on those blocked instances.
Something to consider... Lemmy runs fast with no data in it. Latency issue is tied to how much data they have stored in PostgreSQL. If they aren't holding full copies of all the remote communities, sure it is faster, but your searches and All aren't going to turn up much.
I don't think latency is relevant in an application like lemmy. If you can't get posts from the server it's not latency but the server being overloaded.
If you have the ability then self hosting is the best option
Otherwise I think instances with less users and content will be quicker, but then the all feed will probably be smaller.
I will also recommend my own instance endlesstalk.org. It is very small(10 users) and the server is located in Germany. Hopefully that is close to where you live, so the latency is low.