Not meant as an authoritative or absolute assessment, but this viewpoint may help you come to terms with the "fragmented" nature of the fediverse and understand why it's the reason we are here (or at least why you would want to be). Lemmy and the fediverse is not "the ultimate aggregation platform" or the new old reddit in any singular or unified sense.
Digression: The fediverse, generally speaking, is an infrastructure; a network of communities across various social platforms, including link-aggregators, microblogging sites, video-streaming, forums, media-sharing, and website comment sections all over the internet. Lemmy is an animal sharing an ecosystem (the fediverse) with other animals (such as mastodon), but it isn't the fediverse, and not all animals think alike -- each platform has its own notion of what it means to interact and how you should interact -- but different species are still able to interact to various degrees.
In a similar vein, Lemmy instances are autonomous communities with their own values, purposes, interests, and, more concretely, moderation policies. An instance may choose to defederate with another instance for the same reason a "normal" website may not want to give space to content from just any other website. I like to think of federation in terms of "freedom from [lock-in/harassment/toxicity/ads/sensory assault/information attacks/tired debates/sea-lioning/etc]" while retaining the ability to continue interacting consensually with others on the network (as opposed to "if you don't like it on [centralised platform] you can leave", which usually means "you're free to leave this city and go build your own village in the Andean mountains").
Each instance separately may fill the role of link aggregator - but for members of that community (accounts on that instance) first and foremost, with that community’s values and moderation policies reflected in the perceived quality of content. The ability for an instance to federate with other instances with compatible policies is the benefit here, not an imperative or some duty an instance has towards le fediverse collective. Thus, it may actually help if you view an instance as the community, with its “communities” as its topics, or subforums if you will.
We need to remember that these sites are inherently social: the fediverse is not meant as a resilient information exchange protocol, but as a means for social groups to organise organically rather than be funnelled into the same environmentally controlled silo before inevitably being processed and sold. Part of that process (the former, not the latter) involves disagreement, defederation, migration, formation of new instances serving new niches, causes or ideals, and occasionally bad enough groups will get ostracised because they're intolerable (regardless of whether they think they're playing by the rules or not, because -- unlike on corporate social media -- on the fediverse you're allowed to simply not tolerate intolerable people).
This isn't to invalidate frustrations that arise from, for example, large instances defederating from yours when you haven't done anything wrong; there is the separate problem of a lack of portable identities, which would fix a number of inconveniences if it was a thing (to mention a few: deciding on where to sign up or settle down; migrating when a server goes down/to shit; having more than one interest/association but being forced to choose one community; even just following links to other instances). Luckily, there are reasons to think it can be done - it just hasn't been done yet.
Another popular frustration is that you often want to subscribe to a common topic but some of these are hosted by different instances, and this becomes a bit messy and unmanageable without any clear benefit, especially when the instances are not diverse enough to provide any unique flavour to the content posted. This is another fixable issue, and I suspect we'll see it implemented relatively soon.
I don't know if this is helpful for you but thanks for listening to my TED-talk.
Edit: compile error, expected ')', found EOF. Edit 2: added more paragraphs and some fluff because this became my activity today.
I keep hearing people talk about defederation between major instances being completely normal and to just get over it... but it's really not at all normal. It's temporarily just barely tolerable because the fediverse is (nearly) completely irrelevant and no one cares whether it actually works.
The fediverse didn't invent federated protocols in 2015, they go way back... and the behavior successful federated networks display is completely clear:
Federated networks have a long history, and the clearest lesson from it is that the value of interconnectness is greater than the value of whatever you're arguing with another major instance about. All of these examples DID go through early periods where federation was weaponized or used as a negotiating lever. But once they became important enough to matter, the cost of major peers defederating became high enough to be unthinkable... and something no other proposed benefit could justify. If the fediverse doesn't self-destruct first, it too will eventually become too important to mess with and instances that play games with federation will be made irrelevant. The idea of defederation being "part of the protocol" that is normal for major instances to deploy against other major instances is a temporary one made possible by the low cost of severely damaging the network.
Every example you listed involves a corporate entity trying to make money by providing access to the federated network in question. A connection that is 1 to 1 at that, not one to many.
The fediverse isn't that. At the moment, it's communities connecting with other communities generally without commercial interest. As a result communities sidelined by society thrive here. And that includes bigots and vulnerable minorities.
If it ever becomes corporate, the hate groups will be permanently defederated in one form or another by the commercial entities or the vulnerable minorities will move on when it becomes impossible to exist without bigots
And then one or both groups of folk will find new niches and the pattern will repeat
Internet, email, and the xmpp example cited in a response to my comment all started out this way as well. The ones that grew to have commercial significance did so because they were extraordinarily useful, as the fediverse has the potential to be.
Nothing else you said seems incompatible or in disagreement with what I said. Defederation is both an essential tool to combat abuse AND ALSO a tool whose cost is existentially large when deployed between load-bearing providers in an ecosystem. In spite of the common rhetoric, the cavalier nature with which it is deployed today is not at all a natural end state of federated systems or a result of their technical nature. It's a phase that federated ecosystems go through on the way to either destroying themselves through infighting or growing to become sufficiently useful that people are forced to find other ways to cooperate and resolve disputes.
This probably does have implications for marginalized communities, and it's possible that federation (and defederation) provides a refuge for these communities. But it does so at the cost of relevance and reach.