You have fallen for the ultimate trick: wanting a "big" community.
You only get that from big, centralized social networks that want to maximize the amount of content you are fed, because it maximizes your ad views, and their profits.
Embrace the smallness. Lemmy still has room to grow, and having lot of different options for communication that aren't all owned by billionaires is a good thing. The fact that it isn't constantly trying to earn your attention is a feature, not a bug.