Because academic institutions only award promotions to people who publish in certain journals. Disseminating knowledge is not the goal, because administrators/accrediting bodies are unable/unwilling to assess the value of intellectual contributions. They outsource this to journals which are expected to ensure that research is "legitimate". I'd argue that they are very ineffective at this, but it still gives decision-makers an easy heuristic to use when they wanna see "number go up".

Tenure-track professors typically have a time limit to prove their worthiness of tenure. At the end of that limit, they apply for tenure and either get accepted or practically fired. One way to prove your value is by publishing in "high-impact" journals. There are all sorts of methods to measure the impact of a journal objectively, such as how often its articles are cited in other articles whose impact scores are also measured. These professors that are in serious debt and have spent their entire lives aiming for this position are practically prisoners trying to get out of the whole they inadvertently dug themselves into. A lot of publications are by those professors, so they are aiming for the highest possible impact scores.

Another group is made up of highly acclaimed professors. They all know each other personally and sit on the editorial boards of those high impact journals. Before they officially submit an article, they send it to their colleague that's on the board or knows someone on the board for review, make revisions, then submit for publication. This likely leads to a bias I like to call "a hook up". These professors practically own the journal, so they have no reason to start a new one.

The rest of the professors (mid-level tenured professors) would be the ones likely to create a new journal that isn't privately owned by a publishing company. Being mid-level and tenured, their only drive to excel is their personal desire to improve the field, so they're not as driven by necessity as the tenure-track or personal drive to excel as the highly acclaimed professors. So the reason it doesn't happen is the ones that would make it happened are over-shadowed by the ones that promote the currently established ones.

I think for a new one to come out, we'd need a schism among high-level professors or some major systemic change in academia that impacts the drive for publication.

I think it is because we humans are not rational.

In a free market society, over time, every seller would charge for their service as much as they can. And the service they themselves use, will in turn charge them as much as they can.

This would be an optimal system, if only humans were rational creatures. When prices for a service increase too much, we should stop using it and go for alternatives, which would create more incentives for competition to grow and prices to come down etc.

But i think, we don't think rationally. Sometimes, even when we know something is bad for us, we still do it cuz of lack of self control or other reasons.

Its mainly the prestige.

No one would care if you wrote some unknown tabloid on your resume.

I'm no expert on the topic, but Nature is an exception rather than the rule, given its history and prestige.

Academic journals were around well before the Internet. Real capital investment was required to review applications, provide editing advice, typesetting, printing, and distribution. All of those are still things, now with additional online publishing, which also has its own technology costs.

What's wild and out of whack, of course, is that peer reviewers generally aren't paid, submitters pay to get published, and readers also pay for access. Other than the relatively minimal office staff to keep things running, there's very little overhead. So why is it so damn expensive?

I think the answer is that they can get away with it. You can publish in an open source journal for free, of course, but there may or may not be quality control. Plus, it's an attention economy. If you publish in Science or Nature, you're almost certainly getting prestige that can turbo boost your career because that many people will see and likely cite your work.

And on and on it goes. I think we would pretty strong regulations to stop this system.