We are banning them specifically because they gained that advantage by going through puberty as a gender they aren't competing as. And none of those other women competitors had the ability to do that.
That is the difference. And I think that's a fine reason to ban someone from competing (AT A HIGH LEVEL, NOT CHILDREN'S SPORTS).
Can I point to data that men are taller than women? Stronger than women? Yes... yes I can. Or you can just google it. It's a pretty easily verifiable fact.