Math skills can occasionally be useful, but I don’t see it as a dealbreaker.
The good thing about being good with math is that it usually means you’re a good problem solver, and problem solving is an important skill for programming. But the reverse isn’t necessarily true. You can be good at problem solving but still be bad at math.
I would say if you’re struggling with the programming courses, then maybe look somewhere else. Otherwise, go ahead.