It's worth noting that a number of languages comparable C in use case and performance (including its predecessors COBOL, Fortran and ALGOL) start indexing at 1 just fine because they have proper array types and don't make heavy use of pointers poking and peeking wherever they like (eg by using references instead).

Decoupling indices from memory offsets doesn't get in the way of performance and actually often allows better optimization because the compiler knows you aren't sharing pointers between arrays or some other shenanigans; see Fortran, the GOAT of fast array processing. This also allows improved type safety and thus memory safety; see index types in Ada/SPARK and the fact that it's the only 'legacy' language that's gotten comprehensive compile-time memory safety analysis (though the latter is moreso from the less willy-nilly pointers in general)