I have had the same thought many times in this forum.
Actually, numbers are rarely the problem except when it comes to the various census’ in the OT. And they aren’t really wrong. They are mis-translated and computed incorrectly. It’s a shame that the Hebrew speakers never corrected it, since Numbers 1:2-3 tells how they are to be done, but they do it differently right away. I should probably give the solution here, but I want to see if anyone would look for the answer on their own.