It sounds like you are talking about neutral drift, which per my understanding is neutral because the allele is neutral ecologically, meaning that it does not affect the way it way it interacts with the environment materially one way or the other. If something is neutral, it is neutral, and neutral means that it does not affect the relationship to the ECOLOGY.