Yes, it is unwarranted speculation. Consider again the paper you cited:
One can agree that LGT will likely not interfere with a robust phylogeny for primates that indeed recreates ancient splittings of their populations
EDIT: I gave a more detailed response in another thread, where you made the same argument while citing the same paper. Rather than repeat myself, I simply link to my previous response.
Grace and peace,
Chris Falter