OK, so lets have an outrageous thought, that the so called development of Homo Sapiens was not as linear as ToE would assert.
Neandethals were very tall, right? What if they directly sired a branch of humanity, while, say Cromagnon sired a different branch. Meaning that just a Big cats are not linear, neither is humanity. Giants would not longer be from Angelic stock and the Nephilim would be just another folk tale or myth
Aw shucks!
It would ruin all this provarication and wrangling to make the Biblical view of the Nephilim real. (Not to mention change the linear direction of the development of humanity)
I had better duck now
Richard