Why should I believe those who claim that the Nephilim of Genesis were "fallen angels", especially when....?

You are correct. More likely than not, the passage referred to the line of Seth, the once godly and righteous line, intermarrying with the corrupt line of Cain.