Given how human babies are already at the maximum size a human female can deliver, I suspect you may be on to something. Other species don't have such traumatic births, but human babies barely fit through the birth canal. Neanderthal babies were probably much larger. In particular, their skulls. If a Neanderthal male impregnated a human female, I wouldn't be surprised if it killed her. Conversely, a Neanderthal female probably had an easier time delivering a hybrid baby.