The main idea is simple, and I will rephrase it as follows: offspring of a big man and small woman will tend to have bigger heads relative to the size of the woman's birth canal. On the other hand, offspring of a small man and big woman will not have that problem.
We have some information about differences between Neandertals and modern humans. The former were shorter and more "lateral" skeletally, while UP modern humans appear to have been more linear and taller. Headwise, modern humans had more globular head shapes, while Neandertals more linear ones, with no major differences in brain size between the two species.
If the above are correct, then male Neandertal-female modern human pairings would have a potential problem. Birth is a complex process, but at the end of the day, the most important factor is probably whether the diameter of the head can "fit" in the birth canal: the more it does not fit, the more likely it would seem that a mishap for both mother and offspring would occur.
Combine elongated Neandertal heads with narrow modern human pelves, and you have a potential problem. I am not 100% sure that modern humans and Neandertals differed in pelvis shape, although it would be a reasonable consequence of their overall build, but the same pattern would occur if they did not, simply on account of their different head shapes.
An additional factor involves sexual dimorphism, since male babies tend to be larger than female ones, and so any problems associated with "parental mismatch" might be particularly troublesome for male births.
So, all in all, we have 4 different cases:
- Male H. n + Female H. s. => Male hybrid
- Male H. n + Female H. s. => Female hybrid
- Male H. s + Female H. n. => Male hybrid
- Male H. s + Female H. n. => Female hybrid
It would appear, on the basis of the preceding discussion, that 1-2 would be more troublesome than 3-4, and 1 most troublesome of all. On the other hand, 4 seems to be the most advantageous case.
The most interesting thing about modern-Neandertal admixture is that it seems to have left no traces in uniparental markers, and, indeed, the lack of mtDNA lineages of Neandertal origin has been used to argue against the plausibility of estimated Neandertal admixture percentages.
If my reasoning is correct, then case #4 is particularly worrying, since female hybrids with Neandertal mtDNA ought to be the most easy to bear, and would also be the ones who would contribute Neandertal mtDNA in a mixed population.
On the other hand, case #1 would explain the lack of Neandertal Y-chromosomes, since crossings between male Neandertals and female modern humans that produce male offspring might be particular troublesome, and they would also be the ones to introduce Neandertal Y-chromosomes in the population.
Of course, we don't know enough about the dynamics of the admixture process; it might be possible that other factors influence the abundance of the four cases, both biological and cultural. For example, if modern humans had a behavioral advantage, then modern males may contribute most admixture, and this would make the worrying case #4 even more difficult to explain. On the other hand, how did the admixture take place? bride-stealing vs. rape would result in potential offspring being raised in different groups (father's vs. mother's), and there may also have been unknown cultural taboos involving admixture and offpsring produced from it.
In any case, this brief excursus may be useful for anyone thinking of writing some palaeo-fiction set in the Upper Paleolithic, and I'd love to hear from people who have data at hand that might be pertinent to the above discussion.