Or, maybe, modern humans and Neandertals didn't mix after all?

These are exciting times for human origins research. Right on the heels of the new Sankararaman et al. (2012) paper that not only postulates Neandertal introgression, but aims to date it as well, there is apparently a new paper in PNAS that argues that African population structure, not Neandertal introgression is responsible for the observed pattern of greater Eurasian-Neandertal than African-Neandertal similarity.

I will post the abstract and comments on the paper when I see them. I was first tipped off about this research several months ago by an off-hand comment in a New Scientist article on the topic of human origins. Ewen Callaway reports:

Both papers were presented at conferences more than a year ago. Their publication today raises the question of whether this debate would have been more timely if it had occurred on pre-print servers like ArXiv.org and not at specialist conferences and behind the walls of peer review.

It would have been great indeed if both new papers had been available for comment a long-time ago. This seems like a good example of how the ritual of journal publication may be introducing an unnecessary lag in scientific progress.

The issue is of Neandertal introgression vs. African population structure is a little personal to me. After all, I was in a very small minority of people who were critical of the original Green et al. paper, titling my review of it appropriately as Tales of Neanderthal admixture in modern Eurasians. To their credit, Green et al. discussed the possibility of ancient African structure in their paper, but science reporters and the news media jumped on the story of "sex with Neandertals". I ended up adopting a position similar to what Mike Hammer seems to have described in a recent talk; both African structure and Neandertal introgression could account for the D-statistic pattern, and we didn't know exactly where we were along the continuum.

It is strange indeed that just as I have come around to a grudging acceptance of at least some Neandertal admixture, mainly on the basis of physical anthropology, rather than genetics, a new paper comes out that seems to argue in favor of my original "African structure" thesis forcefully. 

There seems to be little consensus on the topic. A recent paper by Yang et al. (2012) proposed to reject the African structure model, although I was not very happy with their model -- see my comments on that paper. Another article of Ghirotto et al. (2011) denied any Neandertal admixture at all, on the basis of the absence of mtDNA of Neandertal origin in modern humans. Some have argued that negative selection could explain this absence, but that argument carries little force: modern human DNA has shown the ability to adapt to very different energetic requirements, spanning every possible climate zone. It is difficult to imagine that Neandertal mtDNA would have been wiped off completely by natural selection. As I note today in a comment written before I was aware of the newest skeptical paper:

Yes, the lack of Neandertal mtDNA in modern humans is the best argument against very extensive admixture.

As of this writing, the new Eriksson and Manica paper is not yet live, but they seem to be taking the argument to autosomal genetics, and showing how the D-statistic pattern could be due to differential affiliation of Out-of-North Africans/Sub-Saharan Africans with Neandertals. Looking forward to getting a copy and reading it. But, there is some information in the press release:

New research raises questions about the theory that modern humans and Neanderthals at some point interbred, known as hybridisation. The findings of a study by researchers at the University of Cambridge suggests that common ancestry, not hybridisation, better explains the average 1-4 per cent DNA that those of European and Asian descent (Eurasians) share with Neanderthals. It was published today, 13 August, in the journal PNAS.
In the last two years, a number of studies have suggested that modern humans and Neanderthals had at some point interbred. Genetic evidence shows that on average Eurasians and Neanderthals share between 1-4 per cent of their DNA. In contrast, Africans have almost none of the Neanderthal genome. The previous studies concluded that these differences could be explained by hybridisation which occurred as modern humans exited Africa and bred with the Neanderthals who already inhabited Europe. 

However, a new study funded by the BBSRC and the Leverhulme Trust has provided an alternative explanation for the genetic similarities. The scientists found that common ancestry, without any hybridisation, explains the genetic similarities between Neanderthals and modern humans. In other words, the DNA that Neanderthal and modern humans share can all be attributed to their common origin, without any recent influx of Neanderthal DNA into modern humans. 

Dr Andrea Manica, from the University of Cambridge, who led the study said: "Our work shows clearly that the patterns currently seen in the Neanderthal genome are not exceptional, and are in line with our expectations of what we would see without hybridisation. So, if any hybridisation happened - it's difficult to conclusively prove it never happened - then it would have been minimal and much less than what people are claiming now."

To top it all off there is yet another new paper by Mendez et al. (2012) which proposes adaptive introgression of a Neandertal related lineage that has reached high frequency in Papua-New Guinea. As I mentioned before, Mike Hammer -who is a co-author- seems agnostic on whether African structure or Neandertal introgression contribute more to the D-statistic differences between Eurasians and Africans, but, as he mentioned in his talk, and in this paper, he does believe in some level of Neandertal intermixture.

If you are perplexed about what really happened, rest assured that you are in good company. I'm as perplexed as ever myself. At least, now that the two camps have their papers out (Aug 13, 2012, what a day for Neandertal research!) the conversation can begin in earnest and in public.

Stay tuned; the abstract and comments on the paper will appear in this space.

UPDATE I: While I, like everybody else, wait for this paper to appear on the PNAS website, let me summarize the case against Neandertal admixture:

1. The absence of Neandertal mtDNA or Y-chromosomes in the modern gene pool
2. The fact that Neandertals were a West Eurasian distributed species and yet there is no major excess of Neandertal variation in modern West Eurasians
3. The possibility that the Neandertal genome from Vindija may reflect an individual that had both the opportunity (spatio-temporal co-existence) and does look like it experienced gene flow from modern humans.
4. The almost certain existence of African population structure which might be able to generate the observed deviations of the D-statistic for Eurasians relative to Africans.

The current paper seems to address issue #4. Hence, I will talk about this when the paper comes online. But, let's briefly discuss the other three points:

~3% Neandertal admixture in modern humans is non-trivial. As mentioned above, Ghirotto et al. use this observation to argue that we ought to have sampled some Neandertal mtDNA in modern human populations. There are a couple reasons why we might not have:

• natural selection on mtDNA; but it is difficult to see why Neandertal mtDNA would be selected against. The high mutation rate of mtDNA, and the evidence that energetic adaptations occurred in modern human DNA in people living from the tropics to the arctic argue against selection. Moreover, Neandertals too lived in different climate zones, from the Levant to within the arctic circle. 
• male-biased gene flow; if Neandertal gene flow took the form of Neandertal men cornering Ms. Modern Human while Mr. Modern Human was out hunting, then we would expect no mtDNA from Neandertals to have survived. But, although we don't have any Neandertal Y-chromosomes to compare against, the case seems strongest that the modern human Y-chromosome phylogeny is of non-Neandertal origin.

So, while the evidence from uniparental markers does not exclude Neandertal admixture, it seems to argue against it being substantial.

The second idea is that since Neandertals were a West Eurasian species, then West Eurasians had much more opportunity to mix with them than either Africans or East Eurasians. But, we see that East Eurasians don't seem to lack in the Neandertal affiliation department.

This issue was addressed in a paper by Currat and Excoffier, who cleverly exploited the fact that the Neandertal range is now known to have stretched a good way towards the east to "up" the opportunity that migrating humans would have encountered Neandertals as they made their way that way too. But, note, that this is not consistent with a coastal migration theory, and also, East Eurasians would have admixed with a different set of Neandertals that would have been different from European Neandertals like Vindija, and would still be expected to match Vindija less at least for that reason.

The third idea, is that what we are seeing as Neandertal admixture into modern humans may in fact have been modern human admixture into Neandertals. Now, through an analysis of divergence times, it is possible to really claim that some haplotypes where Eurasians match Neandertals share a very old common ancestor and are thus consistent with archaic introgression: but note that the identification of such haplotypes is arguing by example: there is no quantification as to what fraction of the ~3%, such regions with strong evidence for admixture constitute.

A different idea, from the original Green et al. paper is that modern-to-Neandertal gene flow can be excluded on the basis of the following argument: "If Eurasians transmitted some of their DNA to Neandertals, then Neandertals would appear more strongly related to Africans with a strong Eurasian affiliation (like Yoruba) than to those without (like San)".

It is a good argument, which, however, hinges on the assumption that the admixture occurred by people who were already Eurasian-like. This may not be true for two reasons: (i) these were early Eurasians who did not have time to develop a Eurasian-like genetic profile, and (ii) it is possible that the source of the major human expansion (associated with Y-haplogroup CT, mtDNA haplogroup L3, and the 70kya date) may have originated not in Africa, but in Arabia, from humans who had colonized it prior to 100kya.

Moreover, the argument of Neandertal-to-modern introgression seems to be against experience, since it is the successful, expanding populations that usually affect the less successful, contracting ones, rather than the opposite (as we've recently seen in the case of the Khoe-San). Of course, gene flow from modern humans-to-Neandertals may have indeed "modernized" them to the point where they would have been absorbed into expanding humanity. But, the point is that if we are sampling Neandertals from a stage (late Europe at Vindija cave) not devoid of modern admixture, then some or all of the observed ~3% signal may reflect modern-to-Neandertal gene flow.

I will update again, when the paper appears.

UPDATE II (Aug 14): The paper has now appeared.

From the paper:

In our model, Africa and Eurasia are represented as a linear string of identical populations (demes), each of which can exchange migrants only with its adjacent neighbors. The simulation starts at 500 kya, with all demes occupied by the common ancestor of Neanderthals and anatomically modern humans (Fig. 1A). The connection between the African and Eurasian branches was truncated at 320 kya to represent the speciation of Neanderthals outside Africa (Fig. 1B). The populations in the African branch, which eventually turns into modern humans, were allowed to reexpand, eventually colonizing a set of Eurasian demes parallel to the ones occupied by Neanderthals (but without any admixture; Fig. 1C). As in Gronau et al. (33), the timings of the transition to modern demography, and of the subsequent expansion out of Africa, were free parameters in the model. We explored dates ranging from 100 to 200 kya for the transition to modern demography, and from 40 to 80 kya for the expansion.

The divergence time for modern human and Neandertal populations, as well as the Out-of-Africa event are conditioned on, but new human-chimp divergence dates have pushed back the modern human-Neandertal split (if there was one) to 400-800kya, and, probably the Out-of-Africa event to before the 100kya mark. The current paper is conditioned on a 6My human-chimp divergence, so its demographic scenario will probably have to be stretched.

The main idea of this paper is that within the ancestral population of both modern humans and Neandertals (presumably Homo heidelbergensis), demes who were geographically close would be closer related. Hence, West Eurasians would be more closely related to North Africans than South Africans.

There are additional reasons to think so, which are not mentioned in the paper: the Sahara which has acted intermittently as a barrier to gene flow to the south of the continent, as well as the tropical rainforest of the equatorial regions which would represent a significant barrier to gene flow. Indeed, movement across Africa would probably have been impeded due to its long axis across many degrees of latitude, representing different climatic regions. By contrast, North Africa and the Levant occupy a similar latitudinal zone and have experienced similar climatic conditions. Moreover, there is now evidence for the preservation of archaic features in late crania from Nigeria and Congo, as well as genetic evidence for the same for a variety of Sub-Saharan African populations.

With that said, the authors are able to recreate the D-statistic difference between Eurasians and Africans with respect to Neandertal affiliation. This is not driven by Neandertal admixture in Eurasians, but rather from the fact that within the larger ancestral population, Neandertals and North Africans evolved from ancestors that were more closely related to each other than to humans who occupied Sub-Saharan Africa.

From the paper:

When we used our demographic scenarios (described above) to generate synthetic genomes for both Neanderthal and modern human populations, we found that the predicted levels of D for the different pairwise comparisons fitted very well the observed patterns. For Africans vs. Europeans and Africans vs. Asians, the distributions of predicted D values are tightly centered around the corresponding observed values; for European vs. Asians, we recover the expected negative values, even though the predicted distribution is centered around values slightly closer to zero than the observed one (Fig. 2). The same conclusion holds for any choice of African and OOA populations (Fig. S3), which is agreement with the finding that the Aborigines and Eurasian genomes show similar overlap with the Neanderthal genome (25). 

The ability of the paper to largely recreate the D statistic difference between Eurasians and Africans is its most attractive property. It manages to do so without invoking a very special admixture event in the Near East, or route of modern humans into East Asia, or extent of admixture.

Note that the admixture model requires a single or few admixture events in the Near East, as modern humans exited Africa: if West Eurasians had continued to admix with their Neandertal neighbors, then they would end up substantially more Neandertal than East Asians (they are not). East Eurasians would have to follow a very specific northern path to Asia, to give them "equal opportunity" to admix with Central Asian Neandertal groups. Moreover, the number of interbreeding events relative to the modern human population would have to be "just right" for the ~3% Neandertal admixture. The model presented by Eriksson and Manica is simpler, and so much more attractive.

Of course, it could be argued that Sankararaman et al. (2012) have already proven that modern human-Neandertal admixture both took place and was recent. But, putting the issue of direction of gene flow aside for the moment, that paper limited itself to a subset of SNPs (those with minor allele frequency less than 10%), while its methods (based on LD, which decays over time) are inherently limited to uncovering recent gene flow rather than older gene flow (where the LD signal has been wiped out by recombination) and/or common ancestry.

How much of the ~3% signal can be accounted for by the signal of recent interbreeding? How much of it is consistent with common ancestry, rather than recent Neandertal-Modern admixture? 

Eriksson and Manica are able to recover most of it with no admixture. While Sankararaman et al. are making a reasonable case that some of it may be recent, because low-MAF Neandertal-derived alleles are still in some linkage disequilibrium. But, how much?

In the end, I can agree with the final paragraph of the new paper:

In conclusion, we urge caution in inferring recent admixture from geographic differences in genetic overlap between ancient hominins and modern-day populations. Though we do not claim that anatomically modern humans never admixed with other hominins, our results imply that current evidence for such admixture events is inconclusive at best. Future tests, to be convincing, will need to show that the genetic patterns used to invoke hybridization cannot be explained by population structure, for which there is both genetic (28–31) and archaeological evidence (27, 28). A key step toward such evidence will be the addition of ancient genomes of modern humans and other hominins from multiple locations, because they will allow reconstruction of ancient population structure and its possible effects on the genetic patterns in modern humans.

I have a feeling that the conversation has only just begun.

PNAS doi: 10.1073/pnas.1200567109

Effect of ancient population structure on the degree of polymorphism shared between modern human populations and ancient hominins

Anders Eriksson and Andrea Manica

Recent comparisons between anatomically modern humans and ancient genomes of other hominins have raised the tantalizing, and hotly debated, possibility of hybridization. Although several tests of hybridization have been devised, they all rely on the degree to which different modern populations share genetic polymorphisms with the ancient genomes of other hominins. However, spatial population structure is expected to generate genetic patterns similar to those that might be attributed to hybridization. To investigate this problem, we take Neanderthals as a case study, and build a spatially explicit model of the shared history of anatomically modern humans and this hominin. We show that the excess polymorphism shared between Eurasians and Neanderthals is compatible with scenarios in which no hybridization occurred, and is strongly linked to the strength of population structure in ancient populations. Thus, we recommend caution in inferring admixture from geographic patterns of shared polymorphisms, and argue that future attempts to investigate ancient hybridization between humans and other hominins should explicitly account for population structure.

Link