Neandertal admixture vs. Ancient African structure

The case for Neandertal admixture has been made on the basis of the D-statistic. This is a genomewide statistic which quantifies how two modern populations differ from each other at sites where Neandertals carry the derived allele. If population A tends to agree with Neandertals more than population B does, then this is consistent with Neandertal admixture in population A. It is also consistent with admixture in Neandertals from modern humans, and with ancient population structure whereby some humans carry admixture from before the common ancestor of modern humans and Neandertals.

A new paper goes beyond the D-statistic with the goal of trying to decide between two different models: Neandertal admixture in Eurasians or ancient African population structure. It does so by simulating the site frequency spectrum under the two models and comparing the result with the observed shape of the spectrum. From the paper:

Durand et al. (2011) suggested that the ancient structure model results in more variation in gene tree depth than the recent admixture model. Greater variance in tree depth would alter the frequency spectrum but not the D-statistic. Here, we show that the sfs appropriately conditioned can distinguish between recent admixture and ancient structure because it is particularly sensitive to episodes of recent admixture.

This is how Durand et al. (2011) explained this:

However, it is known that ancestral population subdivision result in a higher than expected variance in coalescence times (Wall, Lohmueller & Plagnol 2009). Therefore, ancestral subdivision is likely to result in more variation in gene tree depth when using several samples from the extant population. This in turn will affect the site frequency spectrum of the extant population (Harpending et al. 1998). Designing a statistic to distinguish between the two scenarios will require using more than one sample per population.

Here is how the authors describe their two models, pictured on the left:

[admixture] In this model, we assume that there was a single episode of admixture at time tGF in the past (t = 0 being the present) from Neanderthals to non-Africans after the migration of humans out of Africa (Figure 1a). With probability f, a non-African lineage was derived from a Neanderthal lineage. The parameter f represents the fraction of the non-African genomes of Neanderthal origin. We define the divergence time of non-African and African populations as tH > tGF. We denote by tN > tH the divergence time of Neanderthals and the population ancestral to modern humans.

...

[structure] We assume in this model that the population ancestral to modern humans and Neanderthals was divided into two randomly mating subpopulations (Figure 1b). We assume that subpopulations exchanged migrants symmetrically at rate m per generation. At time T in the past, subpopulations merged into one panmictic population. A similar model was proposed in Slatkin and Pollack (2008). Green et al. (2010) noted that this model could explain the extra similarity of Neanderthals with non-Africans, and Durand et al. (2011) showed that D-statistics could not distinguish between ancient structure and recent admixture for plausible demographic parameter ranges.

I don't quite understand the ancient population structure model as presented in this paper. It appears that time T (when the populations merged into one panmictic population) precedes the split between modern humans and Neandertals.

This is definitely not what I've had in mind when I've spoken about African population structure before; rather, I think that within the genus Homo there were at least two subpopulations (let's call them A and B) that had long-term reproductive isolation. B gave rise to H. sapiens and H. neanderthalensis but some subpopulations of H. sapiens admixed with A long after the sapiens/Neandertal split. The basis for that model is dual: (i) the late persistence of archaic humans in Africa that no longer exist there and hence may have been absorbed by African sapiens and (ii) the greater genetic variation in extant Africans which is consistent with either an Out-of-Africa bottleneck or an Into-Africa event followed by admixture.

In any case, it is worth giving the investigated parameters of the authors' two models:

In the recent admixture model, tH was set to 4,500 generations ago (112.5 kya, Li and Durbin 2011), and tN was set to 12,000 generations ago (300 kya, Green et al. 2010). The tGF parameter was set to 2,000 generations (50 kya), and f was chosen to be 0.05. 

In the ancient structure model, T was varied between 12,000 to 32,000 generations ago, in steps of 2,000 generations. The intensity of ancient migration m was set to 4Nm = {0,1,…,10}. The non-African and Neanderthal populations split 12,000 generations ago (tN), and tH, the population split time between YRI and the non-African populations, was 4,500 generations ago.

I have a few observations:

• The population split between YRI and non-Africans is set at 112,500 years ago. This is most certainly inaccurate, since Yoruba possess mtDNA haplogroup L3 in common with Eurasians (age ~70ky), and they are also near completely within Y-haplogroup DE-YAP (which links them with Eurasians within a ~50ky timeframe). A split between YRI and non-Africans at 112,500 years ago is virtually impossible. An apparent split at that time is plausible once we take into account that this number is inflated by admixture in Africa, and the YRI group may be partially descended partially from a West Eurasian-like population admixing with an older African substratum (early AMH or even older Africans).
• Archaic admixture in Africans has been little studied and there are several new studies in the works which will investigate it. For the time being, an initial admixture study, as well as the late persistence of archaic hominins in Africa appears to be consistent with a very late admixture event, rather than with a long-standing process of admixture between structured African populations.
• In the recent admixture model, the admixture is taken as both instantaneous (at tGF) and recent (50ky). The former assumption seems justified, but it is hard to believe in a model that would have Eurasians deviating from Africans 112,500 years ago and then waiting for 62,500 years to admix with Neandertals. Neandertals began to be neighbors of modern humans from before 100 thousand years in the Levant, and pre-50ky Neandertals such as Kebara are already modern-like, as is Vindija; the latter's deviation towards modern humans is critical because it highlights the very real possibility that the ancient Neandertal genome as currently reconstructed may be in fact partially modern human.

I really don't see why we the two models should differ qualitatively: a postulated abrupt (instantaneous) admixture event with Neandertals (despite the fact that modern humans and Neandertals were clearly neighbors for close to 100 thousand years) vs. a long-standing process of admixture in Africa.

To conclude, the new study is to be applauded for trying to evaluate different schemes for reaching the same observed D-statistics, but I would not throw the African structure hypothesis out the window quite yet.

Mol Biol Evol (2012) doi: 10.1093/molbev/mss117

Ancient structure in Africa unlikely to explain Neanderthal and non-African genetic similarity

Melinda A. Yang et al.

Neanderthals have been shown to share more genetic variants with present-day non-Africans than Africans. Recent admixture between Neanderthals and modern humans outside of Africa was proposed as the most parsimonious explanation for this observation. However, the hypothesis of ancient population structure within Africa could not be ruled out as an alternative explanation. We use simulations to test whether the site frequency spectrum, conditioned on a derived Neanderthal and an ancestral Yoruba (African) nucleotide (the doubly conditioned site-frequency spectrum, dcfs), can distinguish between models that assume recent admixture or ancient population structure. We compare the simulations to the dcfs calculated from data taken from populations of European, Chinese, and Japanese descent in the Complete Genomics Diversity Panel. Simulations under a variety of plausible demographic parameters were used to examine the shape of the dcfs for both models. The observed shape of the dcfs cannot be explained by any set of parameter values used in the simulations of the ancient structure model. The dcfs simulations for the recent admixture model provide a good fit to the observed dcfs for non-Africans, thereby supporting the hypothesis that recent admixture with Neanderthals accounts for the greater similarity of Neanderthals to non-Africans than Africans.

Link