Is Neandertal mtDNA actually so distinctive? One basic assumption of our simulations is that Neandertal mtDNA lineages are distinct relative to the variation found in living humans. The initial studies by the Neandertal mtDNA sequencing groups found that living human and Neandertal sequences differed substantially from each other (Krings et al. 1997, 2000; Ovchinnikov et al. 2000; Schmitz et al. 2002). A reanalysis by Gutiérrez, Sánchez, and Marín (2002) did conclude that Neandertal mtDNA was within the range of living human variation, but this result may have been an artifact of bootstrapping sequences to generate neighbor-joining trees with a rapidly evolving genetic system such as mtDNA, in which the ratio of noisy (i.e., highly inconsistent or homoplastic) to stable sites is high. This approach may create many neighbor-joining trees whose topology is determined primarily by inconsistent sites, so by chance some living humans and Neandertals sometimes group together to the exclusion of other living humans (A. Knight, personal communication). In contrast, Caramelli et al. (2003) found no overlap between modern human and Neandertal mtDNA sequences using multidimensional scaling, and along with living human sequences this analysis included possible (because contamination cannot be ruled out) ancient modern human sequences from Paglicci 12 and 25, Italy (P12 and P25), and Lake Mungo 3, Australia (LM3). Knight (2003) used a phylogenetic approach to Neandertal mtDNA and found four highly consistent synapomorphies that unite Neandertals to the exclusion of all living humans and four highly consistent synapomorphies that unite all living humans to the exclusion of Neandertals (a total of eight highly consistent sites that define the two clades, including an insertion, which is a very rare event). These eight sites are consistent across thousands of living humans and are known to have low mutation rates. Additionally, for preserved sites, the mtDNA sequences from P12 and P25, LM3, and the nuclear mitochondrial insert (thought to have diverged shortly before the oldest coalescence of living human mtDNA sequences [Zischler et al. 1995]) have none of the Neandertal synapomorphies while possessing all of the diagnostic derived sites of living human sequences (A. Knight, personal communication). Recently, mtDNA fragments were extracted from four additional Neandertal and five early modern human fossils that had similar biomolecular preservation. All the Neandertals and none of the ancient modern humans yielded sequences similar to previous Neandertal sequences (Cooper, Drummond, and Willerslev 2004, Serre et al. 2004). It appears, therefore, that the assumption that Neandertal mtDNA is distinct from that of living humans is reasonable.
The second question is quite interesting. The fact that modern human mtDNA is similar to each other and differs from Neandertal mtDNA can be explained in either of two ways: (i) modern humans and Neandertals were two separate lineages evolving on their own, or (ii) modern human mtDNA is the product of selection, i.e., modern humans possess only those mtDNA types that have survived a selective sweep. The implication of this is that formerly, humans had much different mtDNA types which no longer exist because they were culled by natural selection.
The "rapid population growth" model suggests that the relative homogeneity of human mtDNA is the result of the fact that until a few tens of thousands years ago, humans formed a small population, with relatively little mtDNA diversity. Hence, our current mtDNA diversity is due to the fact that even though we now number in the billions, we are still ultimately descended from a small group of individuals in the not-so-distant past. This model contrasts with the "selective sweep" model, in which our genetic diversity is a remnant of past genetic diversity over a long period of time, the remnant that has survived the selective sweep. The authors review the evidence, and suggest that the "rapid population growth" model explains the data better than the alternative:
The significance of ancient Neandertal mtDNA for resolving the fate of Neandertals increases greatly when considered in light of models for modern human origins derived from archaeology. On the basis of mtDNA, if Neandertals survived late in Europe, their per generation contribution to early modern human populations must have been fairly small (<0.2%).Archaeology tends to support the rapid population growth model (Klein et al. 2004, Stiner et al. 1999), as does living human mtDNA (Excoffier and Schneider 1999, Ingman et al. 2000). Other genetic regions are more equivocal about the timing and magnitude of population growth (Harpending and Rogers 2000, Ptak and Przeworski 2002, Wall and Przeworski 2000), but recent studies of SNPs and microsatellites appear to be reaching a consensus consistent with the results for mtDNA (Marth et al. 2004, Zhivotovsky, Rosenberg, and Feldman 2003). Our results stress the importance of fully integrating archaeological, fossil, and genetic evidence in investigations of modern human origins.
This earlier post on human-Neanderthal admixture, estimated at less than 0.1% is also of interest.
CURRENT ANTHROPOLOGY Volume 46, Number 4, August-October 2005
Ancient DNA, Late Neandertal Survival, and Modern-Human Neandertal Genetic Admixture
Timothy D. Weaver and Charles C. Roseman