Technically, We Don't Have That Many Ancestors

If you had really good genealogy records, your family tree could theoretically be huge, with infinite fractal branches. That doesn't happen because the population of the world doesn't get bigger in the past. But even if it were true, you don't carry genetic information from all your distant ancestors. You always inherit 50% from each parent, but you could carry as little as 0% from one grandparent and as much as 50% from another. Those are extremes, and there's an exception for a genetic male, who always gets the Y chromosome from his father's father. But whatever percentage you get from each grandparent, that slice gets smaller with each subsequent generation. The upshot is that after a few generations, you will have no genetic link at all to some of your forbearers. Which is good, because they may have been more closely related to their spouse than is comfortable to think about. This video from MinuteEarth is barely over three minutes; the rest is an ad.