Yes, the genealogy is the biggest issue for a symbolic reading of Adam. For me, it’s not enough to overturn the evidence for a symbolic Adam, partly because of Paul’s words against getting too caught up in genealogies, partly because of how genealogies work elsewhere in Scripture, partly because of the fluidity in this particular genealogy, and partly because my view of Scripture doesn’t expect the genealogy to record God’s knowledge.
Dealing with the last first, there’s no reason to expect Moses or some later author to have accurate information about ancestors many generations in their past. If it is accurate, it could only be through God revealing information they didn’t know. From looking at the whole Bible, I don’t think God has chosen to overcome the blind spots and gaps in the human writers to convey scientific or historical information. Since the Holy Spirit didn’t prevent Paul from writing a confused account of whom he baptized (1 Cor. 1:14–16), I have no problem with a genealogy similarly limited to a human author’s understanding.
Next, ancient genealogies were about who made someone who they are, not who provided their genes. That could include mythical creatures, gods and legendary characters – all who were either inspirational to the individual or who they saw as having similar characteristics to themselves. It could include people within their community who helped to nurture them – not just parents, but also uncles or more distant relatives or neighbours. The further a genealogy goes into the mists of forgotten time, the more these legendary elements dominate.
If one allows that God can inspire any genre of literature, even ancient genealogy, then the same thing may be happening in Genesis 5. It may preserve the barest trace of many individuals remembered by Israelites at the time it was written.
If that’s the case, for most individuals we have only a name, some ages, and a connection to another name. And different manuscript lines of Genesis preserve different numbers. (We dug deep into the purpose of the numbers in a recent thread.) And the names in Seth’s line are eerily similar to Cain’s line a chapter earlier: Cain/Kenan, Enoch/Enoch, Irad/Jared, Mehujael/Mahalalel, Methushael/Methuselah, Lamech/Lamech. It looks like both lines are preserving some of the same memories. Both Lamechs are associated with three sevens; one lives to 777 and the other boasts “If Cain is avenged 7 times, Lamech is 77 times.” This kind of fast-and-loose use of numbers and names better fits a human collection of murky legends than divinely-revealed historical details.
As for the Adam of Genesis 5:3–5, I don’t know if this also preserves the memory of some distant individual or whether it simply links the chain of names to Adam/Humanity as described in the previous verses and chapters. Adam was not a common Hebrew name (much as Humanity isn’t that popular in English), so perhaps a real person, name unknown, became fossilized in the text as Seth’s father.
If we don’t force the genealogy to transcend either the writer’s knowledge or the ground rules of ancient genealogy, there’s no reason to see Genesis 5 as confirming a literal Adam. Portraying the human race going back to a man named Adam is no different to how it portrays the nation of Egypt going back to a man named Egypt (Gen. 10:6) and musicians going back to man named Jubal (Gen. 4:21). Our clear distinction between literally descending from someone and symbolically being in someone fades in ancient times when children were thought to actually exist in their father long prior to conception. Noah curses Canaan because that whole nation was “in” Ham when he did whatever he did, just as Levi and all the other Israelites were “in” Abraham when he paid tithes to Melchizedek. That’s not how we look at things, but it was a common ancient perspective that helps explain several texts that otherwise seem mysterious to us.
So I see a close look at the genealogy pointing away from getting historical detail from them, even as it has obviously been shaped to link Adam/Humanity with Noah and all the nations that descend from his sons. Seeing Adam as both Humanity and a first man made sense in an ancient view where men alone carry their entire lineage of children within their loins (hence the harsh punishment towards any who would harm a male organ, and hence the perception that infertility was entirely the woman’s problem, and hence the focus on men in genealogies), but I don’t see reason to hold on to that today. Adam is us; the ancient view that traced every kind back to a first male is best left behind.