I like to start with the obvious: Romans 5 isn’t the beginning of the letter. By that point Paul has already established how all kinds of people sin and need rescue. Paul doesn’t rest his theology or his thinking about sin, death and salvation on Adam.
Romans 1 retells the Eden story without Adam, Eve, a serpent, a prohibited tree or magic fruit. Yet it still speaks of humans (collectively) having some knowledge of and connection to God from their creation, but breaking that fellowship in a misguided pursuit of wisdom that leaves them confused about creatures and Creator. Certainly there are questions about God’s “wrath” and “giving over”: it’s not an easy passage. But once again, like Eden, both accounts give a theological take on the history of humankind while also confronting us with our own story. Paul just does it with less symbolism and more rhetoric.
In the second half of Romans 5, Paul restates some points he’s already made, now using the figure of Adam parallelled to Christ. And much like his muddle over which Corinthians he baptized, here he seems to recognize the complexity that overwhelms his parallel just after he commits it to paper. Before he finishes his first sentence, he starts clarifying all the ways the two are not equal and opposite. The passage ends up focusing more on differences than similarities, and even the similarities have to blur things to make the two compatible (such as collapsing Jesus’ lifelong obedience into “one man’s act of righteousness,” as @Jay313 noted up-thread).
Since Adam functions as a sermon illustration, not as the evidential basis for Paul’s thinking, the weakness of the parallel doesn’t undermine it. It gives us a mental picture and exposes how much greater Christ is than Adam. Focusing on some of the phrasing, such as “Adam to Moses,” could suggest Adam was a man who lived at a certain time. But when Adam is read as both a symbol for each human and the first humans (much like the “those”/“they” of Romans 1:18–23), “Adam to Moses” can still convey from the beginning of humanity to the giving of the law. A symbolic reading can still make sense of that phrase, and it makes much better sense of how Paul explains death’s dominion by simultaneously pointing to Adam and saying “because all sin.”