As several people have mentioned, exact definitions of terms such as “YEC” are critical here. Ron Numbers’ study, published as the book The Creationists, documents that the version of young-earth creationism that was popularized by Whitcomb and Morris has its origins in Price’s attempts to claim that geology supported the teachings of Ellen White. Numbers comes from an SDA background, and he talked with a number of prominent people in the modern young-earth movement who accepted the accuracy of his history. There was a small tradition of attempting to support a young-earth with scientific data also in the Missouri Synod Lutheran denomination (along with some geocentrism), but apart from those, pretty much everyone had gotten the word that the scientific data supported an old earth by the late 1800’s. However, from what little information I have seen, they seem to have been inconveniently concerned with accuracy and did not approve of “The Genesis Flood”.
In turn, Ellen White got her start in the “Great Disappointment”, when the Millerite predictions of the second coming did not pan out. (SDA and some other groups claim that the prediction was true but not in the expected manner.) The Millerite movement exemplified the “I can interpret the Bible for myself without any checks or balances from the insights of others” attitude, and was thus quite popular but not very coherent.
By 1848, Michael Tuomey, in his Geology of South Carolina, refers to the perception of a conflict between the timescale of geology and the Bible as a thing of the past. The “Scriptural Geologists” do not seem to have had much influence at the academic level; the degree to which the general public was swayed is much less documented [on all issues, not just this one]. I have seen, but not relocated, a book from the mid to later 1800’s in the US that took its timeline straight from Ussher, so I don’t know if it had Adventist or other sectarian influence.
Many modern sources apply false dichotomies and assume that any reference to a relatively young earth or global flood means the same teaching as modern YEC, but in reality there is no connection, for example, between a religious claim that people have been reincarnating through cycles of millions of years and a geologic assessment of earth’s age. (Ironically, Carl Baugh, one of the looser screws of the creation science movement, is featured in a Hare Krishna video that pushes humans reincarnating for millions of years.) Likewise, the catastrophist geologists of the late 1700’s to early 1800’s did not try to cram all geology into Noah’s Flood. They recognized that most geologic layers formed slowly, under ordinary conditions, but thought that occasional major catastrophes accounted for major changes. Noah’s flood was seen as probably describing the most recent of those events (with varying views on how accurate a description it was).
The approach of Ussher, Newton, and others in the 1500’s and 1600’s was quite different from modern young-earth views. For the early to medieval church, the position tended to be that there was no historical evidence for vast time (in contrast to the mythical “histories” of many cultures), while being quite ready to take a fairly figurative approach to Genesis 1. Beginning in the 1500’s, several scholars tried to put together an overall history of the world. The Bible was their most important source for ancient history, but any other sources available [in that day, in Europe, mostly Greek and Roman] were also used, along with intelligent guesswork about the parts not documented. Ussher’s famous 4004 BC was widely thought to be a well-argued estimate, but the dates calculated ranged from higher 3000’s BC (e.g., Isaac Newton) to over 5000 BC. Modern conventions for reporting precision did not yet exist, so the fact that Ussher said 4004 does not mean that he would have objected to any of the other estimates. Note also that 4004 is exactly 4000 years before 4 BC, widely accepted as the year of Jesus’ birth. This is not a coincidence; after all, the Bible doesn’t cover all the years, and there are textual issues requiring scholarly judgement where there are numbers in the Bible, so Ussher had to make some decisions. He was influenced by the tradition of chiliasm, going back to the 2nd century AD. Chiliasm took the verses (Ps 90, quoted in 2nd Peter) about “a thousand years are like a day” to mean that earth history would last for exactly 6 1000 year periods. If Jesus’s birth starts a new 1000 year interval, then creation must be exactly something thousand years before. (Of course, creation in 4004 BC by chiliasm implies the second coming was in 1997, and we’re all left behind - chiliasm has a problem these days beyond highly doubtful exegesis.)
In the mid-1600’s, people began to add geologic evidence to the types of historical data that could be used to put together a history of the earth. (Martin Rudwick’s books are excellent sources on the development of geology; The Earth’s Deep History is a good place to start.) By the late 1600’s, some began to think that geology was pointing to a rather old earth; by the 1770’s it was unambiguously clear to anyone in touch with academic geology that the earth was old. Ironically, one person not closely in touch with academic geology was William Smith, whose work had taught him that different geologic layers occurred in a consistent sequence across England and Wales. As Steve Gould pointed out in a review, Simon Winchester’s claim in The Map that Changed the World that Smith was boldly risking jail by challenging the church in saying that the earth was old is untrue. In reality, it was a couple of minister friends of Smith who pointed out to him that his findings indicated an old earth. Smith was big on identifying layers and had not been much concerned about how they formed. Similarly bogus but biased in the opposite direction from Winchester, Price falsely claims that Smith made up his stratigraphy in support of old earth views. The general theological response to these developments was " so there’s a big chunk of pre-human earth history that the Bible skipped over as theologically unimportant. That’s interesting."
Modern young-earth creationism takes the modernistic approach of insisting that Genesis 1 has to be treated as a modern scientific account, and is not a continuation of historical views.