It goes back farther than that: for the Jews, March 25 was the day Adam was created, and the day the Messiah was supposed to be conceived. That just carried over into the church since Jews were still becoming Christians in large numbers for the first four centuries and some. March 25 was the first day of spring, and therefore the day of new life, so any event that involved new life was tied to March 25 – I even heard that some said that the parting of the sea was on March 25!
That’s not even close to the reason. The idea of celebrating Christ’s birth didn’t gain traction until the fourth century for a simple reason: the church was still heavily Jewish, and the Jews back then didn’t celebrate days of birth, considering it a pagan practice. But by the fourth century the church was becoming more Gentile, a shift that was amplified by the Edict of Milan where the emperor Constantine declared Christianity licit, which allowed Christians to be more publicly open and removed a barrier to people joining. So while there were some localized observance of Nativity celebrations, there was nothing official. But Gentiles celebrated birthdays (though usually not until a person’s third or even fourth year since so many babies died early back then), so the more Gentiles who became Christians, the greater the impetus to celebrate the birth of Christ.
The mother of Constantine, empress Helena, played a part with her emphasis on relics and other Gentile concepts including celebrations of the birthdays of both gods and important ancestors (who were often officially elevated to the status of gods) and other historical figures, two varieties of birthday celebrations which converged in Christmas along with the practice of entire households celebrating the birthdays of the living head of the family, which given Christ’s Resurrection also converged in the Nativity. For Gentile Christians, then, the idea of celebrating the Savior’s birth was a natural expression of the reasons they already held for celebrating birthdays; He was the most important ‘ancestor’, He was God, and He was the living head of the family.
Thus the first public and religious celebration of the Christ Mass happened in Rome near the end of Constantine’s life – or rather the first recorded celebration; it’s likely that lesser bishops than that of Rome took note of Christ’s birth in the liturgy prior to that but there were no noticeable celebrations. But once the bishop of Rome held such a celebration, many bishops in the west would have followed suit. The question that had to be answered first was, “When was He born?”, but the church had a built-in answer to that already: thanks to Jewish tradition, the conception of Jesus was reckoned as being on March 25, so they added nine months and got December 25.
And for many people under Rome’s rule who might have been debating whether to join this rising religion, this new celebration could well have tipped the balance: it showed that ideas they were accustomed to were honored by the church, and the proposed date fell near things they were accustomed to celebrate, so it was easy to join the church and keep their celebrations just with a different focus, The choice of date, then, was an inadvertent but convenient de facto appeal to many not-quite believers along with recent converts, an accidental accommodation.
It also became popular for another reason: most church festivals at that point occurred in spring or summer, and having a winter festival served to balance the calendar somewhat. And it didn’t hurt that the concepts behind several pagan celebrations resonated with who Christ was!