Heresy warning: This post does not question the existence of God or Jesus. It does challenge Biblical inerrancy. If that will ruin your Christmas season, please skip this post.
If you've heard the different versions of the Christmas story enough times, you generally aren't listening very critically when you hear them again. For instance, many people believe that somewhere in the Bible it says there were animals present, or that the birth of Jesus was in the winter, or there were three wise men, when in reality that's just the way we've traditionally imagined it.
However, if you look at Matthew and Luke side by side, some contradictions start to appear right off. Other problems show up when you study the historical background. I'd like to list a few of these problems briefly; if you're interested, you can research further on your own to see what responses apologists have made as well as the responses to those responses.
1. The genealogies.
This is well-known but not really a huge problem. Matthew and Luke list different names in their genealogies of Jesus, even while parts are the same. There are two very simple explanations for this: first, you can be descended from the same person by two different lineages; and second, genealogies of the time didn't distinguish between fathers and grandfathers, so it's not a huge deal that whole generations are skipped.
Of course, you can also believe that the genealogies were never intended to be factual in the first place. It's quite possible that neither Jesus nor Mary knew his exact genealogy (heck, I only know like three generations of mine) and so the evangelists made up something they thought was fitting, for symbolic and typological reasons. It doesn't really bother me that they would do this; I don't think it necessitates that the more obviously intended-as-factual parts of the Gospels are equally untrustworthy.
2. The reason for traveling to Bethlehem
Both Matthew and Luke are intending to deal with a problem in Jesus' credentials -- the Messiah was supposed to come from Bethlehem, but everyone knew he came from Nazareth. The Gospels report that Jesus' hearers thought no one good could ever come from there. So both evangelists had to explain how, despite being well-known as a Nazarene, Jesus really was born in Bethlehem.
Matthew seems to assume that Mary and Joseph lived in Bethlehem all along; at any rate, they are living in a house when the Magi arrive. The reason they have to leave is that Herod is seeking to kill Jesus. They go to Egypt first (to fulfill another prophecy) and later avoid Judea for fear of Herod's son, so they choose to settle in Nazareth. In Luke's account, they live in Nazareth at the beginning, going to Bethlehem only because of the census and then returning home after visiting the temple in Jerusalem, when Jesus was forty days old, to offer him to the Lord.
These accounts are usually mixed together and conflated, so that we can come up with a single narrative. Yet it doesn't seem they can really be reconciled. If the family was only in Bethlehem for the census -- for something under forty days -- why would they have a house where the Magi could visit them? If, as in Matthew's narrative, Herod is seeking for Jesus so that they have to sneak out of Judea, why would it make any sense for the family to go to Jerusalem of all places -- where Herod is -- and allow Simeon and Anna to speak publicly about Jesus to everyone there? The shepherds, as well, are said in Luke's gospel to have spread the news everywhere. It seems it would be difficult for Herod to miss finding him if Bethlehem is overrun with shepherds telling the whole story.
The usual answer is that the Magi show up well after Jesus' birth, and after his dedication at the Temple as well. After all, Herod later tries to kill all infants under two, so Jesus could be any age two or younger when the Magi finally arrive. However, if he were under two and over 40 days, Luke says he'd be back in Nazareth already, and Matthew seems to imply pretty clearly that the choice to go to Nazareth at all was only made later.
3. Dating problems
This one's simple but extremely problematic: Herod died in 4 BC, while Quirinius did not become governor of Syria until 6 AD. These dates are considered pretty solid by historians from multiple sources. So Matthew's account comes ten years earlier than Luke's, rather than at the same time or two years later. Any reconciling of them requires at least small inaccuracies to be admitted in the text -- that the Herod referred to is Herod's son Archelaus, or that Quirinius wasn't really governor yet but still referred to as a governor because the author knew he eventually became governor. The first can't be right, because Matthew specifically mentions Herod dying and being succeeded by Archelaus, and the second is directly contradictory to what Luke actually says ("this is the first census that took place when Quirinius was governor of Syria").
4. Historical context
Unlike much of the gospels, the infancy narratives include information that one might expect to see attested elsewhere. For instance, if a census had been conducted of the whole Roman Empire, we would expect to have heard about it. Yet there is no evidence that any such thing happened. There is a record that there was a regional census in 6 AD, under Quirnius, and it makes perfect sense of course that this is what Luke would have been thinking of. But there is no reason why people would go to the places of their birth for a census; this wasn't the custom for a Roman census so far as we know.
The other big issue is the slaughter of the innocents. If all the babies in an entire town actually got slaughtered, that seems like the sort of thing that would be remembered and recorded. Herod's biographers were big on recording the awful things he did -- would they really have ignored a mass infanticide? In fact, it seems likely that the slaughter of the innocents would have sparked a rebellion. Wouldn't you protest at least a little if soldiers came to your door and demanded to kill your youngest child? But not a word appears about this event anywhere else. Personally I find this comforting, because it's one of the most horrible stories in the Bible, but it does of course cast some doubt on the rest of the infancy narrative if it didn't really happen.
Of course another thing that seems odd is that even within the Gospels, none of the infancy events are remembered by anyone later. When Jesus' neighbors ask, "Isn't this the son of Joseph?" no one says, "Ah, but remember, he wasn't born here, we don't know where he comes from." No one ever brings up that the entire region of Bethlehem heard the story of the shepherds, or that everyone who visited the temple in Jerusalem heard Simeon and Anna talking about him. This isn't impossible or anything, but it sure is unexpected.
Most of the narrative of the Gospels is more or less unrefutable. No one else wrote about Jesus in any detail, and few historical facts are mentioned, so you can't easily fact-check anything in there. And when the gospels contradict one another, that isn't really a problem most of the time because you can always assume the same event happened twice with slight differences -- that Jesus fed five thousand one time, and four thousand another time -- if you're interested in preserving exact numerical accuracy.
The infancy narratives are one of the two places (the other being the Passion narratives) where we can clearly say we're dealing with the same time period and so some things must happen only once. And the differences between the only two accounts we have are so drastic as to cast doubt on the whole story. Is it possible that the evangelists invented their infancy narratives for symbolic and typological reasons? Matthew, of course, as is his habit, is mostly interested in demonstrating that Jesus fulfilled Old Testament prophecies; while Luke wants to talk about the virgin birth and include beautiful canticles. Maybe they never intended to be accurate; or maybe each was recording a completely different tradition already going around in his congregation.
Still, this is worrisome for the accuracy of the Gospels as a whole. If a story is unverifiable for most of it, but all the verifiable parts are verifiably false, what reason would you have to believe any of it? At the very least it downgrades how confident we ought to be in a source, because if some facts in it are false, others may be as well.