Well, don’t read too much into my stale recollection of what I read someone else say about an essay I hadn’t read. 
The originating essay isn’t terribly common. It’s apparently collected in a book about the Roman Church called Where All Roads Lead, and I found at least the relevant excerpt of it. GKC’s making an apologetic for the Roman Church, and in this part he’s focusing on its complexity:
There is a sense in which the Faith is the simplest of religions; but there is another sense in which it really is by far the most complicated. And what I emphasize here is that, contrary to many modern notions, it owes its victory over modern minds to its complexity and not its simplicity. It owes its most recent revivals to the very fact that it is the one creed that is still not ashamed of being complicated.
In contrast, he says:
We have had during the last few centuries a series of extremely simple religions; each indeed trying to be more simple than the last. And the manifest mark of all these simplifications was, not only that they were finally sterile, but that they were rapidly stale. A man had said the last word about them when he had said the first. Atheism is, I suppose, the supreme example of a simple faith. The man says there is no God; if he really says it in his heart, he is a certain sort of man so designated in Scripture. But anyhow, when he has said it, he has said it; and there seems to be no more to be said. The conversation seems likely to languish. The truth is that the atmosphere of excitement by which the atheist lived was an atmosphere of thrilled and shuddering theism, and not of atheism at all; it was an atmosphere of defiance and not of denial. Irreverence is a very servile parasite of reverence, and has starved with its starving lord. After this first fuss about the merely aesthetic effect of blasphemy, the whole thing vanishes into its own void. If there were no God, there would be no atheists.
I think he means that atheism is so simple, that beyond the denial of God there’s nothing more to it. Without God (or the idea of God) there would be nothing at all.
Not only do I think this is right, I know atheists who would agree. I had an atheist friend who had two pins: one the infamous scarlet “A” that atheists often wear, and another with the logo of a secular humanist organization. He said the “A” represented what he wasn’t (i.e. a believer in God), and the other was for what he was (a humanist). Atheism is just the negation of a belief in God, and it’s nothing you can really build a life or philosophy on. Some people think of themselves primarily as atheists, and they seem to concentrate on denying one thing. In my experience, these people are generally really angry about things religious communities have done to them. But most atheists I’ve been friends with were not of the angry variety, and preferred to identify as humanists, with plenty of positive beliefs. God just really wasn’t a topic of value to them.
Honestly, atheists (and more accurately, humanists) have come a long way since Chesterton’s time. They’ve had a century to improve their branding and message. So that has to be taken into account when reading Chesterton on atheists. That, and… well… as I’ve said before, he does generalize quite a bit. However, by all accounts, his relationships with real atheists were very cordial. I’ve already quoted his mail exchange with Wells. G. B. Shaw delivered a eulogy at his funeral (and GKC wrote a whole book criticizing Shaw!).
Christopher Hitchens was apparently working on a piece on Chesterton when he died. I would have loved to see a debate between Hitchens and Chesterton. I think they’d’ve gotten on famously.