In the next decade or so, however, Griswold's reputation took a dive. His posthumous attacks on Edgar Allan Poe certainly left him open to public controversy, but he also suffered privately from the awkward divorce from his second wife, followed by attacks from angry women who tried to break up his third marriage. His face was by then scarred from an explosion in his home, and his wife and daughter were in a horrific train accident in 1853 (his daughter was briefly pronounced dead; his other daughter had been taken from him to live with his second wife). Amid all this, Griswold was no longer the intimidating literary power he once was, leaving him open to attack from even the most minor of poets.
On November 8, 1855, David Bates wrote to him angrily for being excluded from the 10th edition of The Poets and Poetry of America. He wrote that he never asked to be part of the "Temple of Fame" that Griswold had created with his book. Once included, however, he assumed he would stay in its pages (as most others did):
Perhaps I have been too modest. I certainly never begged the honor, or claimed it as a right: and yet I feel that an Author, who has been favourably noticed by the press, both in England and America... deserved that much consideration at the hands of an American in the land that gave him birth.
Bates particularly argues that he is popular enough, and therefore his exclusion is inappropriate. His writing was in school texts, for example, and so he demanded to know why Griswold, this self-proclaimed arbiter of American poetic taste, had neglected him (slyly suggesting it was personal):
As you are doubtless aware of the popularity of some of my poems, will you be kind enough to inform me why I have been treated with apparent neglect, as I am not conscious of having ever wronged you in thought, word or deed.
Whatever the reason, Bates was not reinserted for the 1856 edition — the final edition before Griswold's death in 1857.