Halleck kept the job for the rest of his life, becoming a trusted friend with Astor. When Astor turned philanthropist, one of his projects was establishing the Astor Library (a predecessor to the New York Public Library). He relied heavily on the literary knowledge of Halleck and appointed him a founding trustee. Halleck also assisted in purchasing works of art.
When Astor died in 1848, he was immensely wealthy. Years earlier, Halleck criticized him for accumulating so much money. The humble Halleck reportedly asked him, "Mr. Astor, of what use to you is all this money? I would be content to live upon a couple of hundreds a year for the rest of my life." The millionaire Astor took it literally (or, rather, satirically) and left him an annuity of $200 in his will. When Halleck's readers complained of the paltry sum, the Astor estate increased it to an amount rumored between $1500 and $10,000. It still was not enough for Halleck, who was by then in personal financial distress. He soon recognized, however, that the sum was "a small fortune" if he returned to his home town of Guilford, Connecticut. To that town he retired and remained for the rest of his life. After his death, he was honored with a statue in New York's Central Park.
One of Halleck's most well-known poems is the humorous "Fanny," written in a staggering 175 parts. In its sixth stanza, however, he mentions:
Money is power, 'tis said — I never tried;
I'm but a poet — and bank-notes to me
Are curiosities, as closely eyed,
Whene'er I get them, as a stone would be,
Tossed from the moon on Doctor Mitchill's table,
Or classic brickbat from the tower of Babel.