The patriarch of the Stedman family never returned. He died aboard the Emily two weeks after his last letter en route to Santa Cruz and he was buried at sea. It took four weeks before the family heard of his death. Without him, they packed up their belongings and moved from Hartford to Plainfield, New Jersey to live on the farm of his maternal grandfather. His Puritanically pious grandfather attended to his earliest education, teaching him to read using the family Bible. By the time he was six, he was precocious and already flowing with poetry. On more than one occasion, his mother reported, he refused to go to bed and responded to the request, "Let me alone, please, the poetry is coming." Sure enough, the boy grew up to be an influential man of letters.
At Cedar Creek, the young Stedman was crowded by family, including several cousins. His father's family, however, wanted the Stedman children back in Connecticut. His paternal grandfather even promised a substantial inheritance if Mrs. Stedman complied. She initially refused, despite her financial distress, and attempted to earn extra money by contributing to magazines like Godey's and Graham's. Eventually, she gave up the effort, and the young boy was taken in by his father's brother. Even so, Stedman always had a preference to his mother, who died when he was in his 30s. To her, he wrote the sonnet "A Mother's Picture":
She seemed an angel to our infant eyes!
Once, when the glorifying moon revealed
Her who at evening by our pillow kneeled, —
Soft-voiced and golden-haired, from holy skies
Flown to her loves on wings of Paradise, —
We looked to see the pinions half concealed.
The Tuscan vines and olives will not yield
Her back to me, who loved her in this wise,
And since have little known her, but have grown
To see another mother, tenderly
Watch over sleeping children of my own.
Perchance the years have changed her: yet alone
This picture lingers; still she seems to me
The fair young angel of my infancy.