This stalwart, 1844 brick townhouse with a Greek Revival porch was one of a number of adjacent houses known as Stewart's Row. It was used by Confederate Gen. Robert E. Lee first as a bachelor's officers' quarters and early in 1864, his wife moved in. While Mary Custis Lee, a great-granddaughter of Martha Washington, was incapacitated with rheumatism and would have found the four-story house difficult to navigate, there are stacked porches on the rear, south side of the house, which would have offered river views and breezes. While her stylish neighbors — like Martha Pearce Stanard and next door neighbor, Secretary of War George W. Randolph, enjoyed entertaining, Lee kept a low social profile: Her wartime activities included sewing and mending uniforms for soldiers.
On April 3, 1865, when the United Presbyterian Church just around the corner burned, Mrs. Stanard’s home across the street was destroyed, and even the house next door caught fire.Mary Lee resolutely stayed put and had tea with some of her daughters. It didn’t hurt that a bucket brigade of Richmonders doused her house to reduce the heat. It was here, to 909 E. Franklin St., where Robert E. Lee returned after surrendering at Appomattox to Union Gen. Ulysses Grant. On that occasion he posed in a rear doorway to be photographed by photographer Matthew Brady.
Some years later the house was the home of A. M. Keiley, Richmond's mayor from 1871 to 1876. Then, from 1893-1959 it was headquarters of the Virginia Historical Society. The Lee residence now houses offices of the Homebuilders Association of Virginia.