AS THE CIVIL WAR heated up in the winter of 1862, Robert E. Lee took a moment from inspecting coastal defenses to visit Cumberland Island, Ga. There, amid the rustling saw grass and live oaks bearded with Spanish moss, he found the grave he was looking for, stood alone before it and paid tribute to Henry “Light Horse Harry” Lee, the father he never really knew.
“The spot is marked by a plain marble slab,” Lee wrote his wife, Mary, “with his name, age, & date of his death.” Filial duty discharged, he switched to other details of his Cumberland visit, describing the roses, ripe tomatoes, olive groves and stunted oranges on the grounds of nearby Dungeness mansion. “The garden was beautiful, enclosed by the finest hedge I have ever seen,” he wrote. “It was of the wild olive.” And then…not another word about his difficult, elusive father, whom he had last seen as a small boy. A witness recalled that Lee fell into a long silence as he walked back to the steamer that would return him to war.
During the American Revolution, Light Horse Harry Lee earned his fame as a swashbuckling commander of a cavalry and light infantry brigade. After the war, he emerged as an ardent Federalist who helped insure passage of the Constitution and served as the ninth governor of Virginia. He was also a congressman. But the cunning and audacity that made him formidable on the battlefield and in politics also led to his personal downfall. By turns patriot and scoundrel, upright citizen and confidence man, benefactor and deceiver, Lee ended up debt-ridden, broken in body and spirit, exiled from the country he helped create and marooned from his family, including his youngest son, Robert E. Lee.
Light Horse Harry, born to one of the nation’s most prominent families and educated at the College of New Jersey in Princeton, found his natural calling as a soldier. In 1779, as the Continental Army teetered between success and disaster, Lee, 23, led a handful of dragoons in a daring night raid on Paulus Hook, N.J., where a British garrison dominated New York harbor. Marching 30 miles on forest trails and wading tidal rivers that soaked their gunpowder, Lee’s men surprised the British and, armed only with bayonets and bluster, seized 158 prisoners and melted into the countryside as morning lit up the waterfront.
Washington praised Lee for acting with “great address, intelligence and industry,” but his triumph was clouded by the sort of controversy that would follow him all his life. Senior officers complained that the young major had usurped their authority and acted too hastily. They called for a court-martial. Lee was arrested, but eventually exonerated and promoted to lieutenant colonel. Congress even issued a rare gold medal in his honor.
Washington then sent Lee south to fight under General Nathanael Greene in the Carolinas and Georgia. Lee raided British outposts, slashed supply lines and gathered intelligence that kept the enemy off balance. He also exhibited a certain ruthlessness, ordering his men to display a deserter’s severed head in camp and torturing a loyalist civilian by holding a red-hot shovel to his feet during interrogations. In a particularly grisly incident, Lee’s legionnaires lured loyalist militiamen from North Carolina into a trap by posing as British dragoons and massacred 100 fellow Americans with gunfire, sabers and bayonets.
Even as Lee the insurgent reveled in such violence, a part of him recoiled at the manner in which civilians, both loyalist and patriot, used the Revolution as a screen for settling old blood feuds by arson, theft and torture on a grand scale. Lee thought the South Carolinians were worst of all, exceeding “the Goths and Vandals in their schemes of plunder and iniquity,” he told Greene.
In 1782, with the war’s end in sight, Lee abruptly resigned his commission. He said he felt mentally drained, overlooked and unappreciated. He headed home for Virginia, keen on pursuing the pleasures of peace but so distraught about leaving the army that he could not say his farewells in person. “The ceremony of parting from you & my friends in the army is so affecting, that I wish to decline it personally,” he wrote Greene. His commander was magnanimous. “You are going home,” Greene told Lee, “and you will get married; but you cannot cease to be a soldier.”
True to Greene’s prediction, within months Lee married a wealthy second cousin, Matilda Lee, who inherited Stratford Hall, a red brick plantation house anchoring more than 2,000 acres on the Potomac River. Matilda bore three children before her premature death during childbirth eight years later. Lee soon grew bored with the serious business of farming and followed other relatives into the era’s rough-and-tumble politics, which often seemed akin to combat, with pyrotechnic arguments, warring camps and tenuous truces. In 1785, he became a delegate to the Continental Congress, where he alienated fellow Virginians because of his outspoken views on western trade.
When Congress approved the new Constitution in 1787, Lee won election to Virginia’s ratifying convention, where he butted heads with Patrick Henry and other anti-Federalists who recoiled at the notion of a strong central government. Henry even wanted to change the now famous opening words of the Constitution, “We, the People,” to “We, the States.” Unless a central authority joined Americans together, Lee countered, European powers would easily pick apart the loose confederation of individual states. Lee and the Federalists carried the day, clinching ratification by a vote of 89 to 79.
Even as Lee plunged into politics, he still yearned for the thrill of soldiering, an ambition that eluded him except for a few weeks in 1794. That was when President Washington recalled Lee, then in his third term as Virginia’s governor, to lead the state’s militia against the Whiskey Rebellion in western Pennsylvania. The show of force, from Virginia and other state units, ended the crisis without further bloodshed. Commissioned as a major general for that brief action, Lee clung to the title the rest of his life, and encouraged everyone who called him “General.”
Attributes that made Lee a gifted warrior and influential politician— swiftness, optimism, bravado— served him poorly in private life. He dreamed big, bought big and was soon holding thousands of raw acres in Pennsylvania, Virginia, North Carolina and Georgia, where he envisioned new settlements for a growing country. He expanded family holdings around Stratford. He pushed to make Washington, D.C., the nation’s capital, buying overpriced lots there in anticipation of a building boom that never arrived. In his poorest investment, he joined financier Robert Morris to develop a canal system at Great Falls, Va. The Potomac Company scheme, designed to link Europe with western lands beyond the Alleghenies, was breathless in its scope, “almost too extensive for imagination,” as George Washington described it. In Lee’s exuberant apparition, a new city named Matildaville, in honor of his wife, would rise on 500 acres he owned at Great Falls. He put the arm on his old Princeton classmate James Madison as a prospective investor. “The views of thousands are pointed over the mountains for the support of numerous families,” he assured the future president. “The potomack river will strengthen our connexion by the easy exchange it affords of those things mutually wanted.”
Neither the Potomac Canal nor Matildaville was ever completed. Lee fell further into debt. He tried to dig his way out by borrowing more money and buying more land. The deeper he sank, the more desperate he became. In 1789, angling for an easy profit, he appealed to his friend Alexander Hamilton, then secretary of the Treasury, for inside information about the government’s plans to issue new currency. Hamilton rebuffed him.
Meanwhile, Lee’s family responsibilities grew. Soon after Matilda died in 1790, Lee married Ann Hill Carter, another Virginia blueblood, who bore him six children. As his obligations mounted, the family’s holdings around Stratford shrank from more than 2,000 acres to 236. He put heavy chains on the doors of the plantation house to keep out collectors. Sorting through a mound of papers to deconstruct a particular deal, Lee admitted that it was “so obscured & so mixed with my other transactions that I cannot speak with any precision thereon.” Whether by intention or confusion, he began selling tracts he no longer owned. He stuck Washington and Thomas Jefferson this way, as well as his youngest brother, Edmund Jennings Lee, who was dragged into court in Kentucky when he sold property he thought was his.
In 1798, Lee bought shares in the Dismal Swamp Company from Washington, paying the former president with a worthless bank draft and devalued bank shares. “It is a mode of dealing to which I am not accustomed,” Washington wrote stiffly. Lee was mortified. “No event of my life has given me more anguish,” he told his old commander, as if mysterious outside forces had guided their transaction. Lee still owed Washington money when the president died in 1799. Unable to pay him in cash, Congressman Lee settled the debt with words, memorably eulogizing the first president as “first in war, first in peace, first in the hearts of his countrymen.”
Despite his financial woes, Lee remained the aristocrat, with a taste for good Madeira, a conviction that he was smarter than others and an aversion to the new Republican Party, which the ardent Federalist viewed as a prelude to mob rule. As a member of the Sixth Congress, he tried to block the election of the nation’s leading Republican, Thomas Jefferson, when the presidential contest was thrown into the House of Representatives in 1801. Years after that ill-fated effort to derail Jefferson, Lee nursed a grudge for the sage of Albemarle County, whom he considered a schemer, a coward and the agent of his own declining influence within circles of power.
Lee’s dwindling political influence paralleled his economic decline, which grew increasingly dire. “My necessities are such that every day’s gain in time is momentous to me,” he wrote. After years of sidestepping creditors, Light Horse Harry had earned a new sobriquet: Swindling Harry Lee.
With his reputation in tatters and his ruin assured, Lee reverted to mobility and stealth, the tactics he had employed as a young cavalry officer. He seldom stayed at Stratford, instead roaming the countryside, a ghostly figure calling on relatives and friends who grew accustomed to his abrupt departures. He concocted plans that never materialized: He would go to France to lead revolutionary forces. He would win appointment as an American agent to the West Indies or Brazil, far from the reach of those clamoring for money. In 1808, he swallowed his pride and begged his friend Madison, then secretary of state, for an overseas posting that would relieve “my distressed condition….I have served in war & peace the U.S. I never asked for any office, or even favor before, from govt. or any member of govt. Nor shd I do it now, but for the peculiarity of my condition.”
Madison never answered. The net was closing around Lee. It fell in April 1808, when he surrendered to the sheriff of Westmoreland County, who came with a warrant from creditors. Confined to a 12- by-15-foot cell in the county jail, Lee became a ready target for other creditors. A group from neighboring Spotsylvania County brought suit to recoup $425, a modest sum but enough for another warrant, prompting Lee’s transfer to a Spotsylvania jail. He remained in that “depot of misery,” as he called it, for most of a year. He begged his long-suffering second wife, Ann, to wait for him. She agreed, but resolved to leave Stratford as soon as her husband was freed.
Lee paced his cell, wrote old comrades and focused on the luminous past, when he lived in the saddle, trusting “on the speed and sound- ness of his horse” to elude pursuers. He began a memoir, piling up more than 600 action-packed pages. The book was almost finished when Lee, 55, walked out of jail in March 1810, having agreed to pay off his greatest debts with parcels of land—more than a hundred thousand acres scattered from the Caribbean to the Alleghenies—along with his remaining interest in Stratford.
He headed north to Alexandria, where he hoped to make a fresh start with his growing family, which included a shy 4-year-old named Robert Edward Lee. Ann’s sympathetic relatives had offered free use of a house, but had no more patience with her husband. Kinsmen stopped fronting funds for Lee’s schemes, and he was barred from tapping a modest trust fund established for Ann.
Lee expected his book to make him rich again. “I indulge a hope that it will take a great run when published,” he wrote. But in Alexandria, he discovered that his papers, transported upriver by boat, had been soaked. His hope for the future, reduced to a bundle of soggy pages, was laid out to dry in an empty room of the family’s borrowed home. The baptism proved an omen. When his memoir was published in 1812, reviewers praised Lee for his “surprising quickness and talent” as an author, but the book never earned him a dime.
Relegated to the fringes of political life, Lee nonetheless remained a staunch Federalist, fatefully involving himself in the violence that announced the War of 1812. Weeks after President Madison signed a declaration of war against Great Britain on June 18, 1812, Lee traveled to Baltimore to support a Federalist newspaper editor, Alexander C. Hanson, whose strident antiwar rhetoric had inflamed public sentiment. Rioters attacked the offices of the Federal Republican and Commercial Gazette, destroying the press and burning the building. Hanson defiantly continued producing the paper at another location. When Lee and others rallied behind him, a brick-throwing mob forced them to seek refuge in the city jail. The next day, rioters broke into the jail, dragged Hanson and his cohorts into the street and beat them. One died. Another was stripped, tarred and feathered. Lee and Hanson were knifed and pummeled. One rioter tried to cut off Lee’s nose, disfiguring him; another poured hot wax in his eye to see if he was dead. He was not, but he seemed lifeless, insensible and crumpled on the cobblestones, “as black as a negro, his head cut to pieces” and “covered with blood,” one witness said. Friends carried Lee away and nursed him.
Crippled and mentally broken, he never fully recovered. Children recoiled at the sight of his horribly scarred face. His sight was impaired, his health “deranged,” as he put it. The old defiance and bounce had been beaten out of Light Horse Harry. Now he just wanted to get away—from the creditors who still haunted him, from the family he could not support, from the nation that no longer heeded his counsel.
“Anxious as I am for the happiness of my country, it is not in my power to aid it,” he wrote just before sailing for the Caribbean in April 1813. He spent five years wandering—from Port-au-Prince to Barbados to Guadeloupe to Havana to Puerto Rico to San Pedro and back to Port-au-Prince, then on to the Bahamas—hoping the warm climate would restore his health.
He stayed with friends, wrote bad checks, commandeered rooms and consulted doctors who did little to relieve his suffering. He read Alexander Pope, Confucius, Homer and Demosthenes (the last two in Greek), kept up a blizzard of correspondence with friends and family. “My heart never turns from you a moment,” he wrote to the new master of Stratford, his eldest son Henry Lee IV, “& all dear to it pass hourly in review of my minds eye.” He asked his namesake to “take care of my wife & children” and urged his other sons to avoid debt and to practice self-control, “the pivot upon which the character, fame, and independence of us mortals hang.” He expressed concern about his two youngest sons, Sydney Smith Lee and Robert E. Lee, and wrote Ann that he hoped Robert was “as good as ever…it is his nature.”
Even as Lee indulged himself by ordering pickled oysters, venison hams and flanks of beef from New York, he lamented that his wife’s meager trust barely kept food on the table in Alexandria. “Your self privations cannot be permitted,” he wrote Ann in 1817. “If I ever approach you, I must alter the condition.” Then, perhaps remembering who had placed her in that condition, he blurted: “God of heaven, how cutting to my heart the knowledge of yr. situation.”
Lee’s own situation worsened, aggravated by a painful bladder condition, as well as trauma from his beating in Baltimore. He was deathly ill when, homeward bound, he landed at Cumberland Island in early March 1818 and was given shelter at Dungeness, an estate built by his old commander, Nathanael Greene. But Lee was too weak to make it home to Alexandria. He died a few weeks later at age 62 and was buried in the Greene family plot, escorted by a marine guard and saluted by the USS John Adams. The warship, anchored just offshore, solemnly marked the progress of the funeral party until Light Horse Harry was settled into the sandy soil of the seaside cemetery.
Nearly a half century had passed when Robert E. Lee made his 1862 pilgrimage to Dungeness to pay homage to his dashing, distant father. Harry went to jail when Robert was not yet 2, sailed into exile when he was 6 and died on faraway Cumberland when he was 11. Biographer Elizabeth Brown Pryor calculated that Robert spent no more than a total of 34 months with Harry, most of it as a toddler. Even so, it is easy to see Harry’s audacity burning bright in Robert, who also absorbed his father’s genius for improvisation and implacability on the battlefield. And the son undoubtedly gained from the father’s negative example, learning the self-command, rectitude and frugality that Light Horse Harry Lee habitually preached but almost never practiced.
Robert M. Poole is the author of On Hallowed Ground: The Story of Arlington National Cemetery and Explorers House.
Originally published in the June 2012 issue of American History. To subscribe, click here.
