Share This Article

Spanning the thin strait that separates Upper and Lower New York Bay is the Verrazano-Narrows Bridge. The 13,700-foot-long structure—at the time of its completion, the world’s longest suspension bridge—connects the New York City boroughs of Brooklyn on the east and Staten Island on the west, opening a link to New Jersey. The bridge opened to traffic on November 21, 1964. In its first year of service, 48,000 vehicles a day passed over the span.

Just south of the bridge, on the Brooklyn side, stands historic Fort Hamilton, and within its grounds stands a wood-frame house with a broad front porch that was once occupied by Captain Robert E. Lee and his family. In the early 1840s, Lee, an Army engineer, was assigned to strengthen the defenses of New York Harbor, including Fort Hamilton itself, diamond-shaped Fort Lafayette on a small island a few hundred yards off the Brooklyn shore and Fort Wadsworth on the Staten Island side. The Lees remained in the area off and on for five years.

From 1841 to 1846, Lee often could be seen aboard Flash, the little skiff he used, making sketches, taking measurements and overseeing workmen as they rebuilt large sections of the weather-beaten forts, strengthened batteries for heavy guns and made other badly needed improvements.

While he attended to his work, Mrs. Lee, far from their gracious mansion, Arlington, in Virginia, made a home for her husband, sons Custis and Rooney and daughters Mary, Annie and Agnes at their modest quarters. While the Virginia family remained for one uneventful year after another, many officers came and went from the post. These included future Confederate legend Thomas “Stonewall” Jackson and two young artillerymen, John Sedgwick and Henry J. Hunt, whom Lee later, as commander of the Army of Northern Virginia, confronted at Gettysburg and many other bloody battlefields.

The Lees were part of the Brooklyn scene for so long that the captain became a vestryman at St. John’s Episcopal Church at the fort (and was often accompanied to the church, his office and elsewhere by a menacing terrier named Spec that he had plucked from the Narrows after the dog had abandoned a passing ship).

Lee, in his middle and late 30s during this period, was a familiar and handsome figure as he followed his almost daily practice of riding along the Bay Ridge shoreline to Army district headquarters on Governor’s Island off the tip of Manhattan for conferences about the progress of his work. He had much time along the way to ponder the apparent dormant state of his career. Held to his seemingly unimportant work by his deep reverence for the concept of duty, he could not have even remotely suspected what greatness was in store for him.

It was while his father was off at one of those routine meetings that 8-year-old Rooney—or more formally William Henry Fitzhugh Lee—managed to get his hand caught in a straw cutter he was examining in a barn at the fort. His curiosity nearly cost him the tips of two fingers. But this was not to be the last painful experience the young Lee would have in New York.

Some 15 years after the family’s departure from the New York area, with the advent of the Civil War, Fort Lafayette was turned into a prison for an odd mixture of political and military prisoners, both Northerners and Southerners. As many as 140 prisoners at a time would be confined at the miniature fortress (dubbed the American Bastille). Housed within its somber, 30-foot-high and 8-foot-thick brick and stone walls were blockade runners; bounty jumpers; a former New York hotel owner, who had defaulted on a commitment to pay bonuses to men he had recruited for a Union regiment the hosteler was raising; and officers who had resigned commissions in the regular U.S. Army and had been taken while fighting for the Confederacy. For a while, the prison population also included some members of the Maryland Legislature being held for the simple purpose of preventing them from passing an act of secession that would take their state out of the Union.

One prisoner remembered Fort Lafayette upon arrival as a “repulsive dungeon-looking place with its jaw of a sallyport gaping to receive its victims.” So crowded was the octagonal facility, built around a grass plot used as a parade ground, that most of the inmates slept in four arch-ceilinged casemates which they had to share with the immense 32-pound guns that Captain Lee had long ago positioned to guard the harbor. One prisoner called the pieces, which with their carriages and machinery took up most of the space in the crammed quarters, “the most unpleasant looking bed-fellows imaginable.” The narrow, glazed portholes in the walls did give the prisoners romantic views of the bustling bay alive with ships of all descriptions under sail and steam, but made their island prison stand out, thought one man, as “a vile blot upon that fair scene.”

For those without financial resources or relatives or friends to supplement their diet, food at the prison was the primary hardship to endure. Prepared by a staff of four indifferent Yankee privates and served in a damp, brick-floored mess area the “guests” called the “United States Hotel,” a typical dinner at Fort Lafayette featured “a lump of boiled beef, apparently cut with a hatchet,” on a tin plate. “In each tin cup is a greasy looking mixture, which on examination is seen to be intended for rice soup,” the culinary review of one guest continued. “The piles of good bakers’ bread are strewn along the table, with a few plates of salt at intervals. The rice, nine times out of 10, is cooked no softer than it came from its original package, and the whole greasy mess in its greasy cup, to be eaten with a greasy spoon, seldom fails to turn the stomach of some of the guests.”

Visitors sometimes brought in well-concealed morsels, particularly some sympathetic “lady friends” who made the trip almost weekly from Baltimore. The Rebel prisoners long remembered the latters’ visit at Christmastime in 1863, one beneficiary noting, “the mutton chops were much appreciated on account alone of the labors and exposure occasioned the ladies in getting over the ice to the boat.”

Residents of the village outside Fort Hamilton sometimes went across to the island taking things to the prisoners, but many of their visitors looked upon a boat ride to the fort as an excursion to the zoo. The New York marshal, for example, said he “would sometimes bring down quite a little crowd to see the animals.”

Those prisoners with money could bribe a guard for better provisions or, quite legally, purchase the fare of a sergeant’s wife at Fort Hamilton who was officially sanctioned to deal with the prisoners. It was for that reason that the men being held there guarded their greenbacks so carefully. When a search was about to be made, they hastily stuffed their money in a rat hole or at the bottom of a coal bucket.

To relieve the monotony of their confinement, the prisoners began producing a secret handwritten newsletter they called the Right Flanker and filled it with biographical information about new arrivals, advice on the procurement of provisions and details about escape attempts.

Despite its proximity to Fort Hamilton, the rough, icy waters about the island made swimming to the shore risky even if the escapee managed to get outside the thick walls of the bastion. One Confederate captain, however, was able to get away from Lafayette. The Right Flanker learned and dutifully reported that the officer “plunged boldly into the sea from its walls, upon a dark and stormy night, with a life preserver around him which his friends had procured in a contraband manner, and having floated with the tide, landed on the beach some three miles distant from the fort.”

It was into this totally altered Fort Lafayette that a much-changed, 26-year old Rooney Lee returned. The big, amiable fellow (whom his father teasingly said was too large to be a man and too small to be a horse) had become a brigadier general in the Confederate cavalry. Though not a West Pointer (having gone, instead, to Harvard), Rooney had been given a second lieutenant’s commission in the Regular U.S. Army in 1857 at the behest of General-in-Chief Winfield Scott, who said at the time, “I make this application mainly on the extraordinary merits of the father, the very best soldier I ever saw in the field.”

Rooney had been severely wounded in the thigh at the Battle of Brandy Station on June 9, 1863, and was captured by a Union raiding party a few days later while he was convalescing at a relative’s home. He was at first held at Fortress Monroe, Va., but in November 1863, was ordered transferred to Fort Hamilton, for processing, and then to, of all places, Fort Lafayette. How strange he must have felt crossing to the island from the shoreline where so many years before his father had taught him and his brother Custis to swim.

The Lees had not been forgotten by the permanent residents of the locale. The locals had seen many officers come and go from Fort Hamilton, but the refined, cultivated Lees of Virginia had been different. Learning that one of the sons was in their midst—and painfully wounded at that—they met with him frequently to see to his needs at Lafayette.

They could do little for him, however, when word came that, on December 26, 1863, his frail young wife Charlotte Wickham Lee, his beloved “Chass,” had succumbed to a lingering illness. Efforts to secure his release to return to his home before she expired were unsuccessful. She was buried next to their two children, who had died in infancy. It was not until March 1, 1864, that Rooney Lee was finally exchanged and returned to the war in Virginia.

In March 1866, nearly a year after the end of the Civil War, the last prisoner was released from Fort Lafayette, and the facility lapsed into disuse. What little utility the grim structure might still have was further diminished by a severe fire in 1868.

Its isolation did give the island fort some value as a storage place for munitions during both World Wars. After WWII, one entrepreneur entertained the idea of converting the facility into an offshore nightclub. To thwart that scheme, the legendary urban developer Robert Moses saw to it that it was leased to the city in 1948 by the federal government and the island remained in public hands.

For decades a vehicular connection across the Narrows to link Brooklyn and Staten Island and their converging highway systems had been envisioned and numerous plans were bandied about. A tunnel was begun in the 1920s but halted when the cost became prohibitive.

Finally in 1946, a great $345 million suspension bridge won legislative approval. Designed by the renowned Othmar Ammann, and backed by Moses, the Verrazano-Narrows Bridge was to be 13,700 feet long—at the time, the world’s longest suspension bridge. It would feature two 693-foot-high support towers. The Brooklyn side of the bridge would be anchored on part of old Fort Hamilton and the Staten Island side on another fort, Wadsworth.

To make room for the Brooklyn tower, however, Fort Lafayette and the rocky shoal on which it stood would have to be cleared away. On the site, a cofferdam—a watertight compartment—of vertical steel pilings would be driven to protect caissons into which nearly 200,000 cubic yards of concrete would be poured as a base for the tower.

In February 1960, heavy cranes began the two-week task of leveling Fort Lafayette and transferring the tons of stone and brick rubble produced onto scows. They ferried the rubble across the Narrows to the shallower Staten Island side where it was deposited, load after load, to facilitate construction of the bridge’s west tower. Fort Lafayette—on whose remains the east tower was built— had been deserted for such extended periods that looters had left nothing of value. Demolition workers looking for souvenirs of the job found virtually nothing to pick up as a Civil War reminder. Said one man, “It’s been too well scavenged.”

Today, only the massive Verrazano-Narrows Bridge marks the site on which the young Robert E. Lee had labored dutifully with scant resources at a time when coastal defense was the primary occupation of the Army’s engineering corps.


Originally published in the February 2006 issue of American History. To subscribe, click here.