Into the vault: the operation to rescue Manhattan's drowned internet
At Broad Street, near the tip of Lower Manhattan, the situation is far from normal. Many streets in the area are closed off and packed with trucks, equipment, and generators. Manhole covers are open everywhere. Verizon’s Broad Street central office, which routes local phone, DSL, and FiOS data, resembles a military field base. Walls of sandbags remain around the building, and the constant hum of generators and pumps bounces down the streets. The lobby of the building is covered in plywood to protect any decorations it may have, and the entrance has become a type of checkpoint lit by a string of incandescent bulbs.
On Wednesday, two weeks after the storm, I met with Verizon's Executive Director of Operations, Christopher D. Levendos, who showed me the extent of the damage and repairs. Levendos tells me the 90,000 cubic foot cable vault has suffered a “catastrophic failure,” far worse than the damage done to a similar, but much larger vault at Verizon’s West Street headquarters near the World Trade Center.
I’m told that an estimated 100 people are working here — a collection of contractors, power utility, and Verizon crews — and there seems to be a realization of how much work is left to be completed. As Levendos and I walk past the workers and squeeze between cables into the underground vault, I don’t know what to expect.
A two-day pumping operation has left the cable vault mostly dry, but it doesn’t look right. Cable insulation has been stripped back in areas, cords are cut, chunks of cables lie on the ground, and splice boxes have been torn open.
The 90,000 cubic foot cable vault has suffered a "catastrophic failure"
Levendos explains to me that before crews could even begin removing water, they needed to repair ground-level fuel pumps to feed backup diesel generators on the upper floors. Two mobile generator trailers were brought in, and they remained in use when I visited, as local power utility Con Ed worked to reconnect the building to the grid. Workers then used trucks to pump dry air through the copper wiring — a job that’s typically handled by air pumps in the basement that were rendered useless by the storm surge. It was too late for the decades-old copper wiring, which was submerged for the better part of two days. After crews sent test signals into the copper, Levendos says he was "left with the conclusion here that much of what is around me has been destroyed."
Miles of copper is ruined not only in the cable vault at Broad Street, but also at 20 or so manholes around the area. Even worse, paper insulation in the copper wiring sucks water through the cabling from capillary action, destroying cabling even in dry areas. Levendos says it’s "far too tedious, time consuming, and not effective of a process to try and put this infrastructure back together," so Verizon’s taking the opportunity to rewire with fiber optics instead. Service has been restored to FiOS customers for over a week — unlike copper, fiber optics aren’t damaged by the water. As part of this process, crews have already pulled fiber up the major corridors — including Water, Broad, and Pearl Streets — to ultimately connect the fiber network to buildings.
Despite the progress, huge challenges remain. While fiber optic cabling weathered the storm, the electronics that send light through them are vulnerable to water. Verizon has to analyze the extent of damage done to equipment in buildings they serve and see how much work remains to hook up areas without FiOS. Once fiber is brought to a building’s doorstep, workers still must bring service to each and every unit. Verizon wouldn’t give me a number, but thousands served by copper-based phone and DSL remain without service to this day in Lower Manhattan. For them, the wait will surely continue as the process of bringing fiber up floor by floor progresses.
Hint: Use the 's' and 'd' keys to navigate
This building near the tip of Lower Manhattan was occupied by New York Telephone from the late 1920s before the company became part of Verizon. In addition to office space, it houses equipment to route data worldwide.
A security guard checks in visitors at the Broad Street office's lobby, which has been turned into a construction zone.
Christopher Levendos, Verizon Executive Director of Operations, walks down a previously-flooded stairway leading to the cable vault.
The roughly 90,000 cubic foot cable vault houses both decades-old copper wiring and modern fiber optics. Since the building remains off the grid, it is lit only by temporary lighting.
In some areas, Verizon crews used an infrared camera, normally reserved for steam lines, to spot damaged areas of copper wiring. In one pass, crews could easily identify damaged splice points by the heat given off when the electrified wire met water.
The insulation around a splice point is revealed, here, after workers removed the casing full of water. It remains wet to the touch.
A tangle of copper wiring lays exposed at another damaged splice point. Air pumped into these lines would typically keep water out, but the pumps that feed them succumbed to the storm surge.
One of 13 points where cabling enters the vault has had many of its thick copper cables cut away to make space for new fiber optics inside orange tubing.
Much of the copper wiring inside the vault was installed decades ago. At an older splice point, groupings are marked with hand-written paper labels.
The cable vault at the Broad Street Central Office is about 15 feet deep, and spans an area 150 feet long and 40 feet wide. The storm surge filled it to the top for nearly two days.
After a false alarm caused by exhaust from a generator making its way inside the cable vault, workers waited on the sidelines awaiting the all-clear.
Over 100 workers — about half from Verizon — are working on repairing the site. The rest include some from the local power utility (Con Ed), and contractors.
Several streets at the far southern tip of Manhattan are still closed in areas flooded two weeks earlier to make space for manhole cover access, generators, and pumps.
Contractors cut open a hole in the sidewalk to allow direct access to the cable vault from above, and to help clear the air in the space.
Wire crews literally pull old cabling out using trucks, and thread new fiber optics back through the space previously used by copper.
Sandbags still surround the building, though they failed to hold back the nearly 15-foot storm surge.
Two large generators in trailers power the central office, though multiple smaller ones surround the building to power equipment.
In normal circumstances, the Broad Street central office deals with water due to its low elevation. Its systems couldn't cope with Hurricane Sandy's storm surge.
Air pumps near the cable vault maintain air pressure in the cabling to keep them dry. These pumps failed when the storm surge partially submerged them.
Workers have set up substantial infrastructure around the building for support, cutting off many streets.
One of the two trailer-sized generators that powers switching equipment inside the building to keep FiOS services running.
__._,_.___
Posts
No comments:
Post a Comment