Advertisement

SKIP ADVERTISEMENT

The Bug That Crashed New York’s Wireless Network

Despite warnings, New York City’s technology managers were blindsided by the so-called GPS rollover, failing to install simple but necessary upgrades.

After a wireless network used by city agencies crashed in April, the Police Department relied on mounted license plate readers.Credit...Michael Appleton for The New York Times

[What you need to know to start the day: Get New York Today in your inbox.]

For at least a year, federal officials and technology companies had been warning of the so-called GPS rollover, a once-in-20-year event that had the potential to wreak havoc on computer networks around the world.

The simple remedy involved some necessary upgrades.

Yet somehow, New York City’s technology managers were caught completely off guard, and did nothing to prepare for the calendar reset of the centralized Global Positioning System.

As a result, a wireless network used by city agencies crashed in April, crippling many services that relied on it, including some Police Department license plate readers and a system to remotely control traffic lights. It took 10 days to get the network running again.

Officials at several city agencies, including the Police Department and the Office of Emergency Management, knew about the rollover, according to a report released by the city on Friday.

But officials at the Department of Information Technology and Telecommunications (DoiTT), which was responsible for operating the wireless network, claimed “that they were not aware” of the rollover before it occurred, the report said.

After the network went down, confusion, poor communication and a lack of coordination hampered attempts to get it working again, according to the report, which was compiled by the consulting firm Gartner at a cost of $300,000.

A week before the report’s release, the DoiTT commissioner, Samir Saini, resigned. Mayor Bill de Blasio said that Mr. Saini wanted to return to the private sector, and disputed the notion that his departure was connected to the failure.

The report did not name any of the people who were responsible for the missteps, and the city has not publicly disciplined anyone in relation to the incident.

Nonetheless, the report, with its revelations of poor preparation and the chaotic response, could be embarrassing for Mr. de Blasio, who is running for president and has argued that his experience managing the nation’s largest city makes him more qualified than other candidates.

On the day the wireless network crashed, Mr. de Blasio was in Nevada, an early primary state, as he considered whether to declare his candidacy for president. City Hall initially tried to hide the shutdown. It made no public acknowledgment of the problem and, in response to questions from The New York Times, officials initially characterized it as a routine maintenance issue.

The report does not indicate when Mr. de Blasio was told of the problem or whether he was informed of the confusion surrounding the attempts to get it working again.

In a statement accompanying the report’s release, Laura Anglin, the deputy mayor for operations, who also oversees the information technology department, asserted that “there were no interruptions to city services during the NYCWiN outage,” but acknowledged that “it is critical we learn from this event.”

Ms. Anglin’s statement, however, is directly contradicted by the report, which details several service interruptions. About half of the city-operated signs showing arrival times at bus stops were disabled, as were about 200 cameras that provide online images of traffic conditions; many other tasks handled by the network were knocked offline, requiring city workers to be reassigned to perform the tasks manually.

The report made it clear the episode could easily have been avoided. The wireless network, like many other computerized systems, uses GPS data to keep track of time. The GPS rollover was widely known, and government and industry notices encouraged technology managers to upgrade systems to avoid possible interruptions.

The report’s authors interviewed eight top officials at the information technology department, including Mr. Saini. But the report said that no one at the agency admitted being aware of the approaching rollover. It does not say whether it considered those denials to be credible, given the amount of publicity related to the rollover in the technology industry.

Mr. Saini was hired just a year and a half ago by the mayor and was involved, among other key initiatives, in the modernization of the 911 system. Attempts to reach Mr. Saini were unsuccessful.

According to the report, the system could easily have been upgraded by replacing what is known as the firmware in the dozens of nodes, or antennas, that make up the network.

Northrop Grumman, the contractor that maintains and operates the network at a cost to taxpayers of $37 million a year, also did not alert city officials to the need for an upgrade, the report said.

“Northrop Grumman worked expeditiously” with city officials “to address the GPS rollover event,” said a company spokesman, Tim Paynter, in an emailed statement. Mr. Paynter did not respond to questions about whether Northrop informed the city of the need for upgrades ahead of the rollover.

Many passages in the 35-page report were blacked out, which city officials said was done for security reasons.

The wireless network was built for about $500 million and has been in use since 2008. But today it is used by only about 10 city agencies. The city plans to shut it down in the coming years and shift its wireless needs to commercial carriers, which it says will save money.

But the city has been slow to carry out plans for such a transition. The report recommended that New York review its technology infrastructure and warned that the city “may be exposed to more risk than necessary regarding technology-related incidents.”

A version of this article appears in print on  , Section A, Page 21 of the New York edition with the headline: City Could Have Avoided Wireless Network Crash. Order Reprints | Today’s Paper | Subscribe

Advertisement

SKIP ADVERTISEMENT