(Bloomberg) — AT&T Inc. said a widespread outage that took hours to resolve Thursday was caused by “an incorrect process” while expanding the wireless network.
Most Read from Bloomberg
The software issue interrupted wireless service for hundreds of thousands of subscribers and prompted the FBI and US Department of Homeland Security to investigate the outage.
“Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” an AT&T spokesman said in a statement. “We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve.”
AT&T said all wireless service was restored Thursday afternoon, capping a day of frustration that began in the early hours of the morning New York time. AT&T customers filed more than 1.5 million outage reports on service-tracking website Downdetector.
The federal government began investigating whether the network failure was caused by a cyberattack, according to two US officials familiar with the situation, who requested anonymity to discuss sensitive information.
The Federal Communications Commission also has been in touch with AT&T to try and ascertain the cause, White House spokesman John Kirby told reporters earlier. “DHS and the FBI are looking into this as well, working with the tech industry, these network providers, to see what we can do from a federal perspective to enhance their investigative efforts to figure out what happened here,” Kirby said.
Read More: AT&T Outage Turns Start of US Workday Into the ‘Apocalypse’
Story continues
Early Thursday, mobile-phone customers from multiple carriers started reporting problems, but it soon became clear that AT&T’s network was the culprit. Outages were reported from cities including New York, Houston, Atlanta, Miami, Chicago and Dallas. The service disruption upended communications with emergency responders, and officials took to social media urging AT&T customers to use landlines to call 911 for emergencies.
With about 87 million subscribers, AT&T is the third-largest US retail wireless carrier, behind Verizon Communications Inc. and T-Mobile US Inc., according to data compiled by Bloomberg. Verizon and T-Mobile both said their services were working normally. AT&T shares fell 2.4%.
The impact was felt far and wide. Emma Smits, an AT&T customer, was on the Metra express train into Chicago. Normally, the public relations account executive spends the hourlong commute prepping for client calls, pitching reporters and catching up on overnight tasks. Not on Thursday.
“I couldn’t cross anything off my work to-do list,” said Smits, who watched her fellow commuters turn their phones off and on to try and catch a cellular signal in a futile attempt to send emails or get work done.
Workarounds abounded, sending some back to simpler times. Vanessa Stowe had to screenshot directions to a morning meeting “like it’s 2009 MapQuest.” Sarah Kittel, a communications strategist, had to connect to a nearby restaurant’s Wi-Fi to pay for parking.
“So much of our work and lives depend, sadly or otherwise, on our ability to be reached or reach others at a touch of a button,” Kittel said. “That simply didn’t happen this morning.” Cate Luzio, traveling from New York to Washington for a big client event, was more blunt: “It was a mess.”
Among those impacted were employees of the FBI. In an email to all staff reviewed by Bloomberg, the agency’s Information & Technology Branch said “a nationwide cell service outage is preventing some Google Pixel devices from connecting to cellular coverage. Some users have been able to reconnect to the network by rebooting their Pixel (press and hold the Power button, then press ‘Restart’).”
The public safety network FirstNet was the only government entity impacted, and service there has been restored, said Kirby, the White House spokesman.
FirstNet provides communications services to about 27,500 public safety agencies in a partnership with AT&T.
Ken Corey, a former Chief of Department for the New York City police, said the outage at FirstNet “has impacted 911 response and public-safety communications across the country.” Corey, who spoke via email, leads a group that opposes letting FirstNet manage additional airwaves.
It’s not the first time AT&T has suffered a widespread outage. In 2008, the company dealt with an extensive wireless internet failure in the US Northeast. In that incident, a glitch in the way the company’s network was routing traffic was identified as the culprit and the service failure was relatively short-lived. In 2020, AT&T internet and phone service was knocked out in Nashville by an explosion in the city’s downtown area.
T-Mobile also has been hit with widespread outages, including one last February and another in 2020 that prompted the FCC to launch a probe.
“In my experience, this type of outage can negatively impact financial results in the quarter in which it occurs and cause short-term lost goodwill with customers,” David Heger, an analyst at Edward D. Jones & Co., said in an email. “However, it does not have a longer term impact on the business.”
–With assistance from Brendan Walsh, Antonia Mufarech, Jamie Tarabay, Matthew Boyle, Akayla Gardner, Brendan Case and Miranda Davis.
(Updates with comments from AT&T beginning in the first paragraph. A previous version of this story corrected the spelling of AT&T in seventh paragraph)
Most Read from Bloomberg Businessweek
©2024 Bloomberg L.P.
