When Elon Musk blamed Twitter's outage yesterday on code that "is extremely brittle for no good reason," he ignored the effect his massive layoffs had on Twitter's ability to keep the site running.
Twitter said the outage, which broke links and other functionality for about an hour, was caused by "an internal change that had some unintended consequences." That change, according to a Platformer article, "was part of a project to shut down free access to the Twitter API."
"The company has been building a new, paid API for developers to work with," Platformer wrote, continuing:
But in a sign of just how deep Elon Musk's cuts to the company have been, only one site reliability engineer has been staffed on the project, we're told. On Monday, the engineer made a "bad configuration change" that "basically broke the Twitter API," according to a current employee.
The change had cascading consequences inside the company, bringing down much of Twitter's internal tools along with the public-facing APIs. On Slack, engineers responded with variations of "crap" and "Twitter is down—the entire thing" as they scrambled to fix the problem.
During the outage, clicking on links brought up the following error message: "Your current API plan does not include access to this endpoint." The error message led to widespread speculation that the problem was related to Musk's decision to charge for API access, and Platformer's reporting appears to confirm that.
But the deeper problem is Musk's cost-cutting that has reduced Twitter's staff from about 7,500 to under 2,000. Twitter has fewer than 550 full-time engineers, according to Platformer's report.
"This type of outage has become so frequent that I think we're all numb to it," one employee told Platformer. Yesterday's problem was reportedly the sixth major outage since January 23.
Risk evaluation team was laid off
One employee partially backed Musk's explanation of "brittle" code. "There's so much tech debt from Twitter 1.0 that if you make a change right now, everything breaks," the employee told Platformer. Musk yesterday said the entire code base needs a "complete rewrite."
But having fewer technical employees makes it harder to ensure that changes won't break current functionality and lengthens the amount of time needed to recover from the resulting problems. "This is what happens when you fire 90 percent of the company," another employee told Platformer. Meanwhile, Musk has been demanding fast implementation of major changes.
A Washington Post article about yesterday's outage said that before Musk bought Twitter, "the company had a risk evaluation team that vetted product changes for anticipated problems." But that "team was laid off after Musk's takeover... leading to product rollouts that were riddled with bugs."
"Every mistake in code and operations is now deadly," one former engineer told The Washington Post in November.
“Welp, I just accidentally took down Twitter”
Those individual mistakes have been piling up. "In February alone, Twitter experienced at least four widespread outages, compared to nine in all of 2022, according to NetBlocks," a New York Times article last week said.
"In early February, a Twitter worker deleted data from an internal service meant to prevent spam, leading to a glitch that left many people unable to tweet or to message one another... A week later, an engineer testing a change to people's Twitter profiles on Apple mobile devices caused another temporary outage," the NYT wrote. "The engineer disregarded a past practice of testing new features on small subsets of users and simply rolled out the change—a tweak for Spaces, Twitter's live audio service—to a wide swath of users, two people familiar with the move said."
"Welp, I just accidentally took down Twitter," the engineer, Leah Culver, tweeted on February 15. "You can blame @elonmusk if you like though. 🫠"
Culver joined Twitter in early 2021 when it acquired Breaker, a social podcast app company she co-founded. In December 2022, Culver wrote a thread detailing why she stayed at Musk's Twitter even after most of her colleagues were laid off.
"The new Twitter team is TINY compared to what it was. I have a front row seat to Elon Twitter. TBD what happens next. It certainly is not boring," Culver wrote at the time. Culver was laid off on March 2.
reader comments
286