Twitter and ChatGPT have been knocked offline in an internet blackout blamed on the “unacceptable” failure of a “routine” upgrade at technology giant Cloudflare.
Internet users began reporting problems loading X, previously known as Twitter, around midday on Tuesday. Users of dozens of other apps and websites, including artificial intelligence chatbot ChatGPT, also reported widespread issues.
The crash has been pinned on problems at Cloudflare, an internet infrastructure business that provides cyber security and networking services to hundreds of major websites. The disruption lasted around three hours.
A Cloudflare spokesman said the company was investigating a “spike in unusual internet traffic” shortly before the issues emerged.
However, an executive later admitted the fault was caused by a “latent bug” in its systems that led to a mass internet crash after a “routine configuration change”. Cloudflare confirmed the problem was not caused by a cyber attack.
Dane Knecht, Cloudflare’s chief technology officer, apologised for the massive internet glitch and the time it took to fix, calling the blackout “unacceptable”.
He said: “The sites, businesses, and organisations that rely on Cloudflare depend on us being available and I apologise for the impact that we caused.”
The internet blackout comes just weeks after problems with Amazon’s internet infrastructure division, Amazon Web Services, knocked swathes of the web offline in the largest crash in history.
‘No evidence’ blackout was down to cyber attack, Cloudflare says
A Cloudflare spokesman has offered more detail on today’s blackout - confirming the outage was not down to a cyber attack.
The spokesman said: “The root cause of the outage was a configuration file that is automatically generated to manage threat traffic. The file grew beyond an expected size of entries and triggered a crash in the software system that handles traffic for a number of Cloudflare’s services.
“To be clear, there is no evidence that this was the result of an attack or caused by malicious activity. We expect that some Cloudflare services will be briefly degraded as traffic naturally spikes post incident but we expect all services to return to normal in the next few hours.”
That’s all for today - thank you for joining our coverage.
Cloudflare executive blames bug in ‘routine’ update for mass blackout
A top Cloudflare executive has blamed a “latent bug” in its systems for a mass blackout that took swathes of the internet offline this afternoon.
Dane Knecht, Cloudflare’s chief technology officer, said the worldwide internet crash was not down to a cyber attack, but was down to a problem introduced when it attempted to make “routine” updates to its service.
Mr Knecht said: “I won’t mince words: earlier today we failed our customers and the broader Internet when a problem in the Cloudflare network impacted large amounts of traffic that rely on us.
“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made.
“That cascaded into a broad degradation to our network and other services. This was not an attack. That issue, impact it caused, and time to resolution is unacceptable. Work is already underway to make sure it does not happen again.”
Cloudflare says it has fixed blackout bug
The internet infrastructure business Cloudflare says it has found the cause of the problem that took swathes of the internet offline this afternoon.
In a status update, the company said: “A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal.”
Shares in Cloudflare drop nearly 5pc
Cloudflare’s shares dropped nearly 5pc in early trading in New York this morning as investors sold their stakes in the wake of today’s internet blackout.
The US technology business, which is valued at more than $66bn, saw its stock drop by more than 4.6pc in early trading. It was down about 2.5pc as of 2.45pm on Tuesday.
Websites impacted by the Cloudflare blackout
ChatGPT confirms ‘full outage’
OpenAI, the Silicon Valley business behind ChatGPT, has admitted to a “full outage” of its AI chatbot and Sora, its video generation app.
In a status update, OpenAI said: “We have confirmed that the incident is caused by an issue with one of our third-party service providers. We will provide updates as they become available.”
Trying to open ChatGPT results in the following error message: “Please unblock challenges.cloudflare.com to proceed.”
Cloudflare blames ‘unusual’ web traffic surge for blackout
Tech giant Cloudflare has suggested a surge in “unusual” web traffic may have been to blame for today’s problems across a swathe of websites.
A spokesman for Cloudfare said: “We saw a spike in unusual traffic to one of Cloudflare’s services beginning at 11.20am. That caused some traffic passing through Cloudflare’s network to experience errors.
“While most traffic for most services continued to flow as normal, there were elevated errors across multiple Cloudflare services.
“We do not yet know the cause of the spike in unusual traffic. We are all hands on deck to make sure all traffic is served without errors. After that, we will turn our attention to investigating the cause of the unusual spike in traffic.”
Blackout follows devastating Amazon crash
Today’s Cloudflare glitch comes just weeks after problems at Amazon prompted the world’s biggest ever internet blackout.
More than 3,900 companies were affected by a crash at Amazon Web Services and over 16 million people logged connection problems during the blackout on October 20.
A computer error at Amazon’s vast data centres on the US East Coast caused chaos during the outage last month, rendering sites and apps such as Lloyds Bank, HMRC and Snapchat unusable.
Downdetector down: site for tracking internet blackouts offline
A website widely used to track internet outages has been knocked out of action by the latest Cloudflare blackout.
Downdetector, a service that monitors problems across the web, is widely used to check on the status of websites.
However, the popular service was also hit by today’s Cloudflare issues, with its site only intermittently available and struggling to load.
‘Services are recovering’, says Cloudflare
Cloudflare has issued an update saying its services have begun to recover from today’s blackout: “We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts.”
However, it added that some users may continue to see issues, including in the UK, as it seeks to solve the problem. “During our attempts to remediate, we have disabled WARP access in London. Users in London trying to access the Internet via WARP will see a failure to connect.”
WARP is a tool from Cloudflare that includes additional privacy protections.
What is Cloudflare?
Most people outside of the IT world or technology journalism have probably never heard of Cloudflare, but the IT infrastructure company provides services that help keep swathes of the internet running smoothly.
Cloudflare provides a variety of tools that help keep websites running properly, even if they are being hit with a spike in traffic or experiencing a cyber attack.
About 20pc of all websites rely on Cloudflare to some degree.
It operates a “content delivery network” that helps streamline how websites load for users. These systems help websites cope with high user numbers by spreading the burden of delivering images, videos and app content across a network of servers.
It also provides cyber security services, such as protection from so-called “distributed denial of service” attacks that are intended to overload websites with spam traffic.
Its tools form a key backbone for many websites. But when problems hit Cloudflare’s own systems, they can spread quickly across the web.
Cloudflare investigating ‘service degradation’
Cloudflare has said it is investigating issues with its systems reported by thousands of internet users today after it appeared to knock dozens of popular services offline.
It said: “Cloudflare is experiencing an internal service degradation. Some services may be intermittently impacted. We are focused on restoring service. We will update as we are able to remediate.”
Users were unable to load popular websites, such as X, or certain parts of the website went down. For instance, the search feature on X was continuing to experience bugs and some images were not loading.
Websites down on Cloudflare blackout
Twitter, ChatGPT and dozens of other websites have gone down suddenly or experienced issues this afternoon in an apparent problem with the IT infrastructure business Cloudflare.
You can follow our live blog for updates on the internet blackout.