Roblox Outage - Worst Outage Yet?

Noah · 2021-10-31T22:15:00.780Z

NO

The webpage doesn’t load anymore…

Its still a bit iffy so it should be fixed soon.

Noah · 2021-10-31T22:49:13.688Z

It works again, it seems, how interesting…

OH20_rbLX · 2021-11-01T01:43:59.794Z

Roblox is now officially back online. Also I was on vacation for the entire weekend so I didn’t really check the website status.

Noah · 2021-11-01T01:54:57.777Z

Alright, everyone should now have access to roblox.

image1920×46 4.35 KB

In the banner roblox has said it should slowly come back, roblox will be a bit sluggish for the first couple hours/days, but then it will regain it’s strength again!

xVoid_xyz · 2021-11-01T03:00:00.650Z

Yep everyone is now able to access roblox now.

Noah · 2021-11-01T11:12:34.461Z

Roblox Blog – 31 Oct 21

An Update on Our Outage - Roblox Blog

We recently experienced an extended outage across our platform. Learn more about the outage and steps taken to address it.

Official post regarding the situation…

xVoid_xyz · 2021-11-01T12:31:40.392Z

Yep, just saw that. It’s sad.

OH20_rbLX · 2021-11-01T20:07:58.356Z

I think it’s time for Roblox to improve their servers instead of just fixing it and letting it go.

FriX_0z · 2021-11-02T09:53:45.526Z

Me! It works perfectly, it Should work for you. Try downloading it again

FriX_0z · 2021-11-02T09:54:04.049Z

Mhm, because roblox didn’t expect ALL servers down.

Noah · 2021-11-17T20:36:41.526Z

FriX_0z:

Me! It works perfectly, it Should work for you. Try downloading it again

Downloading it again, that shouldn’t really make a difference unless the app is corrupted on the local device.

OH20_rbLX · 2021-11-17T23:01:29.539Z

Noah:

shouldn’t really make a difference unless the app is corrupted on the local device.

Yeah, unless you delete related files (or from %appdata% on Windows 10/11)

Noah · 2022-01-22T18:50:21.179Z

For those wondering, here is an official statement on what happened.

In a short summary this was what caused the issues:

The outage lasted 73 hours.
The root cause was due to two issues. Enabling a relatively new streaming feature on Consul under unusually high read and write load led to excessive contention and poor performance. In addition, our particular load conditions triggered a pathological performance issue in BoltDB. The open source BoltDB system is used within Consul to manage write-ahead-logs for leader election and data replication.
A single Consul cluster supporting multiple workloads exacerbated the impact of these issues.
Challenges in diagnosing these two primarily unrelated issues buried deep in the Consul implementation were largely responsible for the extended downtime.
Critical monitoring systems that would have provided better visibility into the cause of the outage relied on affected systems, such as Consul. This combination severely hampered the triage process.
We were thoughtful and careful in our approach to bringing Roblox up from an extended fully-down state, which also took notable time.
We have accelerated engineering efforts to improve our monitoring, remove circular dependencies in our observability stack, as well as accelerating our bootstrapping process.
We are working to move to multiple availability zones and data canters.
We are remediating the issues in Consul that were the root cause of this event.