Read-only file systems for some Repls

Resolved
Resolved

Hello – Over 98% of impacted Repls have returned to normal functionality and we are making progress on the remaining 2%.

We are moving to consider the incident resolved, but if you are part of that small group with a still impacted Repl and to make sure we are aware, please let us know by reaching out to support@replit.com.

We genuinely appreciate everyone's support. Thank you.

Updated

We have recovered a large portion of the remaining affected Repls. We anticipate a longer, more in-depth, update likely tomorrow morning.

A full postmortem will also be published in the coming days, on our blog. We will update here once it is available.

Updated

Work continues on remaining affected Repls.

So far over 90% of total affected Repls have recovered and progress on the remainder continues. We anticipate that progress to continue as the current pace for the time being.

Identified

Full read and write access has now been restored to 90% of impacted Repls in US regions and 68% of impacted Repls in Asia regions.

Next update will be Monday morning Pacific Time (UTC-7).

Recovering

We have recovered most of the affected Repls and are continuing to make progress.

Updated

Restoration is ongoing. More files should be restored throughout the day.

Our engineers are diligently working round the clock to keep recovering more.

Check back later for more updates.

Updated

We have restored read/write filesystems to 86% of the impacted Repls, which includes:

  • 88% of impacted Repls in the US-based regions
  • 60% of impacted Repls in the Asia-based regions

We have all hands on deck and are continuing to restore write access to the remaining Repls, working on all regions in parallel. We'll provide another update around 10 AM PT.

Updated

After monitoring filesystem restoration overnight, we are swapping in a fresh team to focus on the remaining Repls. We'll have more details around when to expect final incident resolution as soon as that information is available, and continue to prioritize a smooth recovery. Thank you for your patience until we provide another update in the next two hours.

Updated

The team continues to make progress! Around 70% of impacted Repl filesystems have now been fully restored. We anticipate our next update will be in the morning Pacific time.

Updated

We are continuing to restore impacted Repl filesystems.

Over half of the affected Repls should be back to normal at this point. Expect another update in the next 2 hours.

Updated

The restoration of impacted filesystems continues. No significant updates on our end. Will share another update in 60-90 minutes.

Updated

Restoration of impacted file systems continues. No significant updates to share.

Updated

Progress continues recovering remaining Repls. Next update in an hour.

Updated

Hello – we would like to give a longer update on where we are:

We are currently recovering from a portion of Repls having their file system go read-only. The vast majority of Repls were and are unaffected and a majority of affected Repls have recovered.

However, there is a portion that are still in recovery. Those Repls will exhibit a wide-range of symptoms like I/O issues, problems opening the Repls, and other errors.

Replit has all hands on deck focusing on recovering remaining Repls. Progress is being made, however, we are not able to provide any estimates to full resolution.

Regular updates here will continue.

Updated

A majority of affected Repls have now been recovered. We have all hands on deck working towards recovery for the rest.

Updated

We are still experiencing an incident with Repls and errors in the workspace. Recovery is ongoing.

Updated

We are working on scaling up and speeding up our recovery process

Avatar for
Identified

We are running recovery jobs for all affected filesystems. Next update in an hour.

Avatar for
Updated

Status remains the same. Next update expected tomorrow, Saturday Feb 20th in the morning Pacific Time (UTC -7)

Thank you for patience.

Updated

We are continuing to recover affected Repls. Next update in an hour.

Updated

A majority of affected Repls have recovered, with the remaining portion are still undergoing recovery. Please continue holding off forking any affected Repl or template.

Recovering

We have begun recovery with about half affected Repls returning to functional.

Forking a currently affected Repl or template will not succeed – if you know your Repl is affected, please hold off until it has recovered.

Updated

Some Repls are not loading files or displaying the error message, "Your Repl is having a bit of trouble."

We are investigating and actively recovering those file systems.

Updated

We are continuing to address Repls with affected file systems. We do not have an estimated time to resolution but will continue to update here.

Identified

We are aware of some Repls that have their file system become read-only. We are investigating and actively recovering those files systems.

We will update as we progress.

Began at:

Affected components
  • Repls
    • Asia Cluster
    • North America Cluster