This page is an archive of the incident that occured on 2026-05-10. The incident has since been resolved, but this page is staying up for archival purposes.

Server outage

The dedicated server hosting the Iceshrimp infrastructure has experienced a catastrophic disk failure.

All data is safe, and we're working to restore service as quickly as possible.

Apologies for any inconvenience caused.

- zotan

Incident timeline

• 2026-05-10 03:51 CEST - Server crashes due to disk failure, all services go down.
• 2026-05-10 11:00 CEST - Investigation starts. Disk failure identified as cause of the downtime.
• 2026-05-10 11:51 CEST - Hosting provider has reconnected the affected drive and it appears to be working again, an offsite backup is running.
• 2026-05-10 12:04 CEST - The backup has completed, and all data is safe, even if the drive dies fully now.
• 2026-05-10 12:12 CEST - Steps are being taken to ensure a single disk failure cannot take down the server in the future.
• 2026-05-10 12:38 CEST - The file system is being migrated to ZFS, as I have lost trust in the crash safety of BTRFS.
• 2026-05-10 16:52 CEST - Infrastructure is back online. DNS may still take a couple minutes to sync, but everything should be reachable again.

Resources

• The Arch Linux repository is online and can be found here.