[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Tarsnap outage 2023-07-02

The important information first: All data stored on Tarsnap is safe, but you
can't access it right now.  I'm working to bring it back online.

The detailed story:

At approximately 2023-07-02 13:07 UTC the main Tarsnap server suffered a
"failed system status check" in Amazon Web Services vernacular.  In plain
language, something broke -- either the server hardware, the power, or its
connection to the rest of the EC2 network (including network-attached disks).

At approximately 2023-07-02 13:52 UTC the server rebooted (probably running
on new hardware, considering that it's a virtual machine), but had filesystem
corruption affecting cached metadata.  This cached metadata essentially tells
the Tarsnap server where to find user data in Amazon S3; without it, the
Tarsnap server cannot service requests.  I *think* that the metadata present
is accurate as far as it goes, but it definitely lacks information about
archives stored in the last few seconds before the outage started.

While I could probably "replay" the most recently stored data from the backing
store in S3, the nature of filesystem corruption means that I can't guarantee
that this would work; since Tarsnap is a backup service I have to prioritize
safety, and so I've started the process of reconstructing the Tarsnap server
state from the "known good" data stored in Amazon S3.

If anyone needs data urgently, I can bring the Tarsnap server up in read-only
mode, subject to the caveat that the most recently stored data may not be
available.  Please contact me if this would be useful to you.

It's too early in the state reconstruction process to provide an accurate ETA;
my best guess is somewhere between 12 and 24 hours.

As always, feel free to email me if you have any questions.

Sorry about the headaches,
Colin Percival
FreeBSD Deputy Release Engineer & EC2 platform maintainer
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

Attachment: OpenPGP_signature
Description: OpenPGP digital signature