kill -9 vs Your Save File: The Corruption Path You Haven't Tested

kill -9 vs Your Save File: The Corruption Path You Haven't Tested

The app's roll-history file was zero bytes long. An automated test had force-killed the app in the middle of a write, and the file had been truncated to nothing.

That was the small problem. The big problem was what happened next. Every subsequent write failed, silently, forever. The store read the corrupt file on startup, threw, and the error was caught and dropped somewhere upstream. The app looked completely fine. It just quietly stopped recording anything, and stayed that way on every launch after.

Two gaps that compounded

The "atomic" write was not atomic. The store was meant to write to a temporary file and rename it over the real one, which is the standard trick: rename is atomic on every filesystem that matters, so you get either the old file or the new one and never a half. But there was a fallback path, taken under conditions nobody had thought hard about, that wrote directly to the real file. A kill mid-write on that path means truncation.

The read path treated corruption as an exception rather than a condition. The file was garbage, the parser threw, and nothing handled it. One bad file wedged the entire store. Every write after that went through the same startup read, hit the same exception, and failed before it got anywhere.

Either gap alone is survivable. Together they produce a device that silently loses everything from the moment of one unlucky kill.

The fix

Writes only ever land via temp file plus rename. The fallback path is gone. If the rename fails, the write fails loudly, and the old file is still intact.

A corrupt file on read is quarantined: renamed aside with a timestamp, and treated as missing. The store starts fresh, and the app works. The quarantined file is still there for recovery, because it is someone's data and they might want it back.

Deliberately narrow: a file with a schema version newer than the app knows about still refuses to load. "Newer than me" is not corruption. It usually means the user ran a newer build and then downgraded, and silently discarding their data would be the wrong call.

We verified it on the wedged device itself. Next launch, it quarantined the empty file, created a fresh one, and started recording again. The quarantined file is sitting beside the new one.

Decide at design time what a reader does with garbage

Any file you persist will eventually be truncated by a process kill. Not might. Will. Phones get force-closed, batteries die, test harnesses send SIGKILL. The only question is what your read path does when it meets the result, and you want to have answered that before it happens on a customer's phone.

Three options:

  • Crash. Honest, and at least someone notices. Terrible for the user.
  • Fail silently forever. What we had. The worst of the three, because nobody notices and the damage keeps compounding.
  • Quarantine and recover. Keep the bad file, start clean, carry on. The app works, the data is recoverable, and a log line tells you it happened.

The third is barely more code than the first.

Test it by writing an empty file over your store

This is the part I would not have done without the kill-happy test harness forcing the issue. The test for this bug is trivial: write zero bytes over the store file, launch the app, see what happens. It takes a minute. It was never run, because the happy path worked and the write code looked atomic.

My assistant wrote the fix correctly and quickly. It would also have written the original code, and probably did. The difference is not the quality of the code on either side. It is whether anyone asked "what happens when this file is garbage" before the file was garbage.