Invental/Review lessons/Writes that span systems

8 lessons · 9 checks

Writes that span systems

One user action often writes to a database, object storage and a third-party API. There is no transaction across all three, so the order of the steps decides what a crash leaves behind.

Major

Say what state each failed step left behind

A rollback is only as good as its error messages: each failure point in a multi-step remote change must say exactly what state it left, or the user's “fix” makes things worse.

Read the lesson → Major

A claim across a row, cache and queue needs undo

When a state change spans a database row, a cache and a queue, write the durable row first, make the claim tenant-scoped, and give every step after the claim a way back to the previous state.

Read the lesson → Major

Dry runs inherit every side effect

A “dry run” endpoint inherits every side effect of the real path unless hooks can tell it's a dry run, and it can still say “valid” for data the database will reject (unique constraints). Design both into the API on day one.

Read the lesson → Major

Close the streams you hand to other code

Every time a pipeline passes a stream to code it doesn't control, it owns closing that stream when that code throws. A codemod should never remove the old thing before the new thing is in place.

Read the lesson → Major

Stage, commit, then delete old files

When a write spans a database and object storage, order it so every crash point leaves a recoverable state: stage the new object, commit the row, then delete the old object after the commit. Never delete before the commit.

Read the lesson → Major

Usage metering must never fail the request

Metering and audit writes are side effects of a request, not part of its result: if recording usage can throw, a successful (and already paid-for) model call can reach the user as a failure, or a DB blip can hide the real error.

Read the lesson → Major

Run every rejecting check before any write

Do every check that can reject a target before you write anything for that target; otherwise a retry against a different target succeeds and leaves the first one with an orphaned “in progress” record that nothing will ever finish.

Read the lesson → Major

Re-check the owner after a remote create

When creating a remote resource takes time, the owner can disappear in the meantime. Re-check that the owner still exists after the create returns, and undo the create if not. A disconnect hook can't delete what it can't see yet.

Read the lesson →

The checks, with the lesson behind each

When step two of a two-step change to an external system fails, does your app tell the user what state step one left?+
A rollback is only as good as its error messages: each failure point in a multi-step remote change must say exactly what state it left, or the user's "fix" makes things worse.
For each compare-and-set in your app, does the WHERE clause include tenant/workspace, and what undoes it if the next step fails?+
When a state change spans a database row, a cache and a queue, write the durable row first, make the claim tenant-scoped, and give every step after the claim a way back to the previous state.
If your app has a "preview / validate / dry run" mode, do your side-effecting hooks know they're in it?+
A "dry run" endpoint inherits every side effect of the real path unless hooks can tell it's a dry run, and it can still say "valid" for data the database will reject (unique constraints). Design both into the API on day one.
When a file-processing step throws halfway, does your code close the download or storage stream it opened?+
Every time a pipeline passes a stream to code it doesn't control, it owns closing that stream when that code throws. A codemod should never remove the old thing before the new thing is in place.
If your server crashes between the file upload and the database save, which of the two is left pointing at nothing?+
When a write spans a database and object storage, order it so every crash point leaves a recoverable state: stage the new object, commit the row, then delete the old object after the commit. Never delete before the commit.
If your usage/billing table is down for five minutes, do AI requests still succeed and still show the real provider error?+
Metering and audit writes are side effects of a request, not part of its result: if recording usage can throw, a successful (and already paid-for) model call can reach the user as a failure, or a DB blip can hide the real error.
If an import fails in one destination and the user retries in another, can the first destination end up with a half-created record?+
Do every check that can reject a target before you write anything for that target; otherwise a retry against a different target succeeds and leaves the first one with an orphaned "in progress" record that nothing will ever finish.
If a user disconnects an integration in the second your app is still setting it up, is anything left running on the third-party side?+
When creating a remote resource takes time, the owner can disappear in the meantime. Re-check that the owner still exists after the create returns, and undo the create if not. A disconnect hook can't delete what it can't see yet.

More checks from the same reviews

Yes/no questions from other reviews on this theme. Each one is quick to answer for your own app.

  1. If one record in a list call is deleted mid-request, does your page still load the other 49?

Want a senior reviewer to run these checks on your code? Ongoing pull-request review capacity →

Want these checked on your code?

We run reviews like these on real codebases. Tell us what you are building and we confirm scope before anything starts.

— Get in touch
hi@invental.co ↗
— Or
— What to include

A link or short description of the code or system, what you want checked, your stack, and when you need the answer. No repository access is needed until scope is agreed.

— Or leave a note
We reply within one business day.