One retry step per side effect
Moving a side effect into a retried job turns every “send to all” into a duplicate risk and every payload into stale data: make each send its own retry step and re-check state when the job runs, not when it was queued.
Read the lesson →Release a claim when the work fails
A “claim” flag that guards a side effect must be released when the side effect fails; otherwise the retry that is meant to recover the work is the thing that skips it.
Read the lesson →Finishing a leased job must check the lease
A job lease is only safe if finishing the job checks that you still hold it: otherwise the slow worker you already replaced wakes up and overwrites the new worker's result.
Read the lesson →Never advance a cursor past pending work
One “read up to N” number cannot describe a channel where some records are handled immediately and others wait: the cursor must never pass the oldest record that still owes a delivery.
Read the lesson →Redis hash tags don’t make KEYS cheap
A Redis hash tag decides which node holds a key; it does not make KEYS cheap, so keep an index or counter for any key family you need to enumerate.
Queue dedup ignores jobs already running
A queue's “deduplicate by job id” usually only covers jobs still waiting. Once a job is active, a second trigger needs your own in-flight check, or it becomes a duplicate run.
Read the lesson →Self-requeueing jobs need their own limit
A job that re-enqueues itself needs its own attempt counter in the payload: a new job starts with fresh retries, so the queue's retry limit no longer protects you from a batch that always fails.
Read the lesson →