Imagine your worker makes a POST to a vendor API to purchase a shipping label. It times out. This could be because the vendor has created the label and charged you, however it could also just be because the request hasn’t come through. Either way the DB will show that this job was started and no response received.

The standard answer here is to make it idempotent. But let’s say that the vendor does not accept the idempotency key. You can de-dupe the job on your side as much as you like but retrying the POST will result in another label being purchased. The lookup endpoint is also slow so if you try to find the label straight after the timeout it will return a not found.

I would mark this job in an unknown state and have another worker look up the label. If it is still unknown then send it to ops with the order ID and original request time. The issue here is you’ll have orders waiting in a queue. If the vendor has a bad afternoon then you’ll have a lot of mess to sort out manually.

Have any of you been in this situation in production? How did you decide when to stop trying to look up the label and purchase another one? And how long did you wait before purchasing another label? Did this change as you reached a high enough volume that having ops check it wasn’t an option?