The retry loop that traded twice
For three days my reconciler kept re-queuing exits a human had already placed by hand. The machine's audit log was honest — it just wasn't the whole truth — and one order finally executed twice.
A reconciliation job in my execution system had one sensible mandate: find orders that were attempted but never confirmed filled, and re-queue them so nothing falls through the cracks.
For three consecutive days it re-queued exits that had already happened.
The missing context: because of a broker entitlement gap, some sells in this book get placed by hand — a human at the terminal, an OTP, done. Those fills are real, they appear in the broker’s own order book, and they leave no line in the machine’s audit log, because the machine didn’t place them. The reconciler’s worldview was its own log. Attempt recorded, no fill recorded: incomplete — retry. Every day the human’s work was invisible, and every day the machine re-litigated it. On day three, a cash-management redemption cleared twice.
One truth, two writers
The root cause wasn’t the retry logic. It was an unexamined assumption underneath it: the machine’s log is the complete record of trading activity. That holds only if the machine is the sole actor. The moment two writers touch the same account — human and runner — any log that only one of them writes is a partial ledger, and every decision derived from it inherits the gap.
The bank-reconciliation instinct applies exactly. Your cash book is not the truth; the bank statement is the external record you reconcile to. Here, the broker’s order book is the bank statement. The fix followed directly:
- Verify “failed” against the external record, not the internal one. Before re-queuing anything, the reconciler now checks the live broker book. An order sitting there with no audit line doesn’t mean the log lies — it means a human traded.
- Dedup against executed siblings. A retry candidate whose economic twin — same instrument, same side, same intent — already shows as executed is not a gap. It is a fill wearing different provenance.
- A sell against a flat book resolves, never retries. If the position is already gone, the exit’s purpose is achieved, whoever achieved it. Retrying a sell into a zero position is how you go short by accident.
- Cap retries per order signature. Even correct logic deserves a circuit breaker for the failure mode nobody predicted yet.
The boundary that fell out of it
The deeper reorganisation: the decision layer — the thing deciding what the book should hold — no longer reads execution states at all. No fill logs, no rejection lists, no re-litigating yesterday’s attempts. It re-derives its decisions from the live book as it stands, and hands new intent to the queue. Fills, failures, retries and reconciliation belong to deterministic machinery on the other side of a line.
The reasoning is about failure containment. A brain that reads execution debris will eventually reason from it — and execution debris is exactly where partial ledgers, phantom retries and stale states live. Give the brain the current state of the world and nothing else, and an entire class of compounding errors loses its entry point.
The habit I kept from it is blunt: for every log a system writes, ask who else can change the thing this log describes. If anyone can, the log is a diary, not a ledger — and nothing that trades money should take instructions from a diary.