Transaction State Machines for Multi-Chain Custody: What Actually Breaks
Most wallet backends I have worked on keep a transaction status column. Few of them keep a transaction state machine. You notice the difference the first time a chain does something the happy path did not anticipate, and on real chains that happens about once a week.
A status column answers "what is it now". A state machine answers three harder questions: which transitions are legal, what triggers them, and what happens when the world moves backwards. In custody, where a single transition can move client money, you want all three answered before you trust the system with a withdrawal.
The states you need
The shape I have seen survive contact with BTC, EVM chains and Solana looks roughly like this:
CREATED → POLICY_APPROVED → SIGNED → BROADCAST → PENDING →
CONFIRMED(n) → FINALIZED
with exits to FAILED, REPLACED and EXPIRED,
and one uncomfortable arrow I keep on purpose: CONFIRMED → PENDING, for reorgs.
A few of these deserve defending.
POLICY_APPROVED sits before SIGNED as its own state. The policy engine evaluates limits, allowlists and quorum before anyone touches a key, and you record the approval as its own transition with its own audit entry. If you fold policy and signing into one step, you cannot prove to an auditor which rule approved a transfer, and you cannot fail closed cleanly when the policy engine is down.
BROADCAST and PENDING are different states. Broadcast means "we handed the transaction to a node". Pending means "the network acknowledges it exists". Between the two, transactions get dropped with no error: the node was behind, the mempool evicted it for low fees, the RPC provider accepted it and lost it. If your system treats broadcast as success, these transactions vanish from your books while the signed bytes still exist and can confirm later, which is how you end up paying one withdrawal twice.
CONFIRMED is parameterized and FINALIZED is policy. Confirmations accumulate differently per chain: depth on Bitcoin, finalized checkpoints on Ethereum, commitment levels on Solana. FINALIZED is where your business logic says "this can never come back". That threshold is a per-chain policy decision, so keep it in reviewed configuration where an auditor can see it, rather than in an engineer's head.
REPLACED is a real state rather than an error. Fee-bumping (RBF on Bitcoin, same-nonce replacement on Ethereum) produces a new transaction that competes with the old one. Both exist until one confirms. Model the pair, or your monitoring will report a phantom failure every time a fee bump does its job.
Backwards transitions: reorgs happen
The transition teams forget is the one that goes backwards. A block containing your confirmed transaction can stop being part of the canonical chain. On fast chains with probabilistic finality this is routine; on Bitcoin it is rare but priced in; on Ethereum it is bounded by the finalized checkpoint. When it happens, you walk the transaction back to PENDING, you reverse any ledger effect with a compensating entry (append-only ledgers make this provable, mutable balance columns make it archaeology), and you page someone.
Idempotency is the other half of the design
Most of the real incidents I have seen were retries rather than exotic chain behaviour: a timeout on broadcast, an operator clicking again, a queue redelivering a message. Each of those has to be safe, which means:
- Transitions are idempotent. Applying "mark as broadcast" twice results in one state change and one audit entry.
- Externally-triggered actions carry an idempotency key, so a withdrawal request that arrives twice, through any path, produces one transaction. You dedup at the boundary, on a key the client controls, stored with the transaction itself.
- Signing is exactly-once by construction. You persist the signed payload before broadcast, and a retry re-broadcasts the same bytes without re-signing. Re-sign on retry and you get two valid competing transactions; on UTXO chains that means competing spends of the same inputs.
Where each chain bends the model
On Ethereum, the nonce queue bends it. Transactions from one address confirm in nonce order, so one underpriced transaction blocks everything behind it and "stuck" is contagious. The state machine needs a per-account view on top of the per-transaction one: remediating transaction N gets you nowhere if N-1 is the one that is stuck.
On Solana, transactions expire. A transaction references a recent blockhash and dies after roughly 150 slots if unconfirmed. EXPIRED becomes a first-class terminal state, and the retry path builds a new transaction with a fresh blockhash. For offline or slow signing flows, durable nonces exist because signed transactions cannot wait around.
On Bitcoin, nothing is yours until it confirms. Mempools are local opinion, eviction is silent, and replacement is a feature. The PENDING state carries an age timer, and remediation (fee-bump or child-pays-for-parent) is a transition your policy defines in advance, so ops runs a playbook instead of improvising.
The audit trail belongs in the database
In a regulated environment, each transition needs: previous state, new state, trigger (who or what), timestamp, and the evidence (block hash, node response, policy decision id). Store all of that as immutable, queryable data. When a client asks why a withdrawal took forty minutes, or an auditor asks who approved it, you answer with a query instead of grepping application logs that rotated last month.
What breaks in production
The failures I keep meeting are the same ones. A team treats broadcast as done, and the transaction vanishes from the mempool and from their books at the same time. They skip the REPLACED state, so fee-bumps look like failures and monitoring cries wolf until people stop listening. They re-sign on retry instead of re-broadcasting persisted bytes. On Ethereum they think per-transaction and ignore the nonce queue; on Solana they have no EXPIRED state, so transactions rot in PENDING forever. They credit on CONFIRMED with one global threshold for every chain, and they leave out the backwards transition for reorgs, so reconciliation finds the hole weeks later.
Each of these is a whiteboard fix, until the money has already moved.