Tidewater Payments · 2023 — 2024
01// ledger
An idempotent payment ledger
Replacing update-in-place balances with an append-only ledger, without pausing a platform that settles payments for 9,000 merchants.
problem
Merchant integrations retried on timeouts, and so did our own workers. Each retry could write a second ledger entry, and because balances were updated in place there was no clean history to reconcile against. The finance team fixed around 30 duplicates by hand every week, and every fix was another in-place update.
approach
- Every write carries an idempotency key scoped to merchant, operation and the client's own key. The same key always returns the same receipt.
- Entries are append-only. Balances became projections that can be rebuilt from the log at any point in time.
- Ten weeks in shadow mode. The new ledger received every write and a nightly job compared balances with the old one.
- Cut over cohort by cohort behind a per-merchant switch, smallest merchants first, with a one-command rollback.
code
func (l *Ledger) Post(ctx context.Context, e Entry) (Receipt, error) {
key := idemKey(e.MerchantID, e.Operation, e.ClientKey)
tx, err := l.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return Receipt{}, err
}
defer tx.Rollback()
// Same key, same request: hand back the original receipt.
if r, ok, err := l.findReceipt(ctx, tx, key); err != nil || ok {
return r, err
}
if err := l.appendEntries(ctx, tx, key, e.Legs()); err != nil {
return Receipt{}, fmt.Errorf("append %s: %w", key, err)
}
return l.commitReceipt(ctx, tx, key)
}
outcome
| metric | before | after | unit |
|---|---|---|---|
| Duplicate entries | 31 | 0 | |
| Reconciliation time | 26 | 2 | h |
| p99 write latency | 48 | 35 | ms |