~/ines/projects

Three case studies

Longer write-ups of the work behind the bullet points: what was broken, what I changed, the code at the centre of it and what happened next. Numbers are from production dashboards.

A long row of dark blue server cabinets in a data centre aisle Tidewater Payments · 2023 — 2024

01// ledger

An idempotent payment ledger

Replacing update-in-place balances with an append-only ledger, without pausing a platform that settles payments for 9,000 merchants.

role
Tech lead, team of 5
timeline
14 months
stack
Go, PostgreSQL, Kafka
scale
€2.1B settled

problem

Merchant integrations retried on timeouts, and so did our own workers. Each retry could write a second ledger entry, and because balances were updated in place there was no clean history to reconcile against. The finance team fixed around 30 duplicates by hand every week, and every fix was another in-place update.

approach

  1. Every write carries an idempotency key scoped to merchant, operation and the client's own key. The same key always returns the same receipt.
  2. Entries are append-only. Balances became projections that can be rebuilt from the log at any point in time.
  3. Ten weeks in shadow mode. The new ledger received every write and a nightly job compared balances with the old one.
  4. Cut over cohort by cohort behind a per-merchant switch, smallest merchants first, with a one-command rollback.

code

ledger/post.go Go
func (l *Ledger) Post(ctx context.Context, e Entry) (Receipt, error) {
  key := idemKey(e.MerchantID, e.Operation, e.ClientKey)

  tx, err := l.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
  if err != nil {
    return Receipt{}, err
  }
  defer tx.Rollback()

  // Same key, same request: hand back the original receipt.
  if r, ok, err := l.findReceipt(ctx, tx, key); err != nil || ok {
    return r, err
  }
  if err := l.appendEntries(ctx, tx, key, e.Legs()); err != nil {
    return Receipt{}, fmt.Errorf("append %s: %w", key, err)
  }
  return l.commitReceipt(ctx, tx, key)
}
The whole write path in one transaction. A replayed request finds its receipt and returns before touching the ledger.

outcome

0
duplicate charges in 14 months, from 1 in 40k
−92%
hours spent on manual reconciliation
10wk
in shadow mode with zero balance drift
before → after, per week
Ledger metrics before and after the migration
metricbeforeafterunit
Duplicate entries310
Reconciliation time262h
p99 write latency4835ms
A steel electricity pylon carrying power lines against a clear blue sky Northgrid Energy · 2019

02// ingest

Smart-meter ingest on plain Postgres

Absorbing 1.4 million meters and a quarter-hourly burst of 38,000 readings a second with the database the team already knew.

role
Lead backend engineer
timeline
7 months
stack
Go, PostgreSQL, gRPC
scale
38k readings/s peak

problem

All readings lived in one 9 TB table. Vacuum could not keep up, dashboards timed out every quarter hour, and deleting old data took a nine-hour job that blocked writes. The alternative on the table was a new time-series database and a year-long migration.

approach

  1. Daily range partitions on reading time, created two weeks ahead by a scheduled job.
  2. A batching writer that buffers per partition and writes with COPY, pushing back on the gateway when the buffer fills.
  3. BRIN indexes for time ranges, which are tiny for append-mostly data, plus one B-tree for per-meter lookups.
  4. Retention by dropping partitions, so removing a day of data takes milliseconds.

code

migrations/0042_readings.sql SQL
CREATE TABLE readings (
  meter_id  bigint        NOT NULL,
  read_at   timestamptz   NOT NULL,
  kwh       numeric(12,4) NOT NULL,
  quality   smallint      NOT NULL DEFAULT 0
) PARTITION BY RANGE (read_at);

-- one partition per day, created 14 days ahead by a job
CREATE TABLE readings_2019_06_01 PARTITION OF readings
  FOR VALUES FROM ('2019-06-01') TO ('2019-06-02');

-- BRIN suits append-mostly, time-ordered rows
CREATE INDEX ON readings USING brin (read_at) WITH (pages_per_range = 32);
CREATE INDEX ON readings (meter_id, read_at DESC);

-- retention is a metadata change, not a nine-hour DELETE
DROP TABLE readings_2017_05_31;
The migration that replaced the single table. Old data moved over in the background, one day at a time.

outcome

−46%
storage cost after compression and retention
38k/s
sustained peak without dropped readings
40ms
to drop a day of data, from nine hours
before → after
Ingest metrics before and after partitioning
metricbeforeafterunit
Storage9.14.9TB
p95 dashboard query2300180ms
Vacuum per day141h
Gantry cranes loading stacks of shipping containers onto a cargo ship Parcelwise · 2017

03// ci

CI from 41 minutes to 9

A faster pipeline turned weekly release trains into same-day deploys for a team of 22 engineers.

role
Platform engineer
timeline
3 months
stack
TypeScript, Docker, Jest
scale
180 merges a week

problem

Every change rebuilt every package and ran 6,800 tests on one machine. A green run took 41 minutes, flaky tests made one run in five go red for no reason, and people started batching changes to avoid the wait, which made each release riskier.

approach

  1. A build graph from the package manifests, so a change only rebuilds what depends on it.
  2. Content-hash caching of build outputs and dependencies, shared between branches.
  3. Test sharding by recorded timings across eight runners, balanced so no shard waits on another.
  4. A flaky-test quarantine where each test has an owner and an expiry date.

code

ci/shard.ts TypeScript
// Split test files into N shards of similar total duration.
type Timing = { file: string; ms: number };

export function shard(timings: Timing[], shards: number): string[][] {
  const buckets = Array.from({ length: shards }, () => ({
    ms: 0,
    files: [] as string[],
  }));

  // Longest first, each into the currently lightest bucket.
  for (const t of [...timings].sort((a, b) => b.ms - a.ms)) {
    const lightest = buckets.reduce((min, b) => (b.ms < min.ms ? b : min));
    lightest.files.push(t.file);
    lightest.ms += t.ms;
  }
  return buckets.map((b) => b.files);
}
Greedy longest-first scheduling. Timings come from the previous green run on the main branch.

outcome

9min
median pipeline, from 41 minutes
daily
releases, from a weekly release train
−83%
reruns caused by flaky tests
before → after
CI metrics before and after the pipeline work
metricbeforeafterunit
Median pipeline419min
p90 pipeline5813min
Reruns per week6411

// want the long version of any of these?

Ask me about it
↑↓ move↵ openesc close