~/systems // how_i_think_&_what_i_shipped

The philosophy, and the three systems that prove it

First, how I think about building things that can't fall over. Then the vehicle-data platform itself — one big round trip where jobs travel out to the fleet and telemetry travels back. Three pieces of it are mine.

// engineering_philosophy

How I think about systems

Six years of production distributed systems taught me a few things the hard way — so my systems don't have to learn them at 3 AM.

[01]

Design for failure

Every system fails; good ones fail gracefully. Rollback handling, ACID transactions, idempotency, and concurrency-safe locking are baked in from day one — not patched in after the incident.

[02]

Measure, then optimize

If it isn't on a dashboard, it's a rumor. Real-time metrics, distributed tracing, and Datadog alerts mean I find bottlenecks before users do.

[03]

Simple scales, clever breaks

SOLID principles, proven design patterns, and clear separation of concerns. The most scalable architecture is the one the whole team can reason about.

[04]

Velocity is a team sport

I mentor juniors, run structured code reviews, and partner directly with product owners — because a system is only as strong as the team that runs it.

// production_systems

Three systems, three different ways to keep me up at night

Not project descriptions — engineering stories. These three pieces of the platform are mine.

Control plane — jobs travel to the fleet
Consumer requestinternal & owner-facing
Fleet Data Collection Orchestratorcompile · target · lifecycle
WebSocket Gateway100K live conns
Vehicle fleet2M+ cars · consent-aware
Data plane — telemetry travels back
Vehicle fleetpackets & large files
Unified Data Ingestion25K req/s peak
Object storageADLS
Processingnormalized datapoints
Distribution100+ downstream use cases
N.B. Highlighted boxes are systems I've built or significantly shaped — the rest belong to brilliant neighbouring teams.
CONTROL PLANE · CONSENT-AWARE · codename: ███

Fleet Data Collection Orchestrator

2M+
vehicles per job

A consumer asks for data; this platform turns the ask into a compiled job, targets exactly the right slice of a 2M+ vehicle fleet, and delivers it — only while the car's owner says yes. Powers owner-facing features and internal product analytics alike.

What made it hard
  • Consent is a moving target — owners toggle services anytime; every job must activate, pause, and resume in lockstep, across millions of cars.
  • Fleets aren't uniform — targeting runs on device capabilities that change over time, and job updates must ripple correctly across the selected fleet.
  • Correctness under load — ACID transactions, rollback paths, and concurrency-safe locking keep job state honest through thousands of simultaneous lifecycle events.
  • Synchronous didn't scale — redesigned into an async batch pipeline on Kafka + Redis → RD-ICC Silver Award.
Java 21MicronautKafkaRedisPostgreSQLKubernetesAzure
CONNECTION LAYER · REAL-TIME · codename: ███

Vehicle WebSocket Gateway

100K
concurrent connections

Jobs are useless if they can't reach the car. This is the persistent, full-duplex WebSocket layer between cloud and fleet — holding 100,000 concurrent vehicle connections at peak without dropping a heartbeat.

What made it hard
  • 100K is not 10K, ten times — connection lifecycle, memory-per-socket, and heartbeat storms become first-class architectural problems.
  • Cars don't have office hours — reconnect behaviour and failover designed for a fleet that never sleeps.
  • Any pod, any vehicle — pub/sub routing lets a request land on any pod and still reach the vehicle through the pod holding its socket.
  • Concurrency discipline — Go's goroutines and channels keep 100K live sessions consistent without contention eating CPU, and ping/pong keep-alives tuned so idle sockets don't eat memory either.
GolangWebSocketGoroutinesPub/SubKubernetes
DATA PLANE · INGESTION · codename: ███

Unified Data Ingestion Front Door

25K/s
requests at peak

Everything the fleet says enters here — a horizontally scalable receiver absorbing 25,000 requests per second at peak, from tiny structured packets to large files arriving in chunks, landing safely in ADLS for downstream consumers.

What made it hard
  • Two payload shapes, two specialists — one service tuned for low-latency structured packets, a second for large multi-chunk file streams.
  • Chunked uploads that survive reality — networks drop, cars go through tunnels; chunk assembly and object-storage hand-off are resilient by default.
  • Scale-out, not scale-up — bursty fleet chatter is absorbed by stateless design; Kubernetes just adds replicas.
JavaRESTADLSKubernetesJMeterLoad Testing
// built in the middle of a larger organism — car-side collectors & gateways upstream, MQTT distribution alongside, datapoint processing & distribution downstream — plus QA teams and function owners. senior engineering here is mostly about the contracts between all of us.

$ cd ~ && git log --career →