The Principal Craft
Field Notes on Principal-Level Engineering: Architecture, Distributed Systems, and Production Engineering.
Autopsy of a Hybrid Architecture (2/4): The Temporal Rift in Reactive Cores
Why do retries fail with 401 Unauthorized even when your logic seems perfect? Dissecting the dissociation between Assembly Time and Subscription Time.
Autopsy of a Hybrid Architecture (1/4): The Illusion of Non-Blocking & Thread Starvation
When 5% CPU utilization meets 503 Service Unavailable. Dissecting the cost of waiting and the quantitative failure of hybrid execution models.
Autopsy of Distributed Systems (4/4): The Integrity Fallacy & The DLQ Mandate
A queue with zero lag can still be a crime scene. We conclude the series by dissecting the ‘Catch-and-Commit’ anti-pattern and establishing why a Dead Letter Queue is an architectural mandate, not a luxury.
Autopsy of Distributed Systems (3/4): The Protocol Trap & The Lease Paradox
Distributed brokers use leases to detect life. We dissect how the ‘MAX_VALUE’ configuration trap disables a system’s autonomic immune response, turning transient failures into permanent paralysis.
Autopsy of Distributed Systems (2/4): The Concurrency Collapse & Reactive Starvation
Asynchronous frameworks do not eliminate latency; they merely shift the bottleneck. We dissect how a single blocking I/O call inside a reactive pipeline can trigger a catastrophic thread starvation event.
Autopsy of Distributed Systems (1/4): The Observability Mirage & Zombie Consumers
When a dashboard reports 100% health but throughput is zero, the system isn’t dead—it’s paralyzed. We dissect thread starvation, the illusion of structural health checks, and the math behind a system collapse.
From Contention to Throughput (5/5): The Grand Finale — Database / Queue vs Kafka vs Workflow Engines
The ultimate decision framework. We resolve the holy war between Database / Queue primitives, Event Streams, and Workflow Engines by defining state lifespan and coordination boundaries.
From Contention to Throughput (4/5): Scaling Beyond a Single Database — Partitioning & The Connection Collapse
When a single PostgreSQL instance hits its physical limits, we must shard. We explore the Connection Collapse trap, multiplexing proxies, and intelligent shard routing.
From Contention to Throughput (3/5): Designing for Failure — Lease Systems & Distributed Recovery
When SKIP LOCKED is not enough. We explore the State Model, the Law of Temporal Ownership, and how Fencing Tokens prevent distributed split-brain.
From Contention to Throughput (2/5): Turning PostgreSQL into a Lock-Free Queue — The SKIP LOCKED Pattern
How to resolve massive database lock contention without deploying Kafka. We explore Quantitative Capacity, the Law of Non-Blocking Acquisition, and the SKIP LOCKED paradigm.