The Hidden Cost of Over-Engineering: A Post-Mortem on Our Rewrite

Eighteen months ago, we decided to rewrite our core processing service. The existing service was a three-year-old Python monolith that worked fine. It handled our load with room to spare. But we convinced ourselves that we needed to prepare for “10x scale,” and that the right architecture to handle that was a sophisticated microservices system built in Go with event-driven communication.

Eight months later, we had a system that was genuinely impressive — and genuinely wrong for our needs.

What We Actually Built

We built for a scale we hadn’t reached and might never reach. The new system had better theoretical throughput. It also had three times the operational complexity, twice the infrastructure cost, and significantly higher latency for the 99th percentile of requests due to network overhead between services that could have been function calls.

What We Should Have Done

Profile the existing system first. We had assumed performance problems we had never measured. When we finally ran proper load tests on the original service, it handled 4x our current peak load without breaking a sweat. We rewrote something that didn’t need rewriting.

The Rule We Now Follow

Optimize for your current scale, design for 2x your current scale, and build for the problems you actually have. Over-engineering is a form of speculation — and speculation with engineering time has the same expected value as any other uninformed speculation.