Serving a billion web requests with boring code

In the late 2010's, my startup built a messaging app with millions of daily users sending and receiving >1B messages a month, (somewhere north of 300 messages/sec, IIRC) with a trailing 12 month average uptime of our systems of 99.995%

When we were going through due diligence as part of being acquired by a large tech company, I had to answer a lot of questions about our infrastructure and to "prove" that our boring tech stack could actually achieve this.

That "stack" was 3 pods (2 in one AWS region, 1 in another) comprised of 4 Linux boxes and 3 Windows boxes.

In a stack, Linux boxes were 2 monitoring servers and 2 app servers running an OSS SW package, while the Windows boxes were two app servers running our .NET APIs, and a SQL server.

Fail over was handled by SQL cluster at DB level and ELB at the app level.

While the stack and code were boring, a lot of work went into performance, reliability, and observability.

After we were acquired, my new infra budget was based on their own modeling for scale, resulting in a monthly AWS budget that was 10X my previous annual budget.

I wonder if it’s possible for this to get resubmitted, or given another chance. We often see these “boring” tech write ups, but very rarely applied to something like healthcare.gov.

With hindsight, I’m curious which of the other choices made would have changed the most with a multi page architecture, and have you discovered a cleaner way of avoiding it? Struggling with something similar at the moment.

Talks about boring code, then immediately introduces a completely modern stack that most developers would love to use including grpc, gateways, rds, s3, ec2, testing, separated services & databases, mocking frameworks...

There's nothing boring about this, it's unfortunate that they took to name this "boring code".

Aside from this, the actual implementation details are quite interesting and worth a read. A better title would be "lessons learned using a modern stack for healthcare.gov"

Using such new technology seems anything but “boring” to me.

React was only 5-months old, and Go was only 4-years old, when healthcare.gov was launched.

First released:

- Go (Nov 2009)

- React (May 2013)

- Healthcare.gov (Oct 2013)

https://en.m.wikipedia.org/wiki/Go_(programming_language)

https://en.m.wikipedia.org/wiki/React_(JavaScript_library)

https://en.m.wikipedia.org/wiki/HealthCare.gov

EDIT:

I missed this in the article:

> There are many valid criticisms of react; this piece is an example, and I was aware of the issues already in 2018 when I was building the site.

How is that a billion?

5MM a day is 150MM a month.

Is this a year? That's 1.3B

I generally measure the per second rate. Per millisecond can be mathed quick enough for conversational calculations.

A better title would be "lessons learned using a modern stack for healthcare.gov"

Very interesting post, thanks for sharing.