We Built a Monitoring Platform So Ten Teams Would Stop Reinventing the Same Broken Alarm Clock

Every engineer has, at some point, said the sentence: “It’s fine, we’ll just run a script every 5 minutes to check if the thing is healthy.”

That sentence is a trap. It is the “how hard could it be” of backend engineering. Because the script is the easy 5%. The other 95% is a small mountain of questions nobody wants to answer twice:

• Where does this thing run?
• Who schedules it?
• Where do the results go?
• Where do the logs go, and how does anyone find them at 2 AM?
• How do we know if it’s actually broken, or just having a bad millisecond?
• Who gets paged, and when do we stop paging them?

Multiply that by ten engineering teams, each quietly building their own snowflake version of the same thing, and you get ten slightly different flavors of “works on my dashboard.” That’s the problem we set out to fix — not by writing a better test, but by never making teams write that infrastructure again.

The Idea: Bring Your Own Test (BYOT)

The platform is built around one rule, and the rule is almost suspiciously simple:

Teams bring the test. The platform handles literally everything else.

We don’t care what language you wrote your test in. We don’t care about your framework opinions. A team hands over a test package plus some configuration — which environment to hit, how often to run, a few knobs — and from that point on, they never have to run it manually again.

In other words: we turned “please remember to run your health check” into “please never think about this again,” which is the correct emotional target for any piece of infrastructure.

The Architecture, In Broad Strokes

At a high level, a request comes in through an API Gateway, gets picked up by Lambda, and fans out into storage (S3 for test packages, DynamoDB for metadata) and scheduling. Work eventually lands on an SQS queue, gets picked up by a Lambda runtime running a pre-built image, executes the actual test against dev/staging/prod, and the results flow into S3 (for logs) and Timestream (for time-series history). From there, results get analyzed and, if warranted, turned into Slack messages and PagerDuty incidents — including, eventually, auto-resolving those incidents.

The one architectural decision I’d tattoo on the wall if I were the tattooing type: test execution is decoupled from monitoring and alerting. Running the test and deciding whether to panic about the test are two completely different jobs, and bolting them together is how you end up with a monitoring system that pages someone because it had a bad day, not because production did.

Execution: Lambda, Not a Fleet of Sad EC2 Boxes

We run test packages on Lambda using pre-built images, which means nobody on this team owns a persistent EC2 instance whose sole purpose in life is to periodically wake up, run a script, and go back to sleep. The platform pulls the test artifact from S3 and executes it inside Lambda. This keeps the execution layer generic — the platform cares about the contract (give me a test, I’ll run it and hand you back a result), not about reimplementing every team’s testing framework from scratch.

Why SQS Exists (Besides Making the Diagram Look Fancier)

SQS draws a clean line between two very different statements:

1. “A test needs to run.”
2. “A test is currently running.”

Without a queue, those two ideas get tangled together, and tangled ideas are exactly how you end up debugging why fourteen canaries all decided to execute at once because someone’s cron job and someone else’s retry logic had a disagreement. The queue gives scheduling and execution some breathing room from each other, which matters a lot once you have multiple teams and multiple canaries all sharing the same infrastructure.

Where the Data Lives

Results and logs go into S3 under a deliberately boring, deliberately predictable path:

results/{year}/{month}/{day}/{team}/{event_uuid}

Boring is a feature here. When something fails and someone needs the logs right now, “boring and predictable” beats “clever and bespoke” every time. Before anything gets written, the platform validates the event UUID and file format — because shared infrastructure and “trust me, this is valid JSON” do not mix.

To avoid sending an on-call engineer on what I can only describe as a 2 AM archaeological expedition through raw S3 buckets, there’s a dedicated Lambda that generates presigned URLs on demand. Give it an event UUID and a date, and it hands back a temporary link straight to the relevant logs. The alert goes from “something failed” to “something failed, and here’s the evidence” — which is really the least a monitoring system can do for someone it just woke up.

Timestream: Because One Failed Test Is Not a Crisis

Individual test failures are noisy. Amazon Timestream lets the platform store execution results over time and, more importantly, aggregate results across all of a team’s canaries into a single health signal. This is the difference between a test runner and an actual monitoring platform:

• A test runner asks: “Did test #1847 fail?”
• A monitoring platform asks: “Is this system healthy?”

Nobody wants a page for test #1847. Everybody wants to know if their system is, broadly, on fire.

DynamoDB: The Platform’s Long-Term Memory

DynamoDB holds the operational state — alarm status, auto-resolution progress, latest results, per-canary configuration. When a new team or canary shows up, sensible defaults get initialized automatically, so nobody has to hand-configure “what does a brand-new canary’s alarm state look like” from scratch.

A Sandbox Before the Real Thing

Before a canary graduates to watching production, it gets to practice in a development environment, where the platform keeps a rolling window of its last four results. This exists for a wonderfully unglamorous reason: you do not want to discover your monitoring is broken by waiting for production to actually break. A monitoring test that cries wolf — or worse, sleeps through the wolf — is worse than no test at all.

Slack for Visibility, PagerDuty for “Wake Up, Human”

These two integrations split responsibility cleanly:

• Slack tells the team, near real-time, “here’s your canary’s latest status and logs.” It’s ambient awareness — nobody has to go check a dashboard to know things are fine.
• PagerDuty is for when “fine” stops being true. When failure conditions are actually met, an alarm gets raised, and the platform tracks whether one’s already open — so one flaky dependency doesn’t spawn a fresh incident every five minutes like some kind of alert-generating hydra.

Auto-Resolution: Teaching the System Not to Overreact

Distributed systems have bad days. A single passing test right after a string of failures doesn’t necessarily mean everything’s fixed — it might just mean the system briefly caught its breath. So instead of the naive version:

FAIL → open incident
PASS → immediately close incident

the platform waits for a sustained recovery — for example, three consecutive passes — before auto-resolving:

FAIL, FAIL, FAIL  →  incident opens
PASS              →  keep watching
PASS              →  keep watching
PASS              →  auto-resolve

This one design choice does a lot of quiet work toward preventing alert flapping, which is the pager-based equivalent of a smoke detector that goes off every time someone makes toast.

SNS: Keeping Everyone’s Hands Off the Same Steering Wheel

Rather than letting multiple parts of the system independently poke at the same DynamoDB records — a classic recipe for race conditions — state changes get routed through SNS to a dedicated Lambda responsible for actually applying them. One controlled path in, instead of five processes all reaching for the same lever at once.

An API, Because Someone Has to Ask Nicely

Teams manage their canaries and configuration through an API layer that validates structure and data types on the way in. This sounds unglamorous, and it is — but this is shared infrastructure across roughly ten teams, and “a malformed request corrupted everyone’s monitoring state” is not a sentence anyone wants to say out loud in a postmortem.

Built for Ten Teams, Not One

The platform treats Team, Canary, Environment, Schedule, Execution, Result, and Alarm as first-class, generic concepts — not as a pile of if-statements shaped like one specific application. That’s the actual unlock: the same infrastructure works whether it’s watching one team’s payments service or another team’s internal API, because nothing about it assumes it knows what your service does. It only needs to know how to run your test and interpret the result.

And On Top of All That, a Dashboard

None of this is much fun to reason about through raw AWS console tabs, so there’s a React dashboard on top that gives every team a single place to see their canaries, executions, results, and overall health — without needing to understand the DynamoDB tables or S3 paths underneath.

The Actual Hard Part

Writing one Lambda function is not hard. Writing this:

Scheduling → Queueing → Execution → Result Processing →
Time-Series Storage → Aggregation → Alarm State →
Notification → Incident Resolution

…as a chain where every link has to work, reliably, for other teams’ production systems, is the hard part. And there’s a specific irony baked into the whole project: the system whose job is to tell you production is broken cannot itself be the thing that’s broken. Every design decision above — the queue, the sandbox environment, the validation on inputs, the deduplicated alarms — exists partly in service of that one uncomfortable requirement.

What This Taught Us

The biggest lesson wasn’t about Lambda, or SQS, or Timestream. It’s this:

Running a test is easy. Building the infrastructure around the test is the actual job.

• How do I run it, and how often?
• Where do I store the result, and how do I find it again fast?
• How do I tell “a blip” apart from “an incident”?
• Who needs to know, and when should they stop being told?
• When is it actually safe to say “we’re fine” again?

Once a platform answers those questions once, centrally, every team gets to skip the boring parts entirely and go straight to the part they’re actually good at: writing the test that knows what “healthy” means for their own system.

The platform’s job is to make sure that test never has to run manually, never gets lost in a log bucket, and never fails silently into the void. Everything else — the queues, the tables, the presigned URLs, the flapping-averse alarm logic — is just plumbing in service of that one sentence.

---

A few specifics — the exact SQS/SNS event flows, what a canary technically contains under the hood, how pass/fail thresholds get computed, and the real story behind the scheduling mechanism — are intentionally left a little fuzzy here until we nail down the details. Happy to tighten those sections once we have them.