How we cut probe latency by 40%

Every probe niceuptime runs starts a clock. For years that clock included a scheduling hop we never thought hard about — a queue that batched checks before handing them to a regional runner. It was fine, until our customers' expectations (and ours) outgrew it.

This spring we rebuilt the path. Here's what changed, what broke along the way, and the numbers that made it worth the risk.

The old path

A check moved through three systems before it ever hit your endpoint. Each was individually fast; together they added a median of 180ms of pure overhead before the first byte left our network.

"The fastest check is the one you don't queue."

Moving to a regional scheduler

We pushed scheduling to the edge — each region now owns its own queue and runs checks locally, with a thin control plane keeping configuration in sync. The runner config is small enough to read in one sitting:

# regional runner — checks stay local
scheduler: regional
regions: [us-east, eu-west, ap-sg]
batch_window_ms: 0

The batch_window_ms: 0 line is the whole story — we stopped waiting to group checks and just ran them.

The result

Median check latency dropped 40%, p95 dropped further, and — the part we didn't expect — flapping alerts fell too, because checks now complete inside a single tighter window. Calmer graphs, fewer 3am pages.

We're rolling the same approach out to pulse ingestion next. If you want the gory details, our status page has the migration window logged in full.