The brief
Most monitoring dashboards answer “is everything green?” Watchman was built to answer a narrower, more useful question: does anything here need a human in the next five minutes? Everything else is noise dressed up as data.
Approach
The dashboard ingests metrics over WebSockets and keeps a rolling window in Postgres rather than reaching for a time-series database — at the scale this runs at, a well-indexed relational table with a background rollup job is simpler to operate and easier to reason about than adding a new piece of infrastructure.
Alert on every threshold breach.Naive approachDoesn't work
Alert on sustained breaches during business hours only.Shipped approachWorks
What shipped
- A single-page dashboard, server-rendered, with a thin client layer only where the live-updating view genuinely needs it.
- Quiet-hours suppression, so a flaky staging service doesn’t page anyone at 3am for something that can wait until standup.
- A rule engine simple enough to read in one sitting, deliberately avoiding a general-purpose alerting DSL nobody but its author would understand.
Outcome
Fewer, more trustworthy alerts. The measure of success wasn’t uptime — it was how often the on-call engineer could silence a notification without opening a laptop.