Anomaly monitoring system

Summary

Portfolio demonstration: configurable anomaly monitoring for environments with many systems generating operational metrics. Core loop is scheduled collection → windowed evaluation → alerts, so operators can define collectors, series, and monitors without new code per metric source.

Problem

Large environments often rely on ad-hoc scripts, system-specific tools, and dashboards. That fragmentation creates inconsistency and duplicated effort, and makes uniform anomaly detection hard to apply. The need is a first-class, configurable capability: schedule collection, define named series, apply windowed threshold evaluation, and surface alerts — one system operations can use regardless of the underlying source.

Approach

Spring schedules drive collection jobs against STUB or CloudWatch sources. Observations append to MongoDB series; monitors evaluate sliding windows against rules and write alert records; notification channels fan out (Slack verified on state transitions). Series is the hub for tags, jobs, and monitors — a modular monolith with clear domain contracts.

Monitoring architecture showing scheduled collection jobs, STUB or CloudWatch sources, observation storage, series configuration, window evaluation, alert records, and Slack transition notifications — Monitoring architecture — scheduled collection, observation storage, windowed evaluation, alert records, and transition notifications.

Java 17 / Spring Boot APIs with Basic auth and team-scoped config
STUB and AWS CloudWatch collectors with static-threshold monitors
Slack notification channels on alert state transitions
React + Vite operator console: CRUD, search, graphs, and Test tabs
Config-driven collectors, series, and monitors
Testcontainers, smoke tracer, and GitHub Actions CI

Key decisions: domain-agnostic and config-first design; modular monolith before any microservice split; contracts and ADRs before feature sprawl; setup-test loop (job Test → monitor Test → Slack) as the demo spine; SPA operator console for practical ops UX. Verification layers unit tests through Testcontainers to a smoke tracer, with structured JSON logs and Actuator health for runtime visibility.

Operators define a series, its tag schema, and the jobs and monitors bound to it from the console rather than in code:

Operator console series detail for cw-metrics-series showing enabled state, tag schema with required and cardinality columns, an inbound collection job on a ten minute interval, and a monitor in OK state — Series detail — tag schema plus the collection jobs and monitors bound to one series.

Monitors bind the alert path, evaluation rule, and notification channel in one place:

Monitor detail showing an alert path from ten minute schedule through collection job and series to the monitor, with OK status, tag filter, GT 0 threshold, breach policy, and Slack webhook channel — Monitor detail — alert path from schedule through job and series to monitor, with threshold, breach policy, and Slack channel.

Results & limits

Delivered a coherent, demoable monitoring loop: centralized anomaly detection, config-driven collectors and monitors, consistent windowed evaluation, and an operator console for setup and verification — including CloudWatch ingest and Slack alerts on transitions.

Slack channel showing a monitor FIRING alert for Lambda Errors with OK to ALERT state change, followed by a RECOVERED message with ALERT to OK for the same monitor — Alert transitions — Slack notifications when a monitor fires and when it recovers.

Deferred: learned baselines, multi-team RBAC UX, broader connectors, email delivery, and production SPA hosting.

Links

Reach out via Contact to discuss this project or schedule a walkthrough.