Monitoring

🚦 1. The Service-Oriented Approach

Don’t monitor “servers”; monitor Services. A server might be at 99% CPU, but if the service is still responding in 50ms and users are happy, it’s not an emergency—it’s high efficiency.

The RED Method (For Services)

Popularized for microservices, this focuses on what the user experiences:

Rate: The number of requests per second.
Errors: The number of failed requests.
Duration: The time those requests take (Latency).

The USE Method (For Resources)

Used for the underlying infrastructure (CPU, Memory, Disk):

Utilization: The average time the resource was busy.
Saturation: The degree to which the resource has extra work it can’t service (queuing).
Errors: The count of error events.

🚨 2. Alerting: Signal vs. Noise

The biggest threat to a Programmer is Alert Fatigue. If your team receives 50 alerts a day, they will eventually ignore the one that actually matters.

Symptom-Based Alerting

Never alert on a cause; alert on a symptom.

Bad Alert (Cause): “Database CPU is 90%.” (The DB might be designed to run high).
Good Alert (Symptom): “500 Errors are > 1%.” (Something is definitely broken for the user).

Alert Categorization (Severity)

Level	Action	Channel
Critical (P1)	Immediate intervention required. System is down.	PagerDuty / Phone Call
Warning (P2)	Potential issue, but system is degraded, not down.	Slack / Email (Next Business Day)
Info (P3)	For record-keeping or non-urgent optimization.	Dashboard only

📈 3. Service Level Objectives (SLOs)

This is the most critical concept. You must define what “Good Enough” looks like.

SLI (Indicator): The metric (e.g., “HTTP Request Latency”).
SLO (Objective): The target (e.g., “99% of requests < 300ms”).
SLA (Agreement): The legal contract (e.g., “If we miss our SLO, we pay the customer back money”).
Error Budget: If your SLO is 99.9%, you have a 0.1% “budget” for failure. If you haven’t used your budget, you can deploy risky new features. If your budget is empty, you must freeze deployments and focus on stability.

🛠️ 4. Advanced Monitoring Patterns

Synthetic Monitoring (Uptime Checks)

Don’t wait for a user to find a bug.

Strategy: Use “Canary” scripts that act like users. Every 60 seconds, a script logs in, adds an item to the cart, and checks out. If this fail, you know the entire stack is broken before the first real customer complains.

White-box vs. Black-box

White-box: Metrics from inside the app (e.g., Go routines, Thread pool size, GC pause time).
Black-box: Testing the app from the outside (e.g., “Can I reach the URL?”).

Health Endpoints (`/healthz`)

Every service should have a /health or /ready endpoint.

Programmer’s Move: Don’t just return 200 OK. Have the endpoint check the connection to the Database and the Cache. If the DB is down, the service should report itself as “Unhealthy” so the Load Balancer stops sending it traffic.

⚖️ The Monitoring Stack

Collection: Prometheus (Pull-based) or OpenTelemetry (Push-based).
Storage: VictoriaMetrics or InfluxDB (Time-series data).
Visualization: Grafana.
Alert Routing: Alertmanager (Prometheus) or Opsgenie.

💡 Note: Every alert should have a Runbook link in the description. When an engineer is woken up at 3:00 AM, they shouldn’t have to guess. The alert should say: “This is what is broken, here is the dashboard to look at, and here is the command to fix it.”