Quick answer
Start with what users need: successful requests and timely responses. Define measurable service-level objectives, alert when failures consume the allowed error budget too quickly, and connect every urgent notification to an owner and a response plan. Use infrastructure metrics, logs and traces to investigate—not as substitutes for measuring user impact. This follows the symptom-focused approach in Google’s SRE guidance. (sre.google)
This guide is for teams operating a cloud-hosted API. The worked example uses hypothetical targets and traffic, not measured results. Before implementing it, you need request telemetry, an agreed service owner, a notification channel and permission to inspect the application and its dependencies.
1. Choose signals that describe the user’s experience
A healthy server does not necessarily mean a working service. Google’s SRE guidance distinguishes symptoms, which describe what is going wrong, from causes, which help explain why. Its four “golden signals” are latency, traffic, errors and saturation. (sre.google)
For an API, start with these questions:
| Reader question | Signal to collect | Operational use |
|---|---|---|
| Can users complete requests? | Successful and failed request counts | Measure availability |
| Are responses timely? | Request-duration distribution | Measure latency |
| How much demand is arriving? | Request counts or rate | Interpret failures and load |
| Is capacity constrained? | Queues, resource utilization and limits | Investigate likely causes |
Treat CPU utilization and memory pressure as supporting evidence. A resource threshold deserves an urgent alert when it predicts an actionable failure, not simply because it exceeds an arbitrary number. Google recommends keeping paging rules simple and focused on clear failures. (sre.google)
Choose the measurement boundary deliberately. Application instrumentation may miss requests that never reach the process. External, black-box checks can expose failures from outside the service, while internal monitoring helps diagnose them. (sre.google)
Practical check: Write down where each signal is measured. Do not label server-side timing “end-to-end user latency” unless it actually measures that experience.
2. Define SLOs with explicit counting rules
A service-level indicator, or SLI, measures service behavior. A service-level objective, or SLO, sets a target for that indicator over a defined interval. The error budget is the permitted amount of bad service under that target. Google recommends choosing indicators that reflect user experience and agreeing objectives with relevant stakeholders. (sre.google)
For the hypothetical API, consider two initial objectives:
| Objective | Good event | Target |
|---|---|---|
| Availability | An eligible request completes successfully | 99.9% over a rolling 30 days |
| Timely response | An eligible request succeeds within 500 milliseconds | 99% over a rolling 30 days |
These are illustrative choices, not universal recommendations. Confirm that the latency threshold matches the user journey and that the reliability target has business support.
Document the denominator before creating alerts:
- Which production operations are included?
- What constitutes a valid, eligible request?
- How are timeouts and service-generated rejections classified?
- Which client errors are excluded, and why?
- Are retries counted as individual attempts or part of a user transaction?
In this example, failed requests are also bad events for the timely-response SLI. The two budgets overlap; do not add them together.
Avoid hiding a critical operation inside a service-wide average. Where different operations have different user expectations, consider separate objectives rather than creating an objective for every endpoint. Google’s guidance emphasizes a small, useful set of user-centered indicators. (sre.google)
3. Use metrics to detect trouble, logs and traces to explain it
Metrics provide numerical measurements over time. Logs record events, while traces connect operations within a request as it moves through a distributed system. OpenTelemetry explains how correlated logs and trace spans provide context for investigation. (opentelemetry.io)
Build a response workflow around those roles:
- Metrics: Identify the affected operation, failure fraction and duration.
- Logs: Inspect relevant errors, deployment events and dependency messages.
- Traces: Follow representative failed or slow requests through downstream calls.
For latency objectives, collect a distribution rather than relying on average response time. Prometheus documents how histogram buckets can measure the fraction of observations meeting a threshold, and why averaging precomputed quantiles across instances is generally invalid. (prometheus.io)
For the 500-millisecond example, verify that instrumentation can count successful requests within that threshold. A duration histogram covering every response does not, by itself, distinguish a fast success from a fast failure.
Use consistent service, environment and operation fields. Where supported, include trace identifiers in structured logs so responders can move between evidence sources. OpenTelemetry specifically identifies log-to-trace correlation as useful context. (opentelemetry.io)
Practical check: Follow one controlled test request from the entry point through a dependency and back. Confirm that timing units, outcome classification and request identity agree.
4. Translate error budgets into actionable alerts
Burn rate compares the observed bad-event fraction with the fraction allowed by the SLO:
Burn rate = observed bad-event fraction ÷ allowed bad-event fraction
A burn rate of 1 matches the objective’s allowed failure rate; higher values indicate faster consumption. CloudWatch Application Signals uses this relationship in its SLO burn-rate calculations. (docs.aws.amazon.com)
For the example:
- Availability allows 0.1% bad requests.
- Timely response allows 1% bad requests.
Google’s SRE workbook offers these starting alert patterns for a 30-day objective:
| Response | Long window | Short window | Burn-rate threshold |
|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 |
| Page | 6 hours | 30 minutes | 6 |
| Ticket | 3 days | 6 hours | 1 |
Require both windows in a row to breach; either paging row can trigger an urgent notification. The short window confirms that the problem is still active. These are starting parameters, not a guarantee of appropriate sensitivity for every service. (sre.google)
Worked API example
Assume these hypothetical measurements:
| Window | Eligible requests | Failed requests | Failed or slower than 500 ms |
|---|---|---|---|
| 1 hour | 100,000 | 2,000 | 16,000 |
| 5 minutes | 10,000 | 200 | 1,600 |
For availability, both windows have a 2% failure fraction:
2% ÷ 0.1% = 20× burn
For timely response, both have a 16% bad-event fraction:
16% ÷ 1% = 16× burn
Both objectives exceed the 14.4 threshold in both windows. Under the proposed rule, both qualify for paging. Group them into one service incident rather than demanding two independent responses.
The responder should confirm scope, inspect recent changes and dependencies, then apply an approved mitigation.
Limits: These calculations are request-based. Changing traffic volume affects actual budget consumption, so time-window burn rates do not imply a fixed number of lost requests. Low-traffic services also need different treatment: a few failures can produce extreme ratios, and synthetic checks may provide additional evidence. (docs.aws.amazon.com)
5. Make every notification carry ownership and a decision
AWS Well-Architected guidance calls for a defined response process, clear ownership and an escalation path for each alert. It also recommends grouping related alarms to reduce noise. (docs.aws.amazon.com)
Use three response categories:
- Page: A person must act promptly to limit current impact.
- Ticket: Investigation or corrective work can wait for the agreed working period.
- Dashboard: Information is useful but does not require a notification.
For each paging rule, specify:
- Service and environment.
- Affected objective and breached windows.
- Primary on-call role and backup role.
- Runbook and relevant dashboard.
- Acknowledgment deadline and escalation route.
- Conditions for declaring a wider incident.
Choose deadlines according to service criticality and actual staffing. Do not promise continuous response without continuous coverage.
Test delivery, not just threshold evaluation. Send a controlled notification through the real routing path and confirm that the intended responder can access the evidence and acknowledge it.
Avoid routing every dependency symptom to every team. Group related symptoms under the affected service, then bring in dependency owners when investigation identifies their involvement. This implements the process-per-alert approach without assuming that every alarm represents a separate incident. (docs.aws.amazon.com)
6. Write runbooks that support safe mitigation
A runbook is a documented procedure for achieving a specific outcome. AWS recommends including required tools and permissions, error handling, escalation, ownership and validation by another team member. (docs.aws.amazon.com)
Use this structure:
- Purpose: Which alert is this for, and what outcome is required?
- Prerequisites: Required access, tools and approval.
- Confirmation: How to establish user impact and affected scope.
- Investigation: Which dashboards, logs and traces to inspect.
- Mitigation: Approved actions and their decision criteria.
- Safety: Preconditions, stop conditions and rollback.
- Recovery: Checks that demonstrate service restoration.
- Maintenance: Owner, last review and last exercise.
For the API example, a proposed response sequence is:
- Confirm the breached windows and whether failures are concentrated by operation or location.
- Check recent deployments and configuration changes.
- Inspect failed and slow traces for a shared dependency.
- Consider rollback only through the service’s documented, approved procedure.
- Consider capacity changes only after checking the relevant constraint.
- Stop and escalate when the proposed action’s prerequisites are not met.
- Verify successful, timely requests and continued telemetry collection.
These are investigation prompts, not executable recovery instructions. Replace them with service-specific steps before enabling paging.
Store the runbook where responders can reach it during an incident, and have someone other than its author exercise it. Update it when system behavior or procedures change. (docs.aws.amazon.com)
7. Check telemetry cost, coverage and alert quality
Monitoring has its own operating cost. CloudWatch’s pricing documentation separates chargeable capabilities such as metrics, logs, alarms, application signals and tracing; the applicable charges depend on the features and usage selected. Do not estimate the bill from request count alone. (aws.amazon.com)
Review these controls:
- Metric labels: Avoid unbounded dimensions such as request IDs and user IDs. Every unique label combination creates another time series. (prometheus.io)
- Trace sampling: Define what evidence must remain available. OpenTelemetry describes head and tail sampling, including tail decisions based on errors or latency. (opentelemetry.io)
- SLO inputs: Prefer complete request counters for the example’s ratios rather than assuming a sampled trace set is representative.
- Retention: Set explicit investigation and retention needs, then check the provider’s applicable ingestion, storage and query charges. (aws.amazon.com)
- Missing data: Define how absent telemetry differs from zero traffic or zero errors.
Before launch, test a successful request, a known failure, a slow response, a quiet traffic period and a telemetry interruption. Check counting, notification routing, access and recovery behavior.
Review noisy alerts after incidents. An alert that repeatedly produces no useful action needs a better signal, a different response category or retirement—not merely another dashboard.
The practical starting point is small: one important service, a few user-facing objectives, tested notifications and an exercised response procedure. Expand coverage after those pieces work together.
Read next: Turn RTO and RPO Into a Cloud Backup and Restore-Test Plan