An alert that fires on every exception teaches people to ignore exceptions. An alert that fires only during a total outage arrives too late. The useful middle ground starts with the user experience and the decision the responder can make.
I would separate a page from a ticket. A page should describe an urgent condition with an actionable response. A ticket can capture a slower deterioration, a recurring warning, or work that should happen in the next planning cycle.
Put a denominator next to the count
One hundred failed requests out of a hundred is different from one hundred out of a million. In an illustrative service with 50,000 eligible requests and 250 failures in a window, the observed failure fraction is 0.5%. Whether that violates the objective depends on the SLI definition and evaluation window.
Define eligibility carefully. Do expected client errors count as failures? Are health checks included? Are logs sampled? A numerator from indexed error logs and a denominator from a different population can create a precise-looking but invalid ratio.
Use error-budget consumption to set urgency
Google’s SRE workbook describes multiwindow, multi-burn-rate alerting. Burn rate compares the observed error rate with the rate allowed by the objective. Pairing long and short windows helps distinguish sustained impact from a brief spike.
I would use that method as a starting design, then test it against the service’s traffic and historical incidents. Low-volume services need special attention: one failure can dominate a short window without telling you whether the broader experience is degraded.
Make the alert a small handoff
Include the affected service and environment, the window, current impact, relevant release, dashboard link, and first diagnostic step. Name the owner and escalation route. “Errors high” forces the responder to reconstruct all of this while the clock is running.
Test the no-data path separately. A stopped collector should not be interpreted as perfect availability. Decide whether missing data should page, open a ticket, or trigger a telemetry-health investigation.
Finally, review alerts after incidents and after uneventful pages. Ask whether the alert identified a real action soon enough. Removing a noisy condition can be an improvement, provided you can explain how the important failure will still be detected.