Skip to content
sumanth.aiAN ENGINEER’S WORKSHOP
← All articles

Observability / 2 MIN READ

Keep the evidence, cut the repetition

How to control telemetry volume without removing the clues the next incident will need.

The easiest way to reduce a logging bill is to stop collecting logs. It is also an easy way to discover, during an incident, which records were important. I would rather make the tradeoff explicit before changing the pipeline.

Start by separating ingestion, indexing, retention, and archive access. They answer different questions, and their cost implications depend on your vendor and contract. A reduction at one stage does not necessarily reduce every other line item.

Classify records by the question they answer

For an illustrative content service, routine successful requests might support traffic trends. Detailed application failures support diagnosis. Security-relevant records can have separate retention and access requirements. Debug records may be valuable during a bounded investigation but expensive as a permanent default.

Agree on these purposes with the teams that use the data. Do not let a volume optimization silently change an audit requirement or erase an uncommon but important failure.

Start with low-information repetition

Datadog documents filtering, sampling, and routing strategies for log-volume reduction. A useful candidate is a repeated health-check message whose count is already captured elsewhere. A risky candidate is an entire route family simply because it produces many records.

Before sampling, ask which calculations assume complete data. If you keep a fraction of successful events but all failures, the ratio of retained failure logs to retained total logs is no longer the service error rate. Use suitable metrics or explicitly weighted analysis instead.

Put a budget around dimensions, too

An arbitrary user ID or full URL is usually a poor default metric dimension. The number of unique combinations matters, not just the number of tag names. Prefer bounded dimensions such as environment, service, status class, and a normalized route where they answer the operational question.

Keep detailed identifiers in appropriately protected investigative data when needed. Scrub secrets before transport where possible; reducing retention is not a substitute for avoiding collection of credentials.

Prove the reduced dataset still works

Replay a sanitized incident exercise using the proposed policy. Can a responder identify impact, affected version, error family, and likely dependency? Can they distinguish a healthy service from a failed collection pipeline?

Track both volume and diagnostic usefulness after the change. A smaller bill is measurable. The harder, equally important result is knowing which questions the retained evidence can still answer.

REFERENCES & FURTHER READING

Official documentation and primary sources behind the technical details. Examples in this article are illustrative.

KEEP READING

Observability / 2 min

The first AEM dashboard should answer five questions

Read article ↗
Observability / 2 min

Before another dashboard, fix the log fields

Read article ↗
Browse the whole notebook →