Skip to content
sumanth.aiAN ENGINEER’S WORKSHOP
← All articles

Observability / 2 MIN READ

Follow a request, not a timestamp coincidence

How correlation fields help distinguish a plausible explanation from evidence about the same operation.

Two events happening in the same second can be related. They can also be two unrelated requests on a busy service. Time is a useful starting filter, not a reliable join key.

I prefer an investigation path that follows the request where the instrumentation supports it: user-visible failure, delivery record, application operation, dependency call. Every hop should have an explicit explanation for how it is connected.

Start with stable service identity

Datadog recommends env, service, and version for correlating telemetry. Trace and span identifiers connect logs to an instrumented operation when they are correctly injected and parsed. Adding the same service label to everything does not automatically create distributed traces.

For AEM, confirm that the tracing and logging integration is supported in the particular deployment. If an edge request identifier is not propagated through the application, do not draw a line that the data cannot support.

Use an example with two possible causes

Imagine a page returning 500 while a downstream API is also slow. A timestamp search finds a timeout exception. That is a plausible lead. A trace containing the failing page operation and the timed-out dependency, with matching log context, is stronger evidence.

Even then, ask whether the dependency caused the failure or merely participated in it. Retries, cancellation, and shared resource exhaustion can make the first obvious span an incomplete explanation.

Plan for incomplete evidence

Logs and traces may be sampled independently. Datadog notes that a log can contain a trace identifier even when the corresponding trace was not ingested or retained. A missing trace is not automatically proof of broken correlation.

Write that limitation into the investigation guide. Provide a fallback using service, version, route family, time range, and a safe request identifier where available. Separate “we could not retain this trace” from “we have no evidence this request reached the application.”

Keep identifiers out of the wrong places

A request identifier can be useful in a log search without being a good metric tag. Putting every unique identifier into a metric dimension can produce an enormous number of series. Correlation design and aggregation design solve different problems.

The goal is a defensible chain of evidence. A dashboard should make following it easier, while being honest about the gaps.

Try it / Group repeated log patterns

A small example to explore alongside this article. Inputs stay in this page.

Related tools: Group repeated log patterns ↗

Was this useful?

Account-free. Your response stays in this browser; it is not sent to a server.

REFERENCES & FURTHER READING

Official documentation and primary sources behind the technical details. Examples in this article are illustrative.

KEEP READING

Observability / 2 min

The first AEM dashboard should answer five questions

Read article ↗
Observability / 2 min

Before another dashboard, fix the log fields

Read article ↗
Browse the whole notebook →