A metric can be mathematically correct and still lead an operations team to the wrong conclusion.
That sounds contradictory, but it is a real failure mode in OpenTelemetry. The Metrics SDK protects an application from unbounded memory growth by limiting how many unique attribute combinations it tracks for each metric stream. When that limit is reached, additional measurements are folded into an overflow data point. The overall value remains intact, but the original measurement attributes do not.
The result is subtle: a dashboard showing total requests may still be correct while the same metric grouped by route, tenant, status, or success silently undercounts. An SLO can look healthier than reality. An alert can stop seeing failures. An autoscaling signal can lose the dimension that made it operationally meaningful.
For enterprise observability, cardinality overflow is therefore not just an SDK tuning issue. It is a reliability event that needs ownership, detection, impact classification, and controlled remediation.
Why OpenTelemetry limits cardinality
Metric cardinality is the number of unique attribute combinations reported for a metric stream. If a request counter uses http.route, http.request.method, and success, the SDK maintains aggregation state for each unique combination of those values.
This is useful because dimensions make metrics actionable. Teams can distinguish checkout failures from home-page traffic, compare methods, or calculate service-level indicators. But each distinct combination consumes memory in the instrumented process. Unbounded values such as request IDs, raw URLs, user input, tenant identifiers, or exception messages can cause the number of data points to grow rapidly.
The OpenTelemetry Metrics SDK specification defines a cardinality limit as a safety mechanism. If no more specific setting applies, the default is 2,000 data points per metric stream. A View or MetricReader can define a different limit.
Once the limit is reached, the SDK does not simply discard subsequent measurements. It aggregates them into a synthetic data point marked with otel.metric.overflow=true. That preserves the measured value while bounding the aggregation state held in memory.
This behavior solves one problem, but it cannot preserve every attribute-level interpretation of the metric.
How a correct total creates a misleading breakdown
Assume a request counter records the following measurement:
{http.route=/checkout, success=false} 1
If the metric stream has already reached its cardinality limit, that value is added to the overflow data point:
{otel.metric.overflow=true} 1
The total request count still includes the measurement. However, queries filtering for success=false or grouping by http.route no longer see its original attributes. Both breakdowns undercount.
The important point is that overflow removes the entire measurement attribute set, not only the attribute that caused excessive cardinality. A boolean such as success is low-cardinality, but it becomes unreliable when it was recorded alongside an overflowing high-cardinality dimension.
Resource attributes and instrumentation-scope attributes are treated differently. Context such as service.name remains available on the overflow data point. That helps identify where overflow occurred, but it is not a workaround for putting measurement dimensions in the wrong place.
The enterprise risk is incomplete evidence
Overflow is dangerous because the telemetry pipeline can remain healthy from a conventional infrastructure perspective. The application runs. Metrics continue to arrive. The backend accepts them. The total line on a dashboard looks plausible.
What degrades is the completeness of the operational evidence derived from attributes.
- SLOs: A failure-rate indicator filtered by a status or success attribute can undercount the numerator and report better reliability than users experience.
- Alerts: A condition scoped to a route, tenant, region, or error category can miss measurements that moved into overflow.
- Dashboards: Totals and breakdowns can disagree without making the reason visible to the reader.
- Autoscaling: A metric partitioned by queue, workload class, or tenant can lose the dimension used to make a scaling decision.
- Chargeback: Usage can remain in the total while disappearing from the team, tenant, or product allocation.
- Incident analysis: Responders may trust a breakdown that no longer represents the full population.
This makes overflow a data-quality and decision-assurance problem. The organization has not necessarily lost measurements, but it has lost the evidence needed to attribute them correctly.
Make metric semantics an explicit platform contract
Platform teams should treat every operationally critical metric as a contract between instrumentation owners and metric consumers. That contract must describe more than the metric name and unit.
For each critical metric, record:
- the business or operational decision it supports;
- the allowed measurement attributes and their expected value domains;
- which attributes are required by SLOs, paging alerts, autoscaling, or chargeback;
- the expected number of active combinations and the assumptions behind it;
- the aggregation temporality and collection interval;
- the configured cardinality limit and safety headroom;
- the accountable instrumentation and service owners;
- the response procedure when overflow appears.
This contract turns a hidden SDK behavior into something that can be reviewed and tested. It also prevents a common failure mode: an instrumentation change adds a convenient attribute without recognizing that several dashboards and alerts depend on every measurement retaining other dimensions.
Detect overflow as close to the source as possible
OpenTelemetry provides a standard marker for this purpose: otel.metric.overflow=true. Prometheus-compatible exporters commonly expose it as otel_metric_overflow="true".
The practical control is straightforward: continuously detect overflow and group it by metric and service. The official OpenTelemetry guidance provides an illustrative PromQL pattern based on last_over_time. Organizations should adapt the grouping labels and query window to their own ingestion model, retention, and fleet size.
Detection alone is not enough. The alert must be connected to impact.
- Page: Overflow affects a metric used for paging, an SLO, autoscaling, security monitoring, or a critical business decision.
- Ticket: Overflow affects exploratory, diagnostic, or lower-priority metrics without an immediate control impact.
- Track as debt: A known and accepted high-cardinality use case remains within a documented risk envelope and has an accountable review date.
Dashboards that filter or group by measurement attributes should also display an overflow indicator. A reader should not have to know an SDK implementation detail to determine whether a chart is complete.
Govern instrumentation changes like production changes
Overflow often begins after a change: a route template is replaced by a raw path, a library version adds an attribute, an error field starts carrying user input, or traffic activates more combinations than the original capacity model assumed.
That means instrumentation belongs in the software delivery control model.
Review attribute boundedness
Pull-request checks should identify new or changed metric attributes and classify their value domain. Route templates, methods, status codes, and bounded error categories are generally easier to control than raw URLs, request IDs, session IDs, exception text, or user-provided values.
Test metric contracts
Automated tests can verify that critical instruments retain required dimensions under representative load and that unexpected attributes are filtered through Views or corrected at the instrumentation source. Tests should include burst behavior, not only steady-state averages.
Roll out progressively
Instrumentation and SDK upgrades should move through representative canaries. Compare overflow markers, process memory, exported series, SLO calculations, and backend cost before expanding the rollout.
Observe the observability change
A change is not successful because telemetry still arrives. It is successful when required dimensions remain complete, overflow stays within policy, process memory remains safe, and the backend absorbs the resulting series volume.
Do not respond by raising the limit automatically
A higher limit may be appropriate, but it is not the default answer. It increases how much state the SDK may hold and can allow a cardinality leak to grow further before the safety mechanism becomes visible.
Use a deliberate decision path:
- Identify the affected metric and consumer. Determine which service emitted overflow and which SLOs, alerts, dashboards, scaling policies, or allocations use its measurement attributes.
- Classify the attributes. Find unbounded or accidental values. Remove, normalize, or filter them if they do not belong in metric aggregation.
- Estimate the intended active set. Model the combinations created by bounded dimensions, collection interval, traffic pattern, and burst headroom.
- Review temporality. Delta temporality bounds synchronous aggregation to combinations active in a collection cycle. Cumulative temporality retains state across cycles and can make a growing population overflow predictably.
- Choose the limit deliberately. Raise it only when the dimensions are operationally necessary, the active set is understood, memory impact is acceptable, and overflow remains monitored.
- Validate the recovery. Confirm that the overflow marker clears and that affected queries once again include the expected dimensions. Do not close the issue solely because configuration changed.
If a high-cardinality dimension is required for per-tenant or per-customer objectives, removing it may destroy the purpose of the metric. In that case, the right response may combine delta temporality, a deliberately sized limit, bounded active-set assumptions, backend capacity planning, and explicit monitoring.
Build a cardinality assurance scorecard
Enterprise observability teams need evidence that metric semantics remain dependable across a large fleet. A useful scorecard can track:
- number of services and metric streams emitting overflow;
- duration and recurrence of overflow events;
- overflow events affecting SLO, paging, autoscaling, security, or chargeback metrics;
- time from detection to impact classification;
- time to remove accidental high-cardinality attributes;
- percentage of critical metrics with documented attribute contracts and owners;
- instrumentation changes that introduced or resolved overflow;
- process-memory and backend-series effects of approved limit changes.
These measures keep the focus on decision reliability rather than treating lower cardinality as an end in itself. A metric with more dimensions is not automatically bad, and a metric below the limit is not automatically useful. The goal is controlled, explainable dimensionality that supports known operational decisions.
What leaders should ask
Engineering and operations leaders do not need to tune every SDK. They do need assurance that observability data remains fit for the decisions built on it.
Five questions expose the maturity of the operating model:
- Can we identify every critical SLO, alert, or scaling policy whose source metric is currently overflowing?
- Do dashboard users see when an attribute breakdown is incomplete?
- Are metric attributes reviewed as part of software and instrumentation changes?
- Can teams explain why a cardinality limit was chosen and what active set it assumes?
- Is recovery verified through query completeness, not only through the absence of an SDK error?
If the answer to these questions is no, the organization may be making operational decisions from totals that are correct and dimensions that are not.
Protect the meaning of the metric
OpenTelemetry cardinality limits are valuable. They prevent a badly modeled metric from consuming unbounded process memory. The overflow marker also makes the trade-off observable in a vendor-neutral way.
But the safety mechanism changes what the metric can prove. Once measurements enter overflow, the total survives while attribute-level evidence becomes incomplete. That distinction matters anywhere metrics drive reliability commitments, automated actions, cost allocation, or executive reporting.
The right enterprise response is not to disable the protection or raise every limit. It is to govern metric semantics: define bounded attributes, monitor overflow continuously, connect it to business impact, control instrumentation changes, and verify recovery. Observability is trustworthy only when teams know not just that data arrived, but whether it still supports the decision they intend to make.