Skip to content
Release Radar

Tempo

Monitoring & Observability

High-scale distributed tracing backend

grafana/tempoWebsiteDocumentation
v3.0.3same day as v2.10.8
View on GitHub

Part of Grafana

Release history

v3.0.3

CompareGitHub

Security

  • operations: Update Go to 1.26.5 (CVE-2026-39822, CVE-2026-27145, CVE-2026-42504, CVE-2026-42505, CVE-2026-42507), google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf), golang.org/x/net to v0.56.0 (CVE-2026-46600), golang.org/x/text to v0.39.0 (CVE-2026-56852), and go.opentelemetry.io/otel to v1.44.0 (CVE-2026-41178) (#7726) (@mattdurham)

Enhancements

  • query-frontend: Add Tempo configuration documentation to the MCP server via the docs-config tool and docs://config/overview and docs://config/reference resources (#7521) (@knylander-grafana) The configuration reference is generated from the default configuration, so it stays in sync with the code.
  • query-frontend: Update the TraceQL and metrics documentation served by the MCP server to match current capabilities. (#7375) (@knylander-grafana)

Bug fixes

  • tempo: Fix incorrect version reported by --version, the build-info metric, and /api/status/buildinfo. The build version is now read from the new top-level VERSION file instead of the most recently created git tag, which could belong to a different release. (#7469) (@zhxiaogg)

Changes

  • docs: remove guidance on running multiple monolithic instances. (#7636) (@mattdurham)

Security

  • operations: Update Go to 1.26.5 (CVE-2026-39822, CVE-2026-27145, CVE-2026-42504, CVE-2026-42505, CVE-2026-42507), google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf), golang.org/x/net to v0.56.0 (CVE-2026-46600), golang.org/x/text to v0.39.0 (CVE-2026-56852), and go.opentelemetry.io/otel to v1.44.0 (CVE-2026-41178) (#7725) (@mattdurham)

Security

  • operations: Update google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf) and go.opentelemetry.io/otel to v1.44.0 (CVE-2026-41178) (#7724) (@mattdurham)

🔒 Security 🔒

  • operations: Update Go to 1.26.5 and bump vendored golang.org/x/net and golang.org/x/text to fix CVE-2026-39822, CVE-2026-42504, CVE-2026-27145, CVE-2026-42505, CVE-2026-42507, CVE-2026-46600, and CVE-2026-56852 (#7641) (@mattdurham)
  • deps: Update github.com/apache/thrift to v0.23.0 to pick up upstream security fixes. (#7119) (@renovate)
  • deps: Update golang.org/x/crypto to v0.52.0 to pick up upstream security fixes. (#7258, #7343) (@renovate)
  • deps: Update golang.org/x/net to v0.55.0 to pick up upstream security fixes. (#7128, #7259, #7371) (@renovate)
  • deps: Update golang.org/x/sys to v0.44.0 to pick up upstream security fixes. (#7260) (@renovate)

🛑 Breaking changes 🛑

  • distributor: Dropped support for the OpenCensus receiver. (#7323) (@zhxiaogg)

🧰 Bug fixes 🧰

  • metrics-generator: Fix panic when write_relabel_configs is configured on remote write endpoints. (#7338) (@carles-grafana)

🧰 Bug fixes 🧰

  • tempo: Fix incorrect version reported by --version, the build-info metric, and /api/status/buildinfo. The build version is now read from the new top-level VERSION file instead of the most recently created git tag, which could belong to a different release. (#7469) (@zhxiaogg)

🔧 Changes 🔧

  • tempo: Stop publishing 32-bit ARM binary archives. Release artifacts continue to include amd64 and arm64 binaries. (#7106) (@javiermolinar)

🔒 Security 🔒

  • deps: Build Tempo with Go 1.26.3, which includes upstream security and bug fixes. (#7423) (@ie-pham)

🔒 Security 🔒

  • deps: Update github.com/apache/thrift to v0.23.0 to pick up upstream security fixes. (#7120) (@renovate)
  • deps: Update golang.org/x/crypto to v0.52.0 to pick up upstream security fixes. (#7262) (@renovate)
  • deps: Update golang.org/x/net to v0.55.0 to pick up upstream security fixes. (#7129, #7263) (@renovate)
  • deps: Build Tempo with Go 1.26.3, which includes upstream security and bug fixes. (#7422) (@ie-pham)

🛑 Breaking changes 🛑

  • distributor: Dropped support for the OpenCensus receiver. (#7322) (@zhxiaogg)

🔧 Changes 🔧

  • deps: Updated Prometheus to v0.311.3 and the OpenTelemetry Collector dependencies to v1.52, along with related transitive dependencies (dskit and others). (#7298) (@zhxiaogg)

Tempo 3.0 is a major release that completes the transition to the new ingest/write architecture, removes deprecated 2.x components, graduates TraceQL metrics to general availability, and adds migration tooling for Tempo 2.x users.

This release contains breaking configuration and deployment changes. Review the migration guide before upgrading, especially if you use legacy ingesters, legacy overrides, v2 blocks, OpenCensus, localblocks, or custom live-store/query-frontend settings.

Highlights

  • New ingest/write architecture replaces legacy ingesters.
  • TraceQL metrics is generally available; alerting on TraceQL metrics and the faster read path remain experimental.
  • vParquet5 improvements and faster metrics query paths.
  • Expanded metrics-generator controls for cardinality management.
  • Trace redaction support.
  • Span profiling support via otelpyroscope.
  • Live-store and block-builder correctness and observability fixes.

Useful links:

Features

  • Make individual AST transformations skippable via config and query hints by @stoewer in #7012
  • Add span profiling support via otelpyroscope. Enable with span_profiling: true or -span-profiling CLI flag to attach pprof labels to OTel spans by @simonswine in #7063
  • Add tempo-cli migrate config command for migrating Tempo 2.x configs to 3.0 by @mapno in #6982
  • jsonnet: Add KEDA-based horizontal pod autoscaling support for microservices deployment by @mapno in #6970
  • Add automemlimit support for automatic GOMEMLIMIT configuration. Enable with memory.automemlimit_enabled: true by @oleg-kozlyuk-grafana in #6313
  • Support comparison operators in TraceQL Metrics queries by @ruslan-mikhailov in #6474
  • metrics-generator: Add span filtering to service graphs through filter_policies by @javiermolinar in #6453
  • Add new include_any filter policy for spanmetrics filter by @javiermolinar in #6392
  • Add span_multiplier_key to overrides. This allows tenants to specify the attribute key used for span multiplier values to compensate for head-based sampling by @carles-grafana in #6260
  • metrics-generator: Add per-label limiter to control cardinality by @electron0zero in #6414
    • Adds max_cardinality_per_label per tenant override and new metrics to estimate per-label cardinality demand.
  • Add an extension mechanism for per-tenant overrides by @stoewer in #6758
  • Extend TraceRedactor interface to support hiding complete traces via ErrTraceHidden by @stoewer in #6811
  • Single-binary mode: push distributor local ingest directly to live-store and metrics-generator without Kafka by @javiermolinar in #6729
  • Add experimental drain limiter / span name sanitization to the metrics generator to reduce metrics cardinality by clustering similar span names by @Logiraptor in #6098

Enhancements

  • Query-frontend: split streamed search and metrics responses into smaller gRPC packets for better default client compatibility, and fix final streaming updates after metrics series limits are reached by @mdisibio in #6607 / #7239
  • live-store: lock-free block reads via atomic.Pointer[blockSnapshot]; block deletion is two-phase and crash-safe by @zhxiaogg in #7132
  • Support OR conditions for tag name and tag value autocomplete (search tags v2) by @ie-pham in #6827
  • Expose MinIO retry settings via S3 config by @rwhitty in #6561
  • Reduce default livestore WAL size and align query defaults: max_block_duration 1m to 30s, max_block_bytes 100MiB to 50MiB, complete_block_timeout 1h to 20m, metrics query_backend_after 30m to 15m by @zhxiaogg in #6974
  • Enable native histogram emission for all promauto-registered histograms, including tempo_request_duration_seconds. Both classic and native formats are emitted simultaneously; existing scrapers are unaffected by @zalegrala in #6910
  • tempo-cli: Add --header flag to query api commands for custom headers by @Nouuu in #6768
  • tempo-cli: add redact command to submit trace redaction jobs to the backend scheduler by @zalegrala in #6832
  • Block builder: deduplicate spans within traces during block creation and track removed duplicates via tempo_block_builder_spans_deduped_total metric by @zhxiaogg in #6539
  • metrics-generator: Support extracting span multiplier from W3C tracestate OTel probability sampling threshold via enable_tracestate_span_multiplier config option by @csmarchbanks in #6684
  • Add new alerts and runbook entries by @javiermolinar in #6276
  • Double the maximum number of dedicated string columns in vParquet5 and update tempo-cli to determine the optimum number for the data by @mdisibio in #6282
  • TraceQL metrics: experimental faster read path for most metrics queries, accessible behind the query hint spanonly_fetch=true when unsafe_query_hints is enabled by @mdisibio in #6359
  • TraceQL metrics: add new per-tenant override to opt in or opt out of the new experimental faster read path for most metrics queries by @mdisibio in #6849
  • Vulture: extend data consistency checks to include more strings, integers, and blobs, at resource/span/event scopes, and perform deeper trace content checks by @mdisibio in #6731
  • Improve attribute truncating observability by @javiermolinar in #6400
  • Log truncated oversized attributes by @carles-grafana in #6467
  • livestore: make trace_too_large log line an insight by @carles-grafana in #6371
  • Remove live-store partition owner from ring on shutdown to prevent stale owner entries by @oleg-kozlyuk-grafana in #6409
  • Improved live store readiness check and added readiness_target_lag and readiness_max_wait config parameters. Live store will now, if readiness_target_lag is set, not report /ready until Kafka lag is brought under the specified value by @oleg-kozlyuk-grafana and @ruslan-mikhailov in #6238 and #6405
  • Expose a new histogram metric to track the jobs per query distribution by @javiermolinar in #6343
  • Do deep validation for filter policies in user configurable overrides API by @electron0zero in #6407
  • Allow span_name_sanitization to be set via user-configurable overrides API by @Logiraptor in #6411
  • Add fail_on_high_lag parameter to allow live-store to fail if it is lagged by @ruslan-mikhailov and @carles-grafana in #6363, #6567, and #7066
  • Add support for per-tenant left-padding of trace IDs by @mapno in #6489
  • Add new metric for generator ring size: tempo_distributor_metrics_generator_tenant_ring_size by @zalegrala in #5686
  • Remove explicit runtime.GC() calls in vParquet5 compactor/block creation and CLI by @oleg-kozlyuk-grafana in #6603
  • Reduce allocations in extendReuseSlice growth path during WAL writes and block creation by @mapno in #6863
  • Implemented anti-affinity for pods in same livestore zone by @zhxiaogg in #6757
  • Livestore: skipped WAL complete op during shutdown by @zhxiaogg in #6839
  • Add metric to track livestore block cut reasons by @zhxiaogg in #6922
  • Enable async parquet read mode for WAL completion path by @zhxiaogg in #6967
  • metrics-generator: add leave_consumer_group_on_shutdown to send LeaveGroup on shutdown for immediate partition reassignment instead of waiting for session timeout by @zalegrala in #6575

Breaking Changes

Tempo 3.0 includes several operator-impacting removals and default changes. Review the migration guide before upgrading.

  • The legacy ingester-based write path has been removed. Deployments must migrate to the 3.0 ingest/write architecture.
  • Removed v2 block encoding and the legacy v2 compactor component by @joe-elliott in #6273
    • Removed v2-specific CLI commands: list block, list index, view index, gen index, and gen bloom.
  • Centralized block and WAL config: block_builder and live_store now always use storage.trace.block settings; per-module block config fields are removed by @stoewer in #6647
  • Recent data queries now guarantee complete results by failing when an instance is lagging. Defaults query_frontend.query_end_cutoff to 30s and live_store.fail_on_high_lag to true by @mapno in #7210 / #7232
  • Disabled legacy (flat, unscoped) overrides by default. Tempo refuses to start if legacy overrides are detected. Set enable_legacy_overrides: true or -config.enable-legacy-overrides=true to opt back in temporarily. Legacy overrides will be removed in a future release by @electron0zero in #6741
  • User-configurable overrides config metrics_generator.processors no longer merges with runtime overrides. metrics_generator.processors now takes precedence over runtime overrides, matching every other config in user-configurable overrides. Setting processors: [] disables all processors for the tenant by @electron0zero in #7176 / #7185
  • Enabled RetryInfo by default. distributor.retry_after_on_resource_exhausted now defaults to 5s (was 0) so OTLP clients receive a retry hint on ResourceExhausted errors by @electron0zero in #7088
    • Set to 0 to disable cluster-wide, or set the per-tenant override ingestion.retry_info_enabled: false to disable for a single tenant.
  • Removed duplicate compaction prefix from CompactorConfig CLI flags by @electron0zero in #6909
    • compaction.compaction.block-retention → compaction.block-retention
    • compaction.compaction.max-objects-per-block → `compaction.max-objects-pe

…(truncated)

  • [CHANGE] Set default max_result_limit for search to 256*1024 #6525 (@zhxiaogg)
  • [CHANGE] Upgrade Go to 1.26.2 to fix CVE-2026-25679 #6800 (@ie-pham)
  • [CHANGE] Patch gRPC modules and upgrade go.opentelemetry.io/otel/sdk to v1.40.0 to fix security vulnerabilities #6899 (@ie-pham)
  • [CHANGE] Upgrade go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp to v1.43.0 to fix security vulnerabilities #6890
  • [CHANGE] Upgrade go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp to v1.43.0 to fix security vulnerabilities #6889
  • [CHANGE] Upgrade github.com/go-jose/go-jose/v4 to v4.1.4 to fix security vulnerabilities #6908
  • [CHANGE] Upgrade github.com/antchfx/xpath to v1.3.6 to fix security vulnerabilities #6763
  • [BUGFIX] avoided unbounded mem by capping exemplars hint #6792 (@zhxiaogg)
  • [BUGFIX] updated go version to 1.26.2 for CVE-2026-25679 fix #6779 (@ie-pham)
  • [CHANGE] Set default max_result_limit for search to 256*1024 #6525 (@zhxiaogg)
  • [BUGFIX] avoided unbounded mem by capping exemplars hint #6792 (@zhxiaogg)
  • [CHANGE] Upgrade OTLP SDK #6904 (@ie-pham)
  • [BUGFIX] Update module go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp to v0.19.0 [security] #6891 (@renovate-sh-app)
  • [BUGFIX] Update module go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp to v1.43.0 [security] #6893 (@renovate-sh-app)
  • [BUGFIX] Update module go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp to v1.43.0 [security] #6892 (@renovate-sh-app)
  • [BUGFIX] Update module github.com/go-jose/go-jose/v4 to v4.1.4 [security] #6852 (@renovate-sh-app)
  • [BUGFIX] Update module github.com/antchfx/xpath to v1.3.6 [security] #6764 (@renovate-sh-app)
  • [BUGFIX] Update module google.golang.org/grpc to v1.79.3 [security] #6735 (@renovate-sh-app)
  • [BUGFIX] Update module github.com/buger/jsonparser to v1.1.2 [security] #6744 (@renovate-sh-app)