Skip to content
Release Radar

Tempo Operator

Monitoring & Observability

Kubernetes operator for Grafana Tempo

grafana/tempo-operatorWebsiteDocumentation
v0.22.080 days after v0.21.0
View on GitHub

Part of Grafana

Release history

v0.22.0

CompareGitHub

🚩 Deprecations 🚩

  • tempostack, tempomonolithic: Warn that the Jaeger UI is deprecated when it is enabled. (#1587) The validating webhook now returns a warning when spec.template.queryFrontend.jaegerQuery.enabled of a TempoStack or spec.jaegerui.enabled of a TempoMonolithic is set to true:

    Warning: spec.template.queryFrontend.jaegerQuery.enabled is deprecated and will be removed in a future release

    This is only a warning, existing instances keep working and nothing else changes.

🚀 New components 🚀

  • metrics-generator: Add support for deploying metrics-generator via TempoStack resource. (#1120) To enable the metrics-generator, add a metricsGenerator section to the TempoStack spec:

    spec:
  template:
    metricsGenerator:
      enabled: true
      remoteWriteURLs:
        - http://prometheus:9090/api/v1/write
      processors: ['service-graphs', 'span-metrics', 'local-blocks']

    enabled must be set to true to deploy the metrics-generator component. remoteWriteURLs is required and must contain at least one Prometheus remote write endpoint. processors is optional. span-metrics, service-graphs, and local-blocks are enabled by default.

    Additional metrics-generator configuration can be set via extraConfig.tempo. Refer to the Tempo metrics-generator documentation for all available options.

💡 Enhancements 💡

  • tempostack, tempomonolithic: Add READY column to the output of kubectl get (#1600)

  • tempostack: Make it possible to disable the gateway's Ingress when tenant mode is OpenShift (#1416)

  • operator: Update Kubernetes dependencies to 1.36, Golang to 1.26 and update gateway images (#1552)

  • tempostack, tempomonolithic: Bump Tempo to 2.10.8 and gateway to main-2026-08-13-db2b289 (#1611)

  • operator: Speed up initial pod rollout (#1581) Previously, certificate hash annotations were written to the TempoStack/TempoMonolithic CR and propagated to pod templates in a subsequent reconcile. This required three reconcile loops before pods were rolled out with the correct annotations. Now, hashes are passed directly through to the pod, completing the rollout in a single reconcile.

  • operator: Reduce the number of reconciliations by filtering events on owned resources (#1567)

  • tempostack: The TempoStack status condition should only be ready if all pods are ready (#1582)

  • tempostack: Create a PodDisruptionBudget for each TempoStack component to protect availability during voluntary disruptions. (#1577) A PodDisruptionBudget with maxUnavailable: 1 is now created for the distributor, ingester, querier, query-frontend, gateway and metrics-generator components (the gateway and metrics-generator budgets are only created when those components are enabled). This ensures that voluntary disruptions such as node drains during cluster upgrades can only take down a single replica of a component at a time. Following the Grafana Loki Operator, no budget is created for the compactor.

  • tempostack: Support zone-aware replication of a TempoStack via the new spec.replicationZones field. (#1585) spec.replicationZones spreads the pods of every component across the given topology domains using topology spread constraints, and configures the ingester ring to replicate spans across those zones:

    spec:
  replicationFactor: 3
  replicationZones:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone

    Make sure that spec.replicationFactor is less than or equal to the number of available zones.

    The behavior follows the Grafana Loki Operator, which configures the zones under spec.replication.zones instead.

  • tempostack: Default per-component replica counts from the t-shirt size so non-demo sizes are highly available. (#1345) Previously only the ingester replica count was derived from the size (via the replication factor quorum); all other components defaulted to a single replica, leaving them as single points of failure. Now every non-demo size (1x.pico, 1x.extra-small, 1x.small, 1x.medium) defaults each component to at least 2 replicas, and scales the throughput-bound components (distributor, querier, metrics-generator) at the larger sizes:

    componentpicoextra-smallsmallmedium
    distributor2234
    querier2235
    query-frontend2222
    compactor2223
    gateway2222
    metrics-generator2233

    The ingester replica count continues to be derived from the replication factor (quorum = floor(RF/2)+1). 1x.demo is unchanged and runs a single replica of every component. Explicit per-component replicas overrides always take precedence over the size defaults.

🧰 Bug fixes 🧰

  • operator: Fix stale status field by watching Deployment/StatefulSet status changes. (#1533)
  • tempostack: Mark spec.managementState and spec.tenants.mode optional so they are no longer listed as required in the generated CRDs. Both have defaults, so offline validators (e.g. kubeconform) no longer reject valid resources that omit them. (#1491)
  • operator: Refactor finalizer logic to avoid concurrent writes (#1579)
  • operator: Sort hostnames in certificates (#1584)

Components

Support

This release supports Kubernetes 1.25 to 1.36.

💡 Enhancements 💡

  • operator: Bump Golang to 1.25 (#1427)

  • operator: Bump observatorium/api gateway image to main-2026-05-19-489f301 (#1459)

  • operator: Bump Tempo to 2.10.5 (#1459)

  • tempostack, tempomonolithic: Add env and envFrom fields to inject environment variables into Tempo containers from Kubernetes Secrets or ConfigMaps. (#1135) The new spec.env and spec.envFrom fields allow injecting environment variables into all Tempo containers. Combined with spec.extraConfig and the -config.expand-env=true flag, this enables referencing sensitive values (e.g. a Redis cache password) stored in Kubernetes Secrets using ${VAR_NAME} syntax in the Tempo configuration. The -config.expand-env=true flag has also been added to TempoMonolithic for feature parity with TempoStack.

    Example TempoStack CR with a password-protected Redis cache:

    apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: tempo
spec:
  env:
    - name: REDIS_PASSWORD
      valueFrom:
        secretKeyRef:
          name: redis-creds
          key: password
  extraConfig:
    tempo:
      cache:
        caches:
          - redis:
              endpoint: "redis.host"
              password: "${REDIS_PASSWORD}"

  • tempostack, tempomonolithic: Support ca-bundle.crt key in storage TLS CA ConfigMap for OpenShift trusted CA bundle injection (#1437) ConfigMaps labeled with config.openshift.io/inject-trusted-cabundle use the key ca-bundle.crt, which is now recognized in addition to service-ca.crt and ca.crt.

  • tempostack, tempomonolithic: Use TLS profile (min version and ciphers) from the feature gate or OpenShift APIServer CR. (#1367) When feature gate openshift.clusterTLSPolicy is enabled the TLS profile (min version and ciphers) are obtained from OpenShift APIServer CR. The feature gate tlsProfile can set the profile explicitly and is active only when openshift.clusterTLSPolicy is disabled.

    The TLS profile is then used in all TLS connections, however it can be overridden if (min version or ciphers) are directly specified in the TLS config for a given component in the CR.

    The openshift.clusterTLSPolicy can be enabled via FEATURE_GATES env variable The tlsProfile feature gate can be enabled via TLS_PROFILE environment variable and valid values are: Old, Intermediate, Modern.

    On OpenShift the openshift.clusterTLSPolicy is enabled by default.

🧰 Bug fixes 🧰

  • tempostack, tempomonolithic: Set AWS_DEFAULT_REGION environment variable on Tempo pods when using S3 with CCO credential mode (#1419)

  • tempostack: Add missing ports to NetworkPolicies for query-frontend (Jaeger gRPC port 16685) and gateway (internal HTTP port 8081) (#6061)

  • tempostack, tempomonolithic: Fix race condition causing resources to get stuck in terminating state during deletion (#1423) Certificate management code paths were updating CR annotations (cert-hash, certRotationRequiredAt) without checking if the resource was being deleted. This caused conflicts with the foregroundDeletion finalizer, leaving resources stuck in terminating state for 600+ seconds.

  • tempostack: Add podAntiAffinity to gateway and compactor deployments to improve HA (#1422) Previously, the gateway and compactor deployments did not set podAntiAffinity, allowing all replicas to be scheduled on the same node. This reduces high availability. All other components (distributor, ingester, querier, query-frontend) already had podAntiAffinity configured.

  • tempomonolithic: Fix gateway OTLP HTTP forwarding to use HTTPS when receiver TLS is enabled (#1394)

  • tempomonolithic: Fix OTLP HTTP receiver to use internal mTLS when gateway and HTTPEncryption are enabled (#1415) When gateway and HTTPEncryption were both enabled, the OTLP HTTP receiver was not configured with the internal mTLS certificate (unlike the gRPC receiver), and the gateway always forwarded OTLP HTTP traffic over plain HTTP. This caused TLS verification failures and all OTLP HTTP trace ingestion to be silently dropped. The HTTP receiver and gateway OTLP HTTP upstream now match the gRPC behavior: internal mTLS is used for the receiver and the gateway connects over HTTPS whenever HTTPEncryption is enabled.

  • tempostack: Add resource requests/limits to the tempo-gateway-opa container when using percentage-based resource calculation (#1408)

  • tempostack: Always use HTTPS for S3 storage when token-based authentication is enabled (#1410)

Components

Support

This release supports Kubernetes 1.25 to 1.34.

🛑 Breaking changes 🛑

  • operator: Migrate operator configuration from ConfigMap to environment variables (#1348) The default operator deployment no longer uses a ConfigMap-based configuration file. Configuration is now managed through environment variables.

    To continue using a ConfigMap: Mount your controller_manager_config.yaml ConfigMap and add the --config=/path/to/config.yaml flag to the operator deployment args. Environment variables will still take precedence over the config file.

    To migrate to environment variables: Use FEATURE_GATES for boolean flags (comma-separated, prefix with - to disable) and individual env vars for other settings.

    Example OLM Subscription override:

    spec:
  config:
    env:
    - name: FEATURE_GATES
      value: "prometheusOperator,observability.metrics.createServiceMonitors,-networkPolicies"
    - name: TLS_PROFILE
      value: "Intermediate"

    Available feature gates: openshift.route, openshift.servingCertsService, openshift.oauthProxy, httpEncryption, grpcEncryption, prometheusOperator, grafanaOperator, builtInCertManagement, observability.metrics.createServiceMonitors, observability.metrics.createPrometheusRules, networkPolicies

    Other env vars: DISTRIBUTION, TLS_PROFILE, OPENSHIFT_BASE_DOMAIN, DEFAULT_POD_SECURITY_CONTEXT (JSON), BUILT_IN_CERT_MANAGEMENT_CA_VALIDITY, BUILT_IN_CERT_MANAGEMENT_CA_REFRESH, BUILT_IN_CERT_MANAGEMENT_CERT_VALIDITY, BUILT_IN_CERT_MANAGEMENT_CERT_REFRESH

💡 Enhancements 💡

  • tempostack, tempomonolithic: Add configurable defaultPodSecurityContext to operator config (#1240) The operator now supports a configurable defaultPodSecurityContext in the operator configuration. This allows setting default pod security context fields (fsGroup, runAsUser, etc.) that are merged into pod security contexts when specific fields are not set by the user. For Community distributions, fsGroup is set to 10001 to ensure volume permissions work correctly with certain CSI drivers (AWS EBS, DigitalOcean, etc.) that mount volumes with root:root ownership. For OpenShift, this is set to an empty object {} as SCCs manage security contexts automatically.

  • tempostack, tempomonolithic: Add MCP (Model Context Protocol) server configuration to TempoStack and TempoMonolithic CRDs (#1319) The MCP server allows AI assistants to query tracing data from Tempo. Enable it by setting spec.template.queryFrontend.mcpServer.enabled: true for TempoStack or spec.query.mcpServer.enabled: true for TempoMonolithic.

  • tempostack, tempomonolithic: Add support for extra labels on ServiceMonitor and PrometheusRule objects (#905)

  • networking: Add network policy support for Tempo operands (#1246) Network policies are now enabled by default for all TempoStacks.

    Enabling/Disabling Network Policies

    Network policies are enabled by default. To disable them:

    apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: example
spec:
  networkPolicy:
    enabled: false  # Default is true

    Generated Network Policies

    The operator creates the following network policies for each component:

    Query-Frontend

    Egress (outbound connections):

    • Storage backend (S3/Azure/GCS) - Required for initialization
    • Queriers (ports 9095, 3200) - For query distribution
    • OTLP telemetry export (when configured)

    Ingress (inbound connections):

    • From cluster (port 9095, 3200) - External query access
    • From gateway (when enabled) - For proxied queries
    • From queriers (ports 9095, 3200) - Bidirectional worker communication

    Gateway (when enabled)

    Egress (outbound connections):

    • Distributor (ports 4317, 4318) - For ingesting traces via OTLP
    • Query-frontend (ports 9095, 3200) - For querying traces
    • Query-frontend (ports 16686, 16687) - For Jaeger Query UI and metrics (when JaegerQuery is enabled)
    • Kubernetes API server (port 6443) - For TokenReview/SubjectAccessReview (when OpenShift RBAC multi-tenancy is enabled)
    • OTLP telemetry export (when configured)

    Ingress (inbound connections):

    • From cluster (ports 8080, 8090) - External HTTP/gRPC access for trace ingestion and queries

    Ingester

    Egress (outbound connections):

    • Storage backend (S3/Azure/GCS) - For writing trace blocks
    • OTLP telemetry export (when configured)

    Ingress (inbound connections):

    • From distributor (ports 9095, 3200) - For receiving traces
    • From querier (ports 9095, 3200) - For querying recent traces

    Storage Backend Support

    • S3 (in-cluster): Port extracted from endpoint URL, defaults to 443 (HTTPS) or 80 (HTTP). Allows egress to any namespace/pod for services like MinIO.
    • Azure Storage: Port 443 (HTTPS). Allows egress to external internet (0.0.0.0/0) for Azure Storage endpoints.
    • Google Cloud Storage (GCS): Port 443 (HTTPS). Allows egress to external internet (0.0.0.0/0) for GCS and OAuth2 endpoints (storage.googleapis.com, oauth2.googleapis.com).

    Common Policies

    All components also have:

    • DNS egress (to kube-dns or openshift-dns)
    • Gossip ring communication (memberlist protocol)
    • Metrics ingress (port 3200) - For Prometheus/monitoring scraping from in-cluster monitoring namespaces

  • tempostack, tempomonolithic: Show a warning if an instance without gateway is created on OpenShift (#1359)

  • tempostack: Add t-shirt sizes for simplified TempoStack resource configuration (#1345) Introduces predefined deployment size profiles via the new spec.size field. Available sizes: 1x.demo, 1x.pico, 1x.extra-small, 1x.small, 1x.medium. Each size maps to pre-tested resource configurations based on performance testing. Size also sets a default replication factor (1 for demo, 2 for others). Resource values are based on ingestion rates: extra-small (~100GB/day), small (~500GB/day), medium (~2TB/day).

  • operator: Update Kubernetes dependencies to v1.34.3 (#1324)

  • tempostack, tempomonolithic: Update Tempo to 2.10.0 (#1375) Update Tempo to 2.10.0. Changelog: https://github.com/grafana/tempo/releases/tag/v2.10.0

🧰 Bug fixes 🧰

  • tempostack: NetworkPolicies now dynamically discover Kubernetes API server endpoints and ports instead of hardcoding port 6443 (#1295) This fix resolves connectivity issues on managed Kubernetes services (e.g., Amazon EKS, Google GKE) that expose the Kubernetes API server on non-standard ports like 443 instead of the default 6443.

    What Changed

    The operator now automatically discovers the actual Kubernetes API server endpoints by querying the kubernetes EndpointSlice in the default namespace at reconcile time. This provides:

    • Dynamic port detection: Works with port 443 (EKS, some GKE configs), 6443 (standard K8s), or custom ports
    • Specific IP restrictions: When available, creates /32 CIDR rules for each control plane node IP
    • Fallback: Falls back to 0.0.0.0/0 with discovered port if EndpointSlice lookup fails

    Example

    On an EKS cluster with 3 control plane nodes, the operator now generates:

    spec:
  egress:
  - ports:
    - protocol: TCP
      port: 443  # Discovered from EndpointSlice
    to:
    - ipBlock:
        cidr: 100.105.216.2/32
    - ipBlock:
        cidr: 100.109.70.105/32
    - ipBlock:
        cidr: 100.114.146.103/32

    No configuration changes are required. The operator automatically adapts to cluster's API server configuration.

  • tempomonolithic: Use the gateway serving cert for ingestion TLS if the gateway is enabled (#1372)

Components

Support

This release supports Kubernetes 1.25 to 1.34.

💡 Enhancements 💡

  • tempostack, tempomonolithic: Update Tempo to 2.9.0 (#1308)

🧰 Bug fixes 🧰

  • tempomonolithic: Scrape tempo metrics for monolithic. (#1275)
  • tempostack: Restart pods when certificates are re-generated. (#1301)

Components

Support

This release supports Kubernetes 1.25 to 1.32.

🧰 Bug fixes 🧰

  • github action: Fix release workflow (#1243) Fix the image tag of the must-gather image.

Components

Support

This release supports Kubernetes 1.25 to 1.32.

💡 Enhancements 💡

🧰 Bug fixes 🧰

  • tempostack: Remove deprecated storage.trace.cache setting (#1136)

Components

Support

This release supports Kubernetes 1.25 to 1.32.

🛑 Breaking changes 🛑

  • tempostack, tempomonolithic: Ensure the operator does not grant additional permissions when enabling OpenShift tenancy mode (resolves CVE-2025-2786) (#1145) Ensure the permissions the operator is granting to the Tempo Service Account do not exceed the permissions of the user creating (or modifying) the Tempo instance when enabling OpenShift tenancy mode.

    To enable the OpenShift tenancy mode, the user must have permissions to create TokenReview and SubjectAccessReview.

    This breaking change does not affect existing Tempo instances in the cluster. However, the required permissions are now mandatory when creating or modifying a TempoStack or TempoMonolithic CR.

💡 Enhancements 💡

  • tempostack, tempomonolithic: Add short live token authentication for Azure Blob Storage (#1206) For use short live token on Azure, the secret should contain the following configuration:

    data:
  container:         # Azure blob storage container name
  account_name:      # Azure blob storage account name
  client_id:         # Azure managed identity clientID
  tenant_id:         # Azure tenant ID in which the managed identity lives.
  audience:          # (optional) Audience of the token, default to api://AzureADTokenExchange

  • tempostack, tempomonolithic: Support for AWS STS via cloudcredential operator (#1159)

  • tempostack, tempomonolithic: Add support for GCS Shot Live Token authentication. (#1141) Now storage secret for GCS can contain

    data:
  bucketname:         # Bucket name
  iam_sa:             # a name for your the Google IAM service account
  iam_sa_project_id:  # The project ID for your IAM service account.

  • tempostack, tempomonolithic: Set GOMEMLIMIT to 80% of memory limit, if any (#1196) This golang variable indicate to GoLang GC to be more aggressive when it is reaching out the memory limits. This is a soft limit, so still can produce OOM, but reduces the possibility.

  • operator: Kubernetes 1.32 enablement (#1157)

  • tempomonolithic: Watch storage secrets for tempo monolithic (#1181)

🧰 Bug fixes 🧰

  • tempostack, tempomonolithic: Add parameter to set audience in ID token for GCP Workload Identity Federation (#1209) Now that GCS token allow to set the audience, the secret configuration required channged, now it will require the following:

    data:
  bucketname:    # GCS Bucket  name
  audience:      # (Optional) default to openshift
  key.json:      # Credential file generated using gclient

    File key.json can be created using :

    gcloud iam workload-identity-pools create-cred-config \
  "projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL_ID>/providers/<PROVIDER_ID>" \
  --service-account="<SERVICE_ACCOUNT_EMAIL>" \
  --credential-source-file=/var/run/secrets/storage/serviceaccount/token \
  --credential-source-type=text \
  --output-file="/tmp/key.json"

    credential-source-file= Should be pointing to /var/run/secrets/storage/serviceaccount/token which is the locationn operator mounts the projected volume.

  • tempostack, tempomonolithic: Add namespace suffix to ClusterRole and ClusterRoleBinding of gateway (#1146) This resolves a naming conflict of the ClusterRole and ClusterRoleBinding when two TempoStack/TempoMonolithic instances with the same name, but in different namespaces are created. Only relevant when using multi-tenancy with OpenShift mode.

  • tempostack, tempomonolithic: Fix pruning of cluster-scoped resources (#1168) Previously, when a non-multitenant TempoStack instance was created using the same name as an existing multitenant TempoStack instance, the operator erroneously deleted the Gateway ClusterRole and ClusterRoleBinding associated with the multitenant instance.

    With this change, cluster-scoped resources get an additional label app.kubernetes.io/namespace to signify the namespace of the TempoStack owning this cluster-scoped resource.

  • tempostack, tempomonolithic: Cleanup gateway cluster roles and bindings after deleting tempo instance (#1190) Now the operator uses finalizer to clean up the cluster roles and bindings after deleting the tempo instance.

  • tempostack, tempomonolithic: Allow OpenShift cluster admins to see all attributes when RBAC is enabled. (#1185) This change removes --opa.admin-groups=system:cluster-admins,cluster-admin,dedicated-admin from the OpenShift OPA configuration. This configures the OPA to always return all user's accessible namespaces required by the RBAC feature.

  • tempostack, tempomonolithic: Don't set --opa.matcher=kubernetes_namespace_name when query RBAC is disabled (#1176)

  • tempostack: Fix unimplemented per tenant retention and fix per tenant overrides after tempo 2.3 (#1134) In tempo 2.3 https://github.com/grafana/tempo/blob/main/CHANGELOG.md#v230--2023-10-30 they changes the overrides config which was not properly implemented in the operator.

    This patch also adds support for per tenant retention which was not implemented.

  • tempostack, tempomonolithic: Assign a percentage of the resources to oauth-proxy if resources are not specified, fixed the name (#1107)

  • tempostack: Limit granted permissions of the Tempo Service Account when enabling the Jaeger UI Monitor tab on OpenShift (resolves CVE-2025-2842) (#1144) Previously, the operator assigned the cluster-monitoring-view ClusterRole to the Tempo Service Account when the Prometheus endpoint of the Jaeger UI Monitor tab is set to the Thanos Querier on OpenShift.

    With this change, the operator limits the granted permissions to only view metrics of the namespace of the Tempo instance. Additionally, the recommended port of the Thanos Querier service changed from 9091 to 9092 (tenancy-aware port): .spec.template.queryFrontend.jaegerQuery.monitorTab.prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092.

    All existing installations, which have the Thanos Querier configured at port 9091, will be upgraded automatically to use port 9092.

  • tempostack, tempomonolithic: Update Tempo to 2.7.2 (#1149)

Components

Support

This release supports Kubernetes 1.25 to 1.32.

💡 Enhancements 💡

  • tempomonolithic: Add support for query RBAC (#1131) This feature allows users to apply query RBAC in the multitenancy mode. The RBAC allows filtering span/resource/scope attributes and events based on the namespaces which a user querying the data can access. For instance, a user can only see attributes from namespaces it can access.

    spec:
  query:
    rbac:
      enabled: true

Components