Skip to content
Release Radar

Prometheus Operator

Monitoring & Observability

Kubernetes-native deployment and management of Prometheus

v0.94.114 days after v0.94.0
View on GitHub

Part of Prometheus

Release history

v0.94.1

CompareGitHub
  • [BUGFIX] Restore update permission on finalizer subresources in the operator's ClusterRole, required by OwnerReferencesPermissionEnforcement when blockOwnerDeletion is set. #8830
  • [CHANGE] Update the default Alertmanager version to v0.34.0 and discard zero-value duration fields (retention, clusterGossipInterval, clusterPushpullInterval, clusterPeerTimeout) in Alertmanager resources instead of passing them as CLI flags, which Alertmanager rejects and can cause startup failures. Ignored fields are reported via the IgnoredFields status condition. #8800
  • [CHANGE] Add named enum types for Hetzner, Docker Swarm, and OpenStack service discovery fields in the ScrapeConfig CRD (HetznerRole, DockerSwarmRole, OpenStackAvailability). #8789
  • [CHANGE] Reject empty strings in namespaceDiscovery.names, consulSDConfig.services, and consulSDConfig.tags list fields in the ScrapeConfig CRD. #8786
  • [FEATURE] Add retentionPercentage field to Prometheus and PrometheusAgent CRDs for volume-based retention (requires Prometheus >= v3.11.0). Setting only the percentage no longer falls back to the default 24h time-based retention. #8728
  • [FEATURE] Add clusterPeerName field to Alertmanager CRD to override the --cluster.peer-name flag (requires Alertmanager >= v0.30.0). #8767
  • [FEATURE] Expose status conditions as Prometheus metrics for Prometheus, PrometheusAgent, Alertmanager, and ThanosRuler resources (prometheus_operator_<resource>_status_condition). #8719
  • [ENHANCEMENT] Tighten the operator's ClusterRole by replacing wildcard verbs with explicit permissions per resource. #8752
  • [ENHANCEMENT] Support webhook_url_file for Discord receiver in Alertmanager configuration Secret (requires Alertmanager >= v0.28.0). #8035
  • [BUGFIX] Require Thanos >= v0.42.0 for delayed compaction with object storage uploads, raised from v0.41.0. Thanos v0.41.0 mis-validates the delayed-compaction paths and the sidecar crash-loops; deployments pinned to v0.41.0 fall back to disabling local compaction. #8764
  • [BUGFIX] Add URL validation for Mattermost receiver fields in Alertmanager configuration Secret. #8225
  • [BUGFIX] Fix operator panic when AlertmanagerConfig Pushover receivers use userKeyFile or tokenFile instead of secret selectors. #8775
  • [BUGFIX] Fix operator panic when kubelet Service synchronization fails and the service is unavailable. #8743
  • [BUGFIX] Fix resources stuck during deletion when the informer misses the deletion timestamp update. #8727
  • [BUGFIX] Fix TracingConfig fields in Prometheus CRD not being serialized due to invalid JSON struct tags. #8730
  • [BUGFIX] Fix externalId version gating for Alertmanager SigV4 configuration to require version >= v0.34.0 (was incorrectly set to v0.33.0). #8759
  • [BUGFIX] Fix duplicate kubelet targets for nodes reporting several addresses of the same IP family. #8739
  • [BUGFIX] Fix argument list for Thanos containers when custom TLS ciphers or curves are specified. #8749
  • [CHANGE] Switch from uint to int types in the Go definition of the CRDs and add API validations to reject negative values. #8662 #8666
  • [CHANGE] Keep local Prometheus compaction enabled when the Thanos sidecar uploads to object storage, for Prometheus >= v3.9.0 and Thanos >= v0.41.0. The operator now coordinates uploads through the shipper meta file (--storage.tsdb.delay-compact-file.path, --shipper.meta-file-name, --shipper.ignore-unequal-block-size) instead of disabling compaction. Set spec.disableCompaction: true to keep the previous behavior. #8694
  • [CHANGE] Define a default value of 1 for .spec.shards for Prometheus and PrometheusAgent CRDs. #8691
  • [CHANGE/BUGFIX] Add validation markers to all unsigned int fields to reject negative values. #8662
  • [CHANGE/BUGFIX] Disable metadata sending when the remote-write configuration uses message version v2.0. #8700
  • [FEATURE] Add updateMessage field to the Slack receiver in the AlertmanagerConfig CRD. #8506
  • [FEATURE] Add chunkEncoding field to TSDBSpec for runtime float encoding selection in Prometheus and PrometheusAgent CRDs. #8675
  • [ENHANCEMENT] Use pod's name as the peer name for Alertmanager >= v0.30.0. #8705
  • [BUGFIX] Create IPv6 EndpointSlice for the kubelet Service on dual-stack clusters. #8682
  • [BUGFIX] Fix "namespace not found" errors when the operator watches monitoring and workload resources in different resources. #8658

[!NOTE] The PrometheusTopologySharding and PrometheusShardRetentionPolicy feature gates have been promoted to Beta in this release and are now enabled by default. See the sharding documentation for details.

  • [CHANGE] Add URL validation for the tokenUrl field in OAuth2 configuration across all CRDs. #8579
  • [CHANGE] Add URL validation for the url field in RemoteReadSpec in Prometheus CRD. #8596
  • [FEATURE] Migrate retention options from CLI flags to the config file for Prometheus CRD (Prometheus >= v3 uses the config file; older versions continue to use CLI flags). #8547
  • [FEATURE] Add staleSeriesCompactionThreshold field to TSDBSpec in Prometheus and PrometheusAgent CRDs. #8563
  • [FEATURE] Add labelNameUnderscoreSanitization and labelNamePreserveMultipleUnderscores fields to OTLPConfig in Prometheus and PrometheusAgent CRDs. #8562
  • [FEATURE] Add payload field to Webhook receiver in AlertmanagerConfig CRD. #8507
  • [ENHANCEMENT] Use pod topology labels for zone sharding on Kubernetes >= 1.35 when the PrometheusTopologySharding feature gate is enabled (removes the need for attachMetadata.node=true). #8564
  • [ENHANCEMENT] Add validation for the Slack update_message field in Alertmanager configuration Secret. #8556
  • [BUGFIX] Validate target labels in Probe static configuration to prevent invalid Prometheus scrape configs. #7901
  • [BUGFIX] Fix goroutine leak and data race in pollBasedListerWatcher. #8593
  • [BUGFIX] Validate ProxyConfig in OAuth2 configuration. #8610
  • [BUGFIX] Fix SMTP smarthost format error handling in Alertmanager configuration. #8586
  • [BUGFIX] Fix missing return in admission webhook after marshal failure. #8582
  • [BUGFIX] Fix FindOwner to return nil on meta.Accessor error. #8585
  • [BUGFIX] Fix dropped gzip Close errors in GzipConfig and GunzipConfig. #8573
  • [BUGFIX] Fix panic on malformed key=value flag input (e.g. --labels "key"). #8560
  • [CHANGE] Enforce mutual exclusion of basicAuth, authorization and oauth2 in ScrapeConfig CRD. #8480
  • [CHANGE] Add minimum length validations to string fields in ScrapeConfig CRD. #8479
  • [CHANGE] Add validations for VictorOps receiver in AlertmanagerConfig CRD. #8220
  • [CHANGE] Add validations for OpsGenie receiver in AlertmanagerConfig CRD. #8267
  • [CHANGE] Add validations for Email receiver in AlertmanagerConfig CRD. #8270
  • [FEATURE] Implement shard retention based on Prometheus data retention (it requires the PrometheusShardRetentionPolicy feature gate). #8478
  • [FEATURE] Configure node selector when sharding mode is Topology for Prometheus and PrometheusAgent custom resources (it requires the PrometheusTopologySharding feature gate). #8486
  • [FEATURE] Configure external label with topology information when sharding mode is Topology for Prometheus and PrometheusAgent custom resources (it requires the PrometheusTopologySharding feature gate). #8519
  • [FEATURE] Distribute scrape targets within topology zones when sharding mode is Topology for Prometheus and PrometheusAgent custom resources (it requires the PrometheusTopologySharding feature gate). #8538
  • [FEATURE] Add --promql-options CLI argument to the admission-webhook binary. #8531
  • [FEATURE] Validate PrometheusRule resources selected by Prometheus resources based on the PromQL enabled features. #8545
  • [FEATURE] Add workload identity authentication method for AzureSD in ScrapeConfig CRD. #8489
  • [ENHANCEMENT] Support strategic merge patch of container probes when workloads are configured with HTTPS. #8427
  • [ENHANCEMENT] Support auth_secret_file field for Email receiver in Alertmanager configuration Secret. #8394
  • [ENHANCEMENT] Support smtp_auth_secret_file field in Alertmanager configuration Secret. #8396
  • [ENHANCEMENT] Add externalId field to SigV4 configuration in Alertmanager, Prometheus, PrometheusAgent and ThanosRuler CRDs. #8494
  • [ENHANCEMENT] Add cipherSuites support for Thanos Sidecars and Rulers. #8524
  • [ENHANCEMENT] Add curves support for Thanos Sidecars and Rulers. #8542
  • [ENHANCEMENT] Speed up configuration reloads by watching the config file's parent directory. #7366
  • [ENHANCEMENT] Support Mattermost global webhook URL support in Alertmanager configuration Secret. #8501
  • [ENHANCEMENT] Add Mattermost global webhook URL support in Alertmanager CRD. #8503 #8534
  • [ENHANCEMENT] Support payload field for Webhook receiver in Alertmanager configuration Secret. #8505
  • [ENHANCEMENT] Support attachment fields for Mattermost receiver in Alertmanager configuration Secret. #8508
  • [ENHANCEMENT] Support update_message field for Slack receiver in Alertmanager configuration Secret. #8502
  • [ENHANCEMENT] Add threading configuration for email receiver in AlertmanagerConfig CRD. #8400
  • [ENHANCEMENT] Add healthFilter field for ConsulSD in ScrapeConfig CRD. #8529
  • [BUGFIX] Ensure that inactive shards don't scrape any targets when the sharding retention policy is Retain. #8513
  • [BUGFIX] Fix Telegram bot token validation in Alertmanager configuration Secret. #8465
  • [BUGFIX] Fix Probe ignoring HTTP client settings in scrape configuration. #8461
  • [CHANGE/BUGFIX] Validate that the remote-write URL scheme is either http or https. #8455
  • [FEATURE] Add --repair-policy-for-statefulsets CLI argument to the operator. It defines how the operator manages StatefulSet's pods stuck at an incorrect revision. Users running Kubernetes v1.35+ are encouraged to enable this feature (see troubleshooting guide). #8443
  • [FEATURE] Add schedulerName support to the Prometheus, PrometheusAgent, Alertmanager and ThanosRuler CRDs. #8451
  • [ENHANCEMENT] Add --web.tls-curves CLI argument to the operator and admission-webhook binaries. #8385
  • [ENHANCEMENT] Support minimum TLS version for Thanos gRPC servers. #8438
  • [ENHANCEMENT] Add version label to ThanosRuler pods. #8441
  • [ENHANCEMENT] Add messageText support for Slack receiver in AlertmanagerConfig CRD. #8374
  • [ENHANCEMENT] Add messageText support for Slack receiver in Alertmanager secret config. #8375
  • [ENHANCEMENT] Add forceImplicitTLS support for SMTP email config in Alertmanager secret config. #8384 #8404
  • [ENHANCEMENT] Add forceImplicitTLS support for SMTP email config in AlertmanagerConfig CRD. #8386
  • [ENHANCEMENT] Add forceImplicitTLS support for SMTP global config in Alertmanager secret config. #8405
  • [ENHANCEMENT] Add forceImplicitTLS support for SMTP global config in Alertmanager CRD. #8406
  • [ENHANCEMENT] Add support for global Telegram bot token in Alertmanager CRD. #8372
  • [ENHANCEMENT] Add chatIDFile support for Telegram receiver in Alertmanager secret config. #8376
  • [ENHANCEMENT] Add wechatAPISecretFile support in Alertmanager global config. #8377
  • [ENHANCEMENT] Add authSecretFile support for email config in Alertmanager secret config. #8396
  • [ENHANCEMENT] Add nested field support for PagerDuty description in Alertmanager secret config. #8402
  • [ENHANCEMENT] Add email threading support in Alertmanager secret config. #8388
  • [ENHANCEMENT] Add field and label selectors for ConfigMap watches. #8368
  • [ENHANCEMENT] Improve ScrapeConfig API consistency and validation. #8422
  • [BUGFIX] Fix ThanosRuler config resource status not being updated on initial StatefulSet creation. #8358
  • [BUGFIX] Preserve LastTransitionTime in Prometheus status conditions. #8346
  • [BUGFIX] Make Mattermost text field optional in AlertmanagerConfig CRD. #8363
  • [BUGFIX] Remove nil error wrapping in v1alpha1 duplicate receiver validation. #8379
  • [BUGFIX] Aggregate Available condition across Prometheus shards. #8434
  • [BUGFIX] Reconcile resources with inconsistent status. #8397
  • [BUGFIX] Fix namespace lister/watcher compatibility with Kubernetes v1.35 client-go. #8431
  • [BUGFIX] Fix missing OAuth2 field in IonosSDConfig generation. #8433
  • [BUGFIX] Fix missing fields in AzureSDConfig. #8444
  • [BUGFIX] Validate Microsoft Teams V2 URL in AlertmanagerConfig CRD. #8227
  • [BUGFIX] Fix labelmap relabel action rejecting valid replacement values with template variables for Prometheus 2.x. #8337
  • [ENHANCEMENT] Add hostNetwork field to the Alertmanager CRD. #8281
  • [ENHANCEMENT] Add the crds and full-crds commands to the operator's binary. #8251
  • [ENHANCEMENT] Report deprecated field usage in the Reconciled condition type. #8236
  • [ENHANCEMENT] Avoid unnecessary reconciliation upon creation of the ThanosRuler StatefulSet. #8347
  • [ENHANCEMENT] Add bodySizeLimit to the ScrapeConfig CRD. #8348
  • [ENHANCEMENT] Support http_headers field in the Alertmanager Secret. #8357
  • [ENHANCEMENT] Add the -kubelet-http-metrics flag to enable/disable the HTTP metrics port in the Kubelet endpoint (default=enabled). #8350
  • [ENHANCEMENT] Include operator.prometheus.io/version annotation in the full version of CRDs. #8279
  • [BUGFIX] Validate VictorOps global configuration in the Alertmanager CRD. #8020
  • [BUGFIX] Validate Jira global configuration in the Alertmanager CRD. #8265
  • [BUGFIX] Validate VictorOps receiver's URL in the AlertmanagerConfig CRD. #8258
  • [BUGFIX] Validate Webex receiver's URL in the AlertmanagerConfig CRD. #8255
  • [BUGFIX] Validate Jira receiver's URL configuration in the AlertmanagerConfig CRD. #8230
  • [BUGFIX] Validate OpsGenie receiver configuration in the AlertmanagerConfig CRD. #8267
  • [BUGFIX] Validate WeChat receiver configuration in the AlertmanagerConfig CRD. #8271
  • [BUGFIX] Validate SNS receiver configuration in the AlertmanagerConfig CRD. #8217
  • [BUGFIX] Validate Webex global configuration in the Alertmanager CRD. #7979
  • [BUGFIX] Validate Telegram global configuration in the Alertmanager CRD. #8268
  • [BUGFIX] Restore statefulset's labels if the creation fails with AlreadyExists. #8343
  • [BUGFIX] Fix potential panic due to informer cache races. #8310
  • [BUGFIX] Support probers defined with IPv6 addresses in the Probe CRD. #8354
  • [BUGFIX] Prevent group and repeat intervals with zero duration from breaking Alertmanager. #8126
  • [BUGFIX] Propagate all supported RocketChat attributes for AlertmanagerConfig CRD. #8016
  • [BUGFIX] Add URL validation for WeChat receiver. #8256
  • [BUGFIX] Add URL validation for SNS receiver. #8259
  • [BUGFIX] Fix GCE service discovery for the ScrapeConfig CRD. #8284
  • [BUGFIX] Avoid stale conditions in Alertmanager, ThanosRuler, Prometheus and PrometheusAgent resources. #8304
  • [BUGFIX] Fix race condition when updating rule ConfigMaps. #8290
  • [BUGFIX] Fix race condition when patching finalizers. #8323
  • [BUGFIX] Reconcile ScrapeConfig resources when namespace selection changes. #8334
  • [BUGFIX] Validate webhookURL secret for MSTeams receiver in AlertmanagerConfig CRD. #8294
  • [BUGFIX] Revert maximum version check for EC2/Lightsail SD in ScrapeConfig CRD. #8308
  • [BUGFIX] Relax URL validation in Slack receiver in AlertmanagerConfig CRD to support Go templates. #8299 #8331
  • [BUGFIX] Relax URL validation in PagerDuty in AlertmanagerConfig CRD to support Go templates. #8319
  • [BUGFIX] Relax URL validation in WebhookConfig in AlertmanagerConfig CRD to support Go templates. #8307 #8317
  • [BUGFIX] Relax URL validation in RocketChat receiver in AlertmanagerConfig CRD to support Go templates. #8318
  • [BUGFIX] Relax URL validation in Pushover receiver in AlertmanagerConfig CRD to support Go templates. #8307 #8316
  • [CHANGE] Use narrower selectors for StatefulSet informers in Alertmanager and ThanosRuler controllers. It is recommended to upgrade from v0.85.0 (at least). #8246
  • [CHANGE] Reject EC2/Lightsail SD for Prometheus >= 3.8.0 in ScrapeConfig CRD. #8175
  • [FEATURE] Add podManagementPolicy field to Prometheus, PrometheusAgent, Alertmanager and ThanosRuler CRDs. #8119
  • [FEATURE] Add updateStrategy field to Prometheus, PrometheusAgent, Alertmanager and ThanosRuler CRDs. #8202
  • [FEATURE] Add scrapeNativeHistograms field to Prometheus, PrometheusAgent, ServiceMonitor, PodMonitor, Probe and ScrapeConfig CRDs. #8102
  • [FEATURE] Add scope field to AzureAD remote write configuration. #8240
  • [FEATURE] Add workloadIdentity field to AzureAD remote write configuration. #7998
  • [FEATURE] Add support for PrometheusRule fields groupLabels and queryOffset in ThanosRuler. #8137
  • [FEATURE] Add slackAppToken and slackAppUrl fields to Alertmanager global config for Slack App support. #8238
  • [FEATURE] Add incident.io receiver support to Alertmanager config secret. #8190 #8245 #8228
  • [FEATURE] Add Mattermost receiver support to Alertmanager config secret. #8188
  • [FEATURE] Add apiType field to Jira receiver in Alertmanager config secret. #8218
  • [FEATURE] Add timeout field to PagerDuty receiver in AlertmanagerConfig CRD. #8162
  • [FEATURE] Add timeout field to Slack receiver in AlertmanagerConfig CRD. #8161
  • [ENHANCEMENT] Use minReadySeconds to set --dispatch.start-delay in Alertmanager. #8177 #8201
  • [ENHANCEMENT] Expose native histograms in operator metrics. #8194
  • [ENHANCEMENT] Add NoSelectedResources reason to status conditions. #8124
  • [ENHANCEMENT] Add enableHttp2 and followRedirects fields to HTTP configuration for Probe CRD. #8112
  • [ENHANCEMENT] Add CEL validations for DaemonSet mode in PrometheusAgent CRD (requires the PrometheusAgentDaemonSetFeature featuregate). #7881
  • [ENHANCEMENT] Improve validation for Pushover, PagerDuty and VictorOps receivers in AlertmanagerConfig CRD. #8239 #8113 #8220
  • [ENHANCEMENT] Add apiURL validation for WeChat, OpsGenie and Telegram receivers in AlertmanagerConfig CRD. #8196 #8206 #8199
  • [ENHANCEMENT] Validate URL fields in AlertmanagerConfig receivers (MSTeams, Webhook). #8231 #8125
  • [ENHANCEMENT] Validate URL fields in Alertmanager configuration secret (WeChat, Telegram, Pushover receivers). #7977 #8233 #8232

New Contributors

  • [BUGFIX] Fix the generated Alertmanager configuration for html and monospace fields of pushoverConfig receiver in AlertmanagerConfig CRD. #8153