Skip to content
Release Radar

Valkey Operator

Databases

Kubernetes operator for Valkey clusters

valkey-io/valkey-operatorWebsiteDocumentation
v0.7.025 days after v0.6.0
View on GitHub

Part of Databases

Release history

v0.7.0

CompareGitHub

v0.7.0 adds mTLS certificate-based ACL authentication, hostname-based cluster discovery, version-gated config and a configurable cluster-node-timeout, along with fixes to failure detection and roll recovery. Thank you to everyone who contributed to this release!

Upgrade notes

Kubernetes 1.32 or newer is required

The CRDs now use the CEL format library to validate serverName. A 1.31 API server rejects both CRDs, so the minimum supported version moves from 1.31 to 1.32 (#466, closes #465).

Apply the CRDs before upgrading the operator

The v0.7.0 operator writes new fields (clusterDomain, tls.serverName, tls.clientAuth) to each ValkeyNode. Against the v0.6.0 CRDs those fields are pruned, the operator sees a diff on every reconcile, and the cluster stays in Reconciling/UpdatingNodes without finishing the upgrade roll.

Apply the v0.7.0 CRDs first:

kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.7.0/config/crd/bases/valkey.io_valkeyclusters.yaml
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.7.0/config/crd/bases/valkey.io_valkeynodes.yaml

Helm does not upgrade CRDs shipped in a chart's crds/ directory, so this step applies to Helm installs too. --force-conflicts is needed there, because Helm owns the CRD fields it installed and a plain server-side apply is rejected.

The operator needs delete on pods

The node controller now deletes a pod that its StatefulSet can no longer replace (#410). The pods rule in the ClusterRole gains delete. The kustomize manifests and the v0.7.0 valkey-operator Helm chart include it. If you maintain your own RBAC, add the verb before upgrading.

Upgrading triggers a rolling restart

The pod template changes for every ValkeyCluster, so all pods are rolled one node at a time, replicas first, with a proactive failover before each primary. What changes the template:

  • cluster-node-timeout 2000 is no longer part of the operator's base config (#450), which changes the config hash annotation.
  • Cluster-owned StatefulSets move to the cluster's headless Service as spec.serviceName (#378). serviceName is immutable, so the operator orphan-deletes each StatefulSet and recreates it during that node's roll. The pod is adopted and then replaced by the roll.
  • On TLS clusters, the health probes no longer present the server certificate as a client certificate unless clientAuth.mode is Required (#242), and the metrics exporter gets REDIS_EXPORTER_TLS_SERVER_NAME (#409).

cluster-node-timeout now defaults to 15 seconds

The operator used to hardcode cluster-node-timeout 2000. It now leaves the directive unset unless you set it in spec.config, so clusters move to the Valkey default of 15000 ms after the upgrade roll (#450, closes #434).

Failure detection and automatic failover take longer as a result. A primary that crashes and restarts within 15 seconds rejoins as the primary rather than being failed over. Without persistence it reloads its last RDB snapshot, and its replicas resync from it, so writes since that snapshot are lost. To keep the previous behaviour, set:

spec:
  config:
    cluster-node-timeout: "2000"

Changing spec.config.cluster-node-timeout rolls the cluster.

New TLSEndpointWarning condition on TLS clusters

A TLS cluster that announces pod IPs (the default) now reports TLSEndpointWarning=True with reason TLSWithIPAnnounce. Clients that follow CLUSTER SLOTS to a pod IP usually fail certificate name checks. The condition does not affect Ready. Set networking.discovery.preferredEndpointType: Hostname to clear it (see below).

Features

mTLS certificate-based ACL authentication (#242)

spec.networking.tls.clientAuth controls client certificates:

  • mode: Optional (default, renders tls-auth-clients optional as before), Required (tls-auth-clients yes) or Disabled (tls-auth-clients no).
  • certificateUser: CN or URI maps the client certificate to the ACL user of that name (tls-auth-clients-user), so the client authenticates without AUTH. CN requires Valkey 9.0 and URI requires Valkey 9.1. The default, Disabled, leaves the directive out.

With mode: Required, the operator, probes, exporter and replication links present the node's server certificate. See docs/mtls.md.

Hostname-based discovery (#378, closes #365)

spec.networking.discovery.preferredEndpointType: Hostname makes each node announce <pod>.valkey-<name>.<namespace>.svc.<clusterDomain>, so TLS clients can verify the certificate after following a MOVED or CLUSTER SLOTS redirect. It requires workloadType: StatefulSet. The default stays IP. spec.networking.clusterDomain sets the cluster DNS domain (default cluster.local). The announced name has no trailing dot, so it is valid for SNI (#429, closes #425).

TLS server name for operator connections (#409, closes #405)

spec.networking.tls.serverName sets the name the operator verifies when it dials a node by pod IP. It defaults to valkey-<name>.<namespace>.svc.<clusterDomain>, which was previously hardcoded. ValkeyNodes created before this release fall back to the same default until the cluster updates them (#468, closes #464).

Configurable cluster-node-timeout (#450, closes #434)

cluster-node-timeout can now be set in spec.config. See the upgrade note above for the default change.

Version-gated config (#307, closes #287)

The operator reads the Valkey version from the spec.image tag and drops spec.config directives the image does not support, reporting ConfigurationWarning with reason UnsupportedConfigDirective. tls-auto-reload-interval requires Valkey 9.1.0 or newer. On images whose version cannot be read from the tag (latest, digest-only, custom tags) gated directives are dropped.

Faster ValkeyNode.status.role (#320, closes #261)

status.role is now updated within a few seconds of a failover or restart, instead of on the 60 second resync. A 5 second poller watches live topology and triggers a node reconcile when the role changes. A restarted replica no longer reports primary while it resyncs, and a pod that is not ready reports an empty role.

pprof endpoint (#435)

--pprof-bind-address serves Go's pprof endpoints for profiling the operator. It is off by default. The endpoint is unauthenticated and exposes heap contents, so bind it to localhost. See the developer guide.

Fixes

A pod stuck on a superseded revision is replaced (#410, closes #408)

Under OrderedReady, the StatefulSet controller does not replace a pod that never becomes Ready, so correcting a bad spec had no effect until someone deleted the pod. The node controller now deletes a pod that is not Ready and is on a revision the StatefulSet has superseded, and emits SupersededPodDeleted. It leaves the pod alone while it is loading a dataset or syncing from its primary.

Failure flags are read from the node's peers (#442, closes #441)

The ready-shard count and failover target selection checked the node's own CLUSTER NODES line for fail/pfail, which never carries them. They now ask the peers, so a node its peers report as fail or fail? no longer counts toward a ready shard and is not chosen as a failover target.

A shard with an unidentifiable primary no longer blocks the whole cluster (#452, closes #451)

When the operator could not identify one shard's primary, it stopped reconciling every shard, including the steps that would have repaired that shard. It now skips only that shard's roll and keeps Ready withheld until the primary is identified.

serviceName migration race (#432, closes #424)

A StatefulSet recreate that raced its own delete could leave the pod without a StatefulSet. The operator now re-reads uncached and completes the create.

Lower memory use (#436)

Operator connections to Valkey use smaller buffers and command rings. Before this change, each short-lived control connection allocated about 1.3 MiB, and with the default 128Mi memory limit the operator could be OOM-killed on clusters of around 20 shards or more.

What's Changed

…(truncated)

v0.6.0 introduces TLS API improvements, ACL observability, and several correctness fixes - thank you to everyone who contributed to this release!

Breaking

ValkeyCluster TLS certificate path restructured

spec.networking.tls.certificate is replaced by spec.networking.tls.certificates.server. The secretName field is now required (minimum 1 character) when TLS is set. This is phase 1 of the spec.networking.tls tree (#377, closes #376, part of #360).

# before (v0.5.0)
spec:
  networking:
    tls:
      certificate:
        secretName: valkey-tls

# after (v0.6.0)
spec:
  networking:
    tls:
      certificates:
        server:
          secretName: valkey-tls

We immediately dropped support for the old spec.networking.tls.certificate structure, so upgrading CRDs against a live cluster will cause the secretName to be lost. You can perform these steps to avoid that:

  1. Stop the operator running v0.5.0
  2. Deploy the v0.6.0 CRDs
  3. Deploy the migrated ValkeyCluster CRs (from spec.networking.tls.certificate to spec.networking.tls.certificates.server)
  4. Start operator running v0.6.0

Upgrade notes

Upgrading triggers a rolling restart

The pod template changes on upgrade, so all pods will be rolled — one node at a time, replicas first, with proactive failover ahead of primaries. What changes the template:

  • The controller-internal serverConfigHash annotation is now computed differently (#381)
  • The default metrics exporter image is updated to v1.88.0 (#361), for clusters that do not pin exporter.image

Features

ACL revision tracking (#382, closes #369)

ACLApplied now tracks permission-only edits, not just user/password changes. A disabled sentinel user (_operator_acl_revision) in the aclfile makes the condition honest for all ACL mutations.

ACL apply failures surfaced on cluster status (#400, closes #396)

A failed ACL apply was only visible on the ValkeyNode. The ValkeyCluster now sets Degraded/ACLApplyFailed naming the affected nodes, so kubectl get valkeycluster shows the problem. Ready stays True since the data path is unaffected.

ACL commands validation (#398, closes #395)

commands.allow and commands.deny are now validated at admission. The previous CRD markers were malformed and generated no validation. Entries must be the bare command, category, or subcommand form (get, @read, client|setname); the operator adds the +/- sign itself.

Fixes

Proactive failover no longer triggers on no-op upgrades (#402, closes #401)

Spec changes that rendered an identical pod template (e.g. bool-to-pointer conversion) were misread as a roll and triggered a CLUSTER FAILOVER per shard primary. Now decided by comparing template hashes against WorkloadRevision.

Exporter sidecar preserved on partial overrides (#397, closes #394)

Setting image, resources, args, or securityContext on spec.exporter without explicitly setting enabled silently dropped the metrics sidecar. Now only an explicit enabled: false disables it.

Skip reconciliation during deletion (#374, closes #364)

Deleting a ValkeyCluster caused the operator to recreate resources in a loop. The reconciler now exits when DeletionTimestamp is set.

Default metrics exporter updated (#361)

Updated to v1.88.0.

What's Changed

New Contributors

Full Changelog: https://github.com/valkey-io/valkey-operator/compare/v0.5.0...v0.6.0

v0.5.0 introduces a number of enhancements and bug fixes to ValkeyCluster - thank you to everyone who contributed to this release!

Breaking

ValkeyCluster TLS moved under spec.networking

spec.tls is removed. TLS configuration now lives at spec.networking.tls, and certificate.secretName is required when TLS is set. This is phase 1a of the spec.networking tree (#318).

# before
spec:
  tls:
    certificate:
      secretName: valkey-tls

# after
spec:
  networking:
    tls:
      certificate:
        secretName: valkey-tls

We immediately dropped support for spec.tls, so upgrading CRDs against a live cluster will cause it to downgrade to plaintext (no TLS). You can perform these steps to avoid that:

  1. Stop the operator running 0.4
  2. Deploy the 0.5 CRDs
  3. Deploy the migrated ValkeyCluster CRs (from spec.tls to spec.networking.tls)
  4. Start operator running 0.5

We have done this change to align with bucketing all networking related configs under one field networking, of which more features will be added in the future.

ACL reload restart required

A manual restart of the cluster is required in order to handle missing ACL LOAD permissions.

Features

Zone scheduling axis (#340)

  • scheduling.zone.spread mirrors scheduling.node.spread on topology.kubernetes.io/zone, with shard, primaries, and pods dimensions
  • All three render as topology spread constraints, so a shard with more members than there are zones balances across them rather than becoming unschedulable
  • Admission rejects combinations that would emit duplicate constraints

Zone pinning (#344)

  • scheduling.zone.pinning.zones assigns every pod a fixed zone by round-robin: zones[(shardIndex + nodeIndex) % len(zones)]
  • Adding shards or replicas never moves an existing pod
  • The list is immutable while set
  • Read the persistence warning in the docs before enabling it on a cluster with persistent volumes

Live ACL changes (#319)

  • user and password changes now apply to running nodes via ACL LOAD, with no pod restart
  • A new ACLApplied condition on ValkeyNode reports when the change is live on the server
  • Because a Valkey user can hold several passwords at once, a credential rotation can be done with no auth gap: add the new password, wait for ACLApplied, move clients across, then drop the old one

Staged pod template rolls (#338)

  • the ValkeyCluster controller now owns the authorized pod template via ValkeyNode.spec.workloadRevision
  • Template changes that previously restarted every pod at once (including operator upgrades) now roll one node at a time, replicas first, with failover ahead of a primary
  • A new WorkloadRollPending condition marks a node waiting its turn

Exporter arguments (#345)

  • exporter.args passes command-line flags to the metrics sidecar
  • The connection settings the operator previously passed as flags are now environment variables, so anything you set in args takes precedence

Fixes

Cluster recovers from a simultaneous restart of every node (#333)

  • When all pods restarted at once and every IP changed, each node's persisted nodes.conf pointed at dead addresses with no surviving gossip path, and the cluster stayed Reconciling until someone ran CLUSTER MEET by hand
  • The operator now detects peers whose node ID belongs to a live member at a different address and re-MEETs them, emitting a StaleAddressesHealed event
  • This also covers the case where an operator upgrade itself restarts every pod

Dual-channel replication (#359)

  • The _replication system user was missing +sync
  • With dual-channel-replication-enabled yes, the RDB side channel issues a legacy SYNC, which failed with -NOPERM and left the replica retrying the handshake indefinitely

valkey_operator_build_info labels (#336)

  • Published images reported empty version and branch

Operator ACL (#341)

  • The _operator user is granted config|get so applied configuration can be read back and audited

Logging (#330)

  • "getting system users secret" dropped from info to debug

Known issues

ACLApplied does not signal permission-only changes (#369)

  • The condition compares the user set and password hashes, so adding or removing a user or password moves it through PendingPropagation back to True
  • A change to a user's enabled flag or permissions still takes effect on the server, but the condition does not report a transient state for it, so it is not a signal to wait on for those fields

What's Changed

New Contributors

Full Changelog: https://github.com/valkey-io/valkey-operator/compare/v0.4.0...v0.5.0

⚠️ This contains a couple of breaking changes in how the ValkeyCluster spec is defined.

Notably updates below. A big thank you to the contributors in this release! 🎉

Breaking

New spec.scheduling API

We have moved these fields from the spec:

spec:
  affinity:
  tolerations:
  topologySpreadConstraints:
  nodeSelector:
  priorityClassName:

They are now located under the spec.scheduling field:

spec:
  scheduling:
    affinity:
    tolerations:
    topologySpreadConstraints:
    nodeSelector:
    priorityClassName:

We have also introduced first-class support for node spread topologies:

spec:
  scheduling:
    node:
      spread:
        shard:
          mode: Disabled
        primaries:
          mode: Disabled
        pods:
          mode: Disabled

If you were previously using this configuration to prevent pods in a shard colocating on the same node:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule

Then you will want to use this instead:

spec:
  scheduling:
    node:
      spread:
        shard:
          mode: Required

This now uses a pod anti affinity under the hood instead of a topology spread constraint. Switching to this prevents a conflict in constraints should another be defined against the topology key used for spreading on hosts, against a whenUnsatisfiable condition.

See the Scheduling docs for more info:

podDisruptionBudget refactor

spec.podDisruptionBudget is no longer a string but now a struct.

spec.podDisruptionBudget: Managed is now the equivalent of:

spec:
  podDisruptionBudget:
    mode: Cluster

spec.podDisruptionBudget: Disabled is now the equivalent of:

spec:
  podDisruptionBudget:
    mode: Disabled

Added

  • priorityClassName to ValkeyCluster.spec
  • terminationGracePeriodSeconds to ValkeyCluster.spec
  • You can now pull in valkey-operator as a go module via go get github.com/valkey-io/valkey-operator

Fixed

  • Resharding via atomic slot migration now works again

What's Changed

New Contributors

Full Changelog: https://github.com/valkey-io/valkey-operator/compare/v0.3.0...v0.4.0

This release of valkey-operator comes with incremental bug fixes and stability improvements. Notable additions include:

  • Added dedicated user for replication
  • Configure Valkey with shutdown-on-sigterm failover by default
  • Adding Prometheus metrics
  • Add podSecurityContext field to ValkeyCluster and ValkeyNode CRDs
  • Add metrics.exporter.securityContext field to ValkeyCluster CRD to override the SecurityContext applied to the exporter sidecar
  • Failover now targets the replica with the highest replication offset
  • Cluster recovery when a majority of primaries are lost
  • spec.shards is now required on ValkeyCluster resources; manifests that omit it will be rejected after the CRD upgrade

What's Changed

New Contributors

Full Changelog: https://github.com/valkey-io/valkey-operator/compare/v0.2.0...v0.3.0

This is our second alpha release of Valkey Operator which includes several features, improvements, and bug fixes since the last iteration, notables include:

  • Configuration applied live against nodes, without the need of rolling the pods
    • Only a small subset of configurations supported currently
  • Deploy a minimal pod disruption budget to ensure only 1 node in a cluster is disrupted
  • Support a topology spread constraint, allowing primaries and replicas of a shard to be placed on different Kubernetes nodes
  • Roll true replicas first before primaries
  • Support images from private container registries
  • Reduce the memory of the operator when deployed to a larger cluster
  • Support for alpine images
  • Several cluster stability improvements and fixes

Upgrading in-place is not supported, you may have to tear down your Valkey Cluster and redeploy it.

A huge thank you to all the contributors in this release! 🎉

What's Changed

New Contributors

Full Changelog: https://github.com/valkey-io/valkey-operator/compare/v0.1.0...v0.2.0

First alpha release of valkey-operator, supporting ValkeyCluster CRDs with some basic lifecycling operations.

This release serves as an initial release to begin testing the operator in non-production environments.

Please note: This is still in active development, breaking changes against the alpha CRD are expected. Data loss can be expected in this early release. Do not run this in production.

What's Changed

New Contributors

…(truncated)