Skip to content
Release Radar

k6 Operator

Testing

Kubernetes operator for distributed k6 load testing

grafana/k6-operatorWebsiteDocumentation
v1.6.056 days after v1.5.0
View on GitHub

Part of Grafana

Release history

v1.6.0

CompareGitHub

✨ New features

To bring PrivateLoadZone mode closer to how other modes of testing in GCk6 work, we've expanded the API and what is configured through it. (issue)

.spec.args in TestRun CRD

To fully accommodate the above change, a new field .spec.args was added to a TestRun CRD (PR): it takes an array of strings as arguments to k6, which allows setting arbitrary values as tags and env vars of the k6 process. This is, in effect, a way to bypass the complex tricks around string escaping and the inability to set certain values in .spec.arguments (issue), which was a fundamental limitation of that configuration type. The new way to set k6 arguments in TestRun CRD looks like this:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: k6-sample-with-args
spec:
  parallelism: 2
  script:
    configMap:
      name: k6-test
      file: test.js
  args:
    - --tag
    - note=arbitrary values are possible "here"  # passed to k6 as is
    - -e
    - PRICE_LABEL=$$100 # passed to k6 as is
    - -e
    - BASE_URL=$(BASE_URL) # Kubernetes env var will be expanded
  runner:
    env:
      - name: BASE_URL
        value: https://test.k6.io

When .spec.args is present, it overrides .spec.arguments. But if .spec.args is empty, the .spec.arguments work as before, so this change is backwards compatible.

The PLZ tests were switched to using the .spec.args under the hood.

[!WARNING] Please make sure to update CRD definitions properly during upgrade, to avoid any issues.

🐛 Bug fixes

We finally have a fix for trying to create start / stop jobs more than once (issue). It has been spamming logs with false positive messages. Thanks, @Guflly!

📦 Helm

The bug with invalid lengthy names during Helm chart installation was solved by deliberately trimming all resource names to 63 characters. Thanks, @moko-poi!

[!WARNING] If you have been using a lengthy release name with service.enabled: false, you might encounter the rename of Deployment as part of this upgrade. On a healthy cluster, it shouldn't be an issue. But it makes sense to keep an eye on observability setup, if any: watch out if it picks up the change in name correctly in this case.

🛠️ Maintenance

kube-api-linter was added as a custom plugin to our golangci-lint workflow, with some basic rules enabled already. Thanks, @moko-poi!

There was a cleanup of potential bugs with shared state around the GCk6 client in the TestRun controller. That should make k6-operator more reliable at scale.

A small refactor to improve readability of the default values in the code (issue). Thanks, @moko-poi!

Key dependencies update:

  • Golang bumped to 1.26 (we're almost at the 1.27 release now)
  • Kubernetes group bumped to 1.36
  • controller-runtime bumped to 0.24.1
  • go.k6.io/k6/v2 bumped to v2.2.0

<!--

Container images

controller: ghcr.io/grafana/k6-operator:controller-v1.6.0 starter: ghcr.io/grafana/k6-operator:starter-v1.6.0 runner: ghcr.io/grafana/k6-operator:runner-v1.6.0 runner_k6_version: v2.2.0 helm_chart_version: 4.6.0 -->

Full changelog: https://github.com/grafana/k6-operator/compare/v1.5.0...v1.6.0

⚠️ Deprecation warning

.spec.scuttle is deprecated and will be removed in the future. Please see the docs on how to set up Istio and update your TestRuns. Note: This change comes with the RBAC update to publish Kubernetes events.

✨ New features

We've added the configuration of schedulerName to all Pods in the TestRun. Thanks, @The-Flash-Routine!

🐛 Bug fixes

Initializer Shell command was refactored into a multi-line script to track additional error cases, like missing k6 binary and other non-standard errors from k6 invocation (issue). Now all such cases are logged by the initializer Pod, and its Job fails explicitly. This should help diagnose the errors in the k6 setup.

terminationGracePeriodSeconds is set to the default value (30s) in all Pods, removing an accidental drift between runners and other Pods. We do not recommend interrupting k6 runners, but if it can't be avoided, this change should help preserve some useful data.

📦 Helm

There was a bug in the Helm chart causing erratic and hard-to-diagnose behaviour of the PrivateLoadZone CRD, which was fixed and released as chart 4.4.1. Full description of the impact is here.

🛠️ Maintenance

This is another maintenance-heavy release.

e2e tests were switched to using expect() instead of fail() in k6 tests: that guarantees the call to teardown in case of failure. This is esp. important in tests based on the xk6-environment.

There has been a lot of work ongoing to improve our security stance:

  • Helm release workflow was moved to use the GitHub App Token Broker (GATB) system.
  • Pinning digests in dependencies, including ones in Dockerfile. Updates of the digests will be handled by renovate.
    • This change also means that the GO_BUILDER_IMG argument is no longer supported.
  • Improving memory handling to prevent accidental leaks and tightening concurrency flow in PLZ.

Notable dependency updates:

  • go.k6.io/k6/v2 instead of v1 as before
  • golangci/golangci-lint to v2.12.1

Last but not least, the release process was heavily revamped to maximize the amount of automation and reduce the maintainer's toil.

<!--

Container images

controller: ghcr.io/grafana/k6-operator:controller-v1.5.0 starter: ghcr.io/grafana/k6-operator:starter-v1.5.0 runner: ghcr.io/grafana/k6-operator:runner-v1.5.0 runner_k6_version: v2.0.0 helm_chart_version: 4.5.0 -->

Full changelog: https://github.com/grafana/k6-operator/compare/v1.4.0...v1.5.0

✨ New features

Default tags in k6 metrics

We have changed the hard-coded tag passed to all runner pods: previously, it was job_name. Now, it is testrun_name. As described in the issue, this change removes duplicate metrics and reduces cardinality.

[!WARNING] Some dashboards around k6-operator test runs might need updating.

E.g., if you have a query like this: my_cool_metric{job_name=~"my-testrun-.*"} You should change it to: my_cool_metric{testrun_name="my-testrun", instance_id=~".*"}

PrivateLoadZone CRD: Secrets support

PrivateLoadZone tests now support secrets out-of-the-box, issue. See more details about this feature in the docs and in the video here.

TLDR: if you have secrets stored in Grafana Cloud k6 and a script that references them, you can run this script with PrivateLoadZone as well.

PrivateLoadZone CRD: pod template expansion

We expanded the pod template in the PrivateLoadZone CRD to support labels and annotations, issue. This configuration will be applied to all Pods started by k6-operator for this PrivateLoadZone.

As mentioned in the release notes for 1.3.0, this is done gradually via relaxation of validation rules. In practice, this means that only a limited number of fields are available at the moment, but there will be more in the future.

[!NOTE] PrivateLoadZone doesn't support mutability yet, so it must be re-created anew if you want to add a new config to it.

🛠️ Maintenance

Notable version updates:

  • go.mod at Go 1.25
  • Docker images at Go 1.26
  • golangci-lint at 2.10.1

There has been a lot of polishing around unit tests and e2e tests to improve overall structure and simplify onboarding and long-term maintenance. Among those changes, xk6-environment was finally refactored, addressing a long-running tech debt, and can now be built easily from its repo. This extension is used for isolation of e2e tests.

Lastly, there is now a govulncheck GitHub Action workflow, PR, running weekly and for PRs.

Full Changelog: https://github.com/grafana/k6-operator/compare/v1.3.2...v1.4.0

✨ New features

We added two major features in this release.

Firstly, there is now an option to skip initializer pod step by setting .spec.initializer.disabled: true in the TestRun CRD. Thanks, @Kristina-Pianykh! Note:

  • By skipping the initializer pod, there's a risk of starting N runner pods and having them all fail on a misconfigured script.
  • This option is recommended for advanced users who have a stable testing setup and have done due diligence for their scripts as described here.
  • Lastly, this option will be ignored for cloud output tests as they require initializer pod execution.

Secondly, we're adding a new option of PodTemplate to the PrivateLoadZone CRD to unlock extended configuration. It supports only a limited number of fields now:

apiVersion: k6.io/v1alpha1
kind: PrivateLoadZone
metadata:
  name: <NAME>
  namespace: <NS>
spec:
  token: <TOKEN>
  resources:
    limits:
      cpu: 400m
      memory: 1000Mi

  podTemplate: # a new, optional field
    spec:
      securityContext:
        runAsUser: 100
        runAsGroup: 100
        fsGroup: 100
      tolerations:
        - key: "app"
          operator: "Equal"
          value: "blue"
          effect: "NoSchedule"
      containers:
        - name: k6
          securityContext:
            allowPrivilegeEscalation: false

No other field can be passed to the .spec.podTemplate: it'll be blocked by Kubernetes validation, with an error about unknown field. This configuration will be applied to all Pods started by k6-operator for this PrivateLoadZone.

[!NOTE] PrivateLoadZone doesn't support mutability yet, so it must be re-created anew if you want to add a new config to it.

🛠️ Maintenance

Notable updates from the automated renovate bot:

  • Update of module go.k6.io/k6 to v1.6.1 (PR)

We now have a JSON schema validation check for all new PRs: if there's a PR opened for the Helm chart, a new GitHub Workflow will check if the JSON schema was updated as it should be, without waiting for human review. Thanks, @railgun-0402!

A couple of new commands were added to the Makefile to simplify maintenance:

  • make e2e-update-latest to help populate e2e/latest folder (PR). At the moment, it is mainly meant for the release process.
  • make patch-helm-crd to help copy changes in CRD to the Helm chart (PR). It can be used both in the release process and during normal PRs.

Full Changelog: https://github.com/grafana/k6-operator/compare/v1.2.0...v1.3.0

✨ New features

PrivateLoadZone tests got an enhancement around logic for the setup function. Now, when setup fails with an error, it causes an abort of the test.

🐛 Bug fixes

There had been a regression of volume claim setup: it was fixed and released as v1.1.1.

The resources field in PrivateLoadZone CRD wasn't being validated at the k6-operator level and resulted in a rather obscure error from the Cloud. It is now validated early with a CEL validation rule: .resources.limits cannot be empty.

📦 Helm

It is now possible to pass optional manager.dnsConfig and manager.dnsPolicy to the Helm chart. Thanks, @kworkbee!

A couple of bugs in the Helm chart were fixed:

  • Service labels weren't set in Service as expected. Thanks, @kworkbee!
  • Namespaced mode (rbac.namespaced=true) is fully functional now: it creates Roles instead of ClusterRoles where applicable.
    • Note: this mode sets the WATCH_NAMESPACE environment variable to point to the namespace with all resources. Don't use this mode together with custom WATCH_NAMESPACE values in manager.env: the deployment might not work.

🛠️ Maintenance

Quarterly maintenance is part of this release:

  • controller-runtime to v0.22.4
  • k8s group to v0.34.1
  • go.k6.io/k6 to v1.4.2
  • controller-tools to v0.19.0

Additional small updates to CI have also been included.

Lastly, we're now relying on renovate to help with dependency updates: the initial config was added, and we'll be polishing it more in the future.

Full Changelog: https://github.com/grafana/k6-operator/compare/v1.1.1...v1.2.0

✨ New features

There are a couple of additions to the TestRun CRD in this release:

  1. It's now possible to set .spec.runner.priorityClassName, .spec.starter.priorityClassName, and .spec.initializer.priorityClassName to help avoid unwanted evictions of pods. Thanks, @vsoloviov!

  2. Init containers can have custom resources set as .spec.runner.initContainers[*].resources, for cases when preparation for the test run requires more resources. Thanks, @gcaldasnu!

Additionally, this release contains a resolution to configurable path in VolumeClaim. Thanks, @moko-poi! Now it's possible to set the path like this:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: k6-test-with-pvc
spec:
  script:
    volumeClaim:
      name: dynamic-pvc
      file: /foo/script.js # Path to the script
      readOnly: true
  parallelism: 1

It's a backwards-compatible change, so existing TestRuns should continue to work as is.

Another small improvement was to improve log output of curl containers, turning them to JSON. Thanks, @moko-poi!

Last but not least, BackoffLimit of starter Job is now set to zero, to avoid additional creation of starter or stopper pods on failure to reach k6 runners. It's worth noting that curl containers are configured with three retries to ensure that such a failure is not transient. Thanks, @moko-poi!

🐛 Bug fixes

A simple validation for .spec.arguments has been added to avoid accidental Golang panic on misconfigured TestRun CRD. Note, this is not a full validation of all possible arguments: such validation is a job of k6 CLI and is expected to be done by a user before writing it down in the TestRun spec.

📦 Helm

Helm chart received a fix to ensure that manager.serviceAccount.create option is taken into account. Thanks, @bcrisp4!

🛠️ Maintenance

Logs of the k6-operator were adjusted to include the host value for GCk6 API calls, to assist with troubleshooting.

Full Changelog: https://github.com/grafana/k6-operator/compare/v1.0.0...v1.1.0

:tada: k6-operator 1.0 is here!

We're happy to announce k6-operator v1.0.0: a milestone that marks our commitment to the k6-operator project, switching to Semantic Versioning 2.0.0, and improving the stability guarantees. Starting from this release, we're formalizing our approach to versioning, release schedule, and maintenance updates.

We wouldn't have been here if not for the support of our amazing community. Thank you for providing feedback and so often lending a helping hand. :yellow_heart: :blue_heart:

📜 Documentation

Regular maintenance updates are a must for long-term stability of the k6-operator project. Now, we've added the description of how maintenance updates are handled in the k6-operator.

With the switch to Semantic Versioning, we've formalized our understanding of what each type of version increase means in the k6-operator. Read more on that in this doc.

Additionally, we commit to making regular minor releases every 8 weeks. The planning for each release can be seen in the corresponding GitHub milestone.

Last but not least, we've published an upgrade guide to help you set up your upgrade workflows around the k6-operator deployment.

✨ New features

We've fixed the technical debt when aggregation variables for metrics were not passed from Grafana Cloud k6 to the k6 runners in PrivateLoadZone tests. Now, the PLZ tests execute in a similar way to how cloud output tests do, in terms of metrics processing.

🐛 Bug fixes

This release contains a bug fix to validate that the .spec.parallelism value in the TestRun CRD is positive. If it's not, a corresponding error message will appear in the logs.

📦 Helm

There were a couple of additions to the Helm chart:

  • Added the service.portName configuration option to the Helm chart. This allows you to configure a name for the HTTP port where metrics of the k6-operator app are served.
  • Added the manager.logging.development boolean configuration option to the Helm chart. This allows you to switch the default logging level from development mode to production mode. Refer to the issue for the details. Thanks, @Kristina-Pianykh!

🛠️ Maintenance

In this release, we're bumping most of our Golang dependencies:

  • Dockerfile Golang 1.25; PR.
  • controller-runtime 0.22.1; PR.
  • k8s.io group 0.34.0; PR.
  • go.k6.io 1.2.3; PR.
  • golangci-lint v2.4.0; PR.
  • Other Golang dependencies brought up to latest; PR.

Full Changelog: https://github.com/grafana/k6-operator/compare/v0.0.23...v1.0.0

✨ New features

We improved the security stance of PrivateLoadZone tests: now the Grafana Cloud k6 token is not visible in k6 Pods' definitions. This enables admins to configure the cluster so that users of the PrivateLoadZone have access to the k6 Pods but not to the GCK6 token.

Starting from this release, the k6 Operator no longer uses Scuttle-based images for runner pods by default (issue). Instead, it is using plain grafana/k6:latest, which is guaranteed to point to the latest official release of k6.

:warning: Deprecation warning

The image of Scuttle-based runner will still be built on each release, so if you need it, you can configure it with .spec.runner.image. However, we're going to deprecate that image as well as remove Scuttle from TestRun CRD and the k6 Operator. If you are currently using Scuttle, please switch to using native sidecars. See this issue for details, and this documentation for how to configure Istio with the k6 Operator on up-to-date clusters.

:bug: Bug fixes

The .spec.script.volumeClaim.readOnly option is now set for VolumeMount instead of Volume. This fix moves the read-only option to the k6 container level, allowing for greater flexibility in complex setups. Thanks, @The-Flash-Routine! If you have been using this option, please double-check if this fix changes any implicit behaviour in your TestRun workflow.

In rare cases of bad timing, PrivateLoadZone would fail to deregister upon deletion. The logic for it has now been improved.

📦 Helm

There have been a few issues with the Helm release 3.14 after the kubebuilder update. Thanks to the reports from our users and contributions from @chris-pinola-rf, these were fixed and released as patch releases. PRs: https://github.com/grafana/k6-operator/pull/599 https://github.com/grafana/k6-operator/pull/606 https://github.com/grafana/k6-operator/pull/620

If you encounter an issue with Helm setup, please share the details.

📜 Documentation

This release comes with several significant additions to the documentation.

Firstly, there is a machine-generated Markdown with full reference to the CRD types. It can be accessed in the docs folder here.

Secondly, there is a contributing guide that describes the main points to pay attention to when creating an issue or PR for the k6 Operator. It can be accessed in the CONTRIBUTING.md

The troubleshooting guide was split into the TestRun part and PrivateLoadZone part, with some additions and clarifications.

We've also added the Istio guide to the public documentation here. If you need to use the k6 Operator on the cluster with Istio, please refer to this guide.

Finally, there were several other smaller improvements to the docs. Thanks to @mostafa and @heitortsergent for the help!

:hammer_and_wrench: Maintenance

There were small fixes to the e2e test suite (https://github.com/grafana/k6-operator/pull/608 https://github.com/grafana/k6-operator/pull/622). Additionally, controller-tools was updated to v0.18.0.

Full Changelog: https://github.com/grafana/k6-operator/compare/v0.0.22...v0.0.23