Introducing KubeRay v1.6 and v1.7

Special thanks to these KubeRay contributors: Anyscale (Jui-An Huang, Nai-Jui Yeh, Jun-Hao Wan), Google Cloud (Andrew Sy Kim, Ryan O'Leary, Aaron Liang, Richa Banker), Hark (Kai-Hsun Chen), Alibaba Cloud (Kun Wu), Microsoft (Alima Azamat, Mark Rossetti), Red Hat (Pat O’Connor, Laura Fitzgerald, Bryan Keane), opensource4you (Han-Ju Chen, Jia-Wei Jiang, Jie-Kai Chang, Che-Yu Wu, Fu-Sheng Chen, Chih-Ting Yeh)

We’re excited to announce the release of KubeRay v1.6 and v1.7, building on the strong momentum of Ray on Kubernetes. These releases introduce several enhancements for running Ray on Kubernetes,  incorporating over 381 commits contributed by approximately 75 contributors.

Key Highlights

• History Server: Beta release with faster startup, lower memory usage, ray auth token support, and automated collector sidecar injection.

• Security: Native NetworkPolicy support, automated mTLS certificate management, and Kubernetes RBAC-based authentication.

• RayService: Incremental upgrades graduate to beta with rollback support.

• RayJob: RayCronJob support, independently restartable sidecar submitters, and improved deletion policies.

• RayCluster: Recreate upgrade strategy, configurable built-in Ingress, Autoscaler V2 worker group priorities, and multi-host worker indexing graduate to beta.

• GCS Fault Tolerance: Embedded RocksDB storage eliminates the need for an external Redis dependency and improves cluster deletion reliability.

• Observability: Kubernetes platform events are surfaced in the Ray Dashboard to simplify infrastructure failure troubleshooting.

LinkHistory server (beta)

The KubeRay History Server is an independent deployment that lets you access a cluster's logs and events even after the RayCluster has been deleted. This is especially useful for RayJob, which creates an ephemeral Ray cluster for a single job and tears it down once the job completes. The History Server supports RayCluster, RayJob, and RayService.

It consists of two components: the collector and the history server. The collector runs as a sidecar container on head and worker pods and uploads logs and events to object storage; the history server is a standalone deployment that serves the Ray Dashboard API on top of that data.

The alpha version, introduced in v1.6, processed all events on every startup and kept them in memory. In v1.7, we re-architected it and are promoting it to beta. The new version adds lazy loading, LRU cache, and other improvements that reduce memory usage and startup time. It also includes the following highlighted features:

• A cluster selector page for easily navigating to the dashboard of any live or historical Ray cluster.

• The ability to view historical Ray Data datasets and Ray Serve applications

LinkAutomated collector sidecar injection (alpha)

KubeRay v1.7 introduces automated collector sidecar injection to make adopting the History Server easier. Previously, users had to manually define the collector container in every head and worker pod template, including its environment variables and volume mounts, as well as the event export settings on the Ray containers. Now, you can simply set historyServerOptions.collectorOptions in the RayCluster spec, and KubeRay automatically injects a properly configured collector sidecar into all head and worker pods.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-historyserver
spec:
  historyServerOptions:
    collectorOptions:
      image: quay.io/kuberay/collector:v1.7.0
      env:
      - name: STORAGE_BACKEND
        value: "GCS"
      - name: GCS_BUCKET
        value: "my-gcs-bucket"
  ...

This feature is disabled by default. To use it, enable the RayClusterHistoryServer feature gate.

LinkHistory Server authentication support

Ray token authentication (spec.authOptions) and the History Server were introduced independently in KubeRay—token authentication shipped in v1.5.1, followed by the History Server alpha in v1.6. Until now, the two features could not be used together: neither History Server component propagated the x-ray-authorization header required by the Ray Dashboard.

As a result, when token authentication was enabled, Dashboard-backed data such as job history and Serve applications could not be retrieved, while requests that relied on live cluster proxying failed altogether.

KubeRay v1.7 closes this gap in both History Server components as part of its promotion to beta, bringing History Server support to auth-enabled Ray clusters.

LinkCollector

The Collector now authenticates with the Ray Dashboard using either the shared token Secret or, when enableK8sTokenAuth is enabled, its projected Kubernetes ServiceAccount token. With automatic Collector sidecar injection, no additional configuration is required. KubeRay automatically derives the appropriate credentials from authOptions:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-historyserver-auth
spec:
  authOptions:
    mode: token
  historyServerOptions:
    collectorOptions:
      image: quay.io/kuberay/collector:v1.7.0

LinkHistory Server

Start it with --use-auth-token-mode=true to proxy to auth-enabled live clusters: it reads each cluster's auth Secret, attaches the credential, and strips client-supplied authorization headers so viewers cannot override it. Apply service_account_auth_token_mode.yaml for the required Secret read access.

Live cluster proxying supports the shared-token mode only; enableK8sTokenAuth works for the collector but not yet for the history server.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: historyserver
  labels:
    app: historyserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: historyserver
  template:
    metadata:
      labels:
        app: historyserver
    spec:
      serviceAccountName: historyserver
      containers:
      - name: historyserver
        ...
        image: historyserver:v0.1.0
        imagePullPolicy: IfNotPresent
        command:
        - historyserver
        - --use-auth-token-mode=true

LinkSecurity Enhancements

LinkNetwork policy support (alpha)

KubeRay v1.7 introduces native NetworkPolicy support. Instead of manually creating NetworkPolicy resources for the head and worker pods of each RayCluster, you can now configure networkPolicy directly in the RayCluster spec. Choose an isolation mode: DenyAll, DenyAllIngress, or DenyAllEgress, and explicitly allow traffic with ingress or egress rules for the head and worker pods. Traffic between pods within the same Ray cluster is never blocked.

The following example denies all traffic except Prometheus metrics scraping on the head pod:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-network-policy
spec:
  networkPolicy:
    mode: DenyAll
    head:
      ingressRules:
      - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <prometheus-namespace>
        ports:
        - port: 8080
          protocol: TCP

This feature is disabled by default. To use it, enable the RayClusterNetworkPolicy feature gate.

LinkmTLS support (alpha)

Starting with KubeRay v1.7, you no longer need to manually manage certificates when enabling mTLS for Ray. The operator integrates with cert-manager to automatically issue certificates for the head and worker pods, securing communication between them with mTLS.

To use it, simply add tlsOptions to the RayCluster spec:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-mtls
spec:
  tlsOptions:
    enabled: true

This feature requires enabling the RayClusterMTLS feature gate. Note that cert-manager must be installed in the cluster.

LinkToken authentication with Kubernetes RBAC

Token-based authentication has been supported since Ray 2.52.0 and KubeRay v1.5.1, where the RayCluster controller automatically generates a token and stores it in a Kubernetes Secret shared by all clients. KubeRay v1.6 enhances this in two ways:

1. Authentication and authorization can now be delegated entirely to Kubernetes. With authOptions.enableK8sTokenAuth enabled, clients authenticate using their own Kubernetes ServiceAccount tokens, and Ray authorizes each request by checking whether the caller has the ray:write custom verb on that specific RayCluster. This lets you control who can access a Ray cluster with standard Kubernetes RBAC, using per-user identities instead of a single shared token. This feature requires Ray 2.55.0 or later.
   

   apiVersion: ray.io/v1
   kind: RayCluster
   metadata:
     name: raycluster-k8s-auth
   spec:
     authOptions:
       mode: token
       enableK8sTokenAuth: true
     ...
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: ray-writer
   rules:
   - apiGroups: ["ray.io"]
     resources: ["rayclusters"]
     verbs: ["ray:write"]

   For details, please see the documentation.

2. Users can now provide their own token Secret by setting authOptions.secretName, instead of having KubeRay generate one. The Secret must contain the token under the auth_token data key.

   apiVersion: ray.io/v1
   kind: RayCluster
   metadata:
     name: ray-cluster-with-auth
   spec:
     enableInTreeAutoscaling: true
     authOptions:
       mode: token
       secretName: ray-cluster-token

LinkRayService Enhancements

LinkRayService Incremental Upgrade (beta)

The incremental upgrade strategy for RayService was introduced as an alpha feature in KubeRay v1.5. Since then, the feature has been enhanced with rollback support and several reliability improvements based on testing and user feedback.

Rollback allows an in-progress upgrade to be reverted by changing the RayService spec back to the version currently serving traffic. KubeRay then shifts traffic and capacity back to the original cluster step by step and removes the new one, with no downtime. For details, see the documentation.

As of KubeRay v1.7, the incremental upgrade strategy has graduated to beta. The RayServiceIncrementalUpgrade feature gate is enabled by default.

LinkRayService Kueue integration

Starting with Kueue v0.17.0 and KubeRay v1.6.0, Kueue manages the RayService custom resource as a top-level job instead of going through the RayCluster it creates. This enables RayService workloads to be dispatched across multiple Kueue-managed clusters through MultiKueue. See here for more details on KubeRay’s integration with Kueue.

There is currently a limitation with RayService upgrades, affecting both incremental upgrade and zero-downtime upgrade. During these upgrades, a RayService may temporarily manage two RayClusters while traffic is migrated from the old cluster to the new one. MultiKueue currently does not support Elastic RayService. If you're interested in the progress, see kueue#11102 for more details. 

LinkRayJob Enhancements

LinkRayCronJob (alpha)

Starting with KubeRay v1.6, you can use RayCronJob to schedule RayJobs natively on a cron schedule, without relying on external cron tools. KubeRay v1.7 adds time zone support, letting you specify the time zone used for scheduling. If no time zone is set, it defaults to the local time zone of the KubeRay controller.

The following example shows how to create a RayCronJob:

apiVersion: ray.io/v1
kind: RayCronJob
metadata:
  name: example-raycronjob
spec:
  schedule: "*/5 * * * *"
  timeZone: "America/Los_Angeles"
  jobTemplate:
    # Everything below here is a standard RayJob spec

This feature is disabled by default and requires enabling the RayCronJob feature gate. See documentation for more details.

LinkRayJob SidecarMode retry (alpha)

RayJob SidecarMode, introduced in KubeRay v1.5, runs the job submitter as a sidecar container in the head pod. KubeRay v1.7 introduces the SidecarSubmitterRestart feature gate. Leveraging improved error code handling in Ray 2.54.0 and the ContainerRestartRules feature, which is enabled by default in Kubernetes v1.35, the submitter container can now restart independently on transient errors without restarting the ray-head container. On restart, the submitter checks the job status first; if the job is still running, it simply reattaches to the log stream instead of resubmitting.

For more detailed information, please refer to the documentation.

LinkReduced wget dependency

Previously, KubeRay relied on wget for several health checks, including the readiness and liveness probes of Ray pods and the wait loop that RayJob SidecarMode uses to verify GCS health before submitting a job. This became a problem for Ray images that don't ship wget, such as the upcoming slim Ray images: probes would fail and sidecar job submission could break before the job even started.

Starting with KubeRay v1.6, this dependency is removed in most cases. The RayJob SidecarMode GCS wait loop now uses Python's urllib, which is always available since the submitter already depends on the Ray CLI, and this works across all supported Ray versions. In addition, Ray 2.53.0 introduced a unified /api/healthz health check endpoint, allowing KubeRay to switch pod probes to native Kubernetes HTTP probes instead of exec-based wget commands.

One case still requires a composite check: RayService worker readiness needs to verify both the local raylet and the Serve proxy, which a single HTTP endpoint cannot express today. Work is ongoing in the Ray community to introduce a unified serving readiness endpoint so that this last check can also become a plain HTTP probe (see ray#60925).

LinkRayJobDeletionPolicy (beta)

The RayJobDeletionPolicy feature, introduced as alpha in KubeRay v1.3, lets you configure fine-grained cleanup behavior after a RayJob finishes through spec.deletionStrategy, such as deleting the whole RayCluster, only the worker pods, or the RayJob itself. As of KubeRay v1.6, this feature has graduated to beta and the RayJobDeletionPolicy feature gate is enabled by default. For more details, see the documentation.

LinkRayCluster Enhancements

LinkRecreate upgrade strategy

Previously, updating certain fields in the RayCluster spec, such as the container image, requires users to either recreate the RayCluster or manually suspend and resume it after all Pods have been deleted. This is not convenient for users deploying with GitOps systems like ArgoCD.

KubeRay now supports a spec.upgradeStrategy.type field on RayCluster. Setting it to Recreate tells the controller to delete all Pods in the cluster whenever the spec changes, and then create them again from the updated spec. Applying the new spec is all that's needed to roll out the change.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-kuberay
spec:
  upgradeStrategy:
    type: Recreate
 … 

Note that Recreate deletes all Pods before creating new ones, so any Ray workload running on the cluster is lost. This strategy is intended for clusters that can be safely restarted.

LinkCustomizable Ingress options

KubeRay provides optional built-in ingress support that automatically creates a Kubernetes Ingress resource when spec.headGroupSpec.enableIngress is enabled. This feature was originally designed for simple NGINX-based setups and covers only basic exposure scenarios. 

The Ingress API was introduced earlier than the Gateway API and remains widely adopted, so we receive many requests to make it more configurable. Previously, users who needed to customize the generated Ingress typically had to create or modify the Ingress themselves. 

KubeRay v1.7 introduces spec.headGroupSpec.ingressOptions, allowing users to customize the generated Ingress to better fit their deployment requirements.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-ingress
spec:
  headGroupSpec:
    enableIngress: true
    ingressOptions:
      host: ray-dashboard.example.com
      path: /
      pathType: Prefix
      tls:
      - hosts:
        - ray-dashboard.example.com
        secretName: ray-dashboard-tls

LinkPriority field in the worker group API for Autoscaler V2

Ray Autoscaler V2 previously used the order of workerGroupSpecs to break ties when multiple worker groups could satisfy the same resource request. This made it difficult for users to explicitly control which worker group should be preferred.

Ray 2.56 introduces priority-aware worker group selection in Autoscaler V2. KubeRay v1.7 now exposes this capability through the priority field in workerGroupSpecs, allowing users to define their preferred worker groups.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-autoscaler-priority
spec:
  enableInTreeAutoscaling: true
  autoscalerOptions:
    version: v2
  workerGroupSpecs:
  - groupName: high-priority-worker-group
    priority: 10
    …
  - groupName: low-priority-worker-group
    priority: 0
    …

LinkRayMultiHostIndexing (beta)

The RayMultiHostIndexing feature was introduced as an alpha feature in KubeRay v1.5. It provides stable replica and host indexing for multi-host worker groups, which is useful for distributed accelerator workloads that require each worker to have a consistent host or replica identity.

In KubeRay v1.6, this feature is promoted to Beta following unit and end-to-end testing, as well as manual validation with multi-host Ray workloads using SkyRL and the JaxTrainer Ray Train API. This is useful for multi-host workloads (especially with multiple slices) and helps enable improvements like elasticity in Ray Train.

LinkGCS Fault Tolerance

LinkEmbedded RocksDB backend (alpha)

Starting from KubeRay v1.7 and Ray 2.57.0, an embedded RocksDB storage backend for Ray GCS is supported. Previously, enabling GCS fault tolerance required an external Redis, requiring users to deploy and operate a Redis instance. Now, users can use the embedded RocksDB storage backend, which is linked directly into the GCS process and persists state to a local path, with no external service required. KubeRay exposes this through gcsFaultToleranceOptions.backend: rocksdb. The operator automatically provisions a PersistentVolumeClaim (PVC), mounts it on the head pod, and sets the required environment variables. The PVC is garbage-collected together with the RayCluster by default; set deletionPolicy: Retain to keep the data after the cluster is deleted so that a new cluster can recover from it, or use claimName to bring your own PVC.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-embedded-gcs-ft
spec:
  gcsFaultToleranceOptions:
    backend: rocksdb
    storage:
      size: 1Gi
      # deletionPolicy: Retain   # keep the data after the cluster is deleted
      # claimName: my-gcs-pvc    # or bring your own PVC

This feature is alpha in both Ray and KubeRay, and is disabled by default. To use it, enable the GCSFaultToleranceEmbeddedStorage feature gate. See the sample YAML for a complete example.

LinkRedis cleanup finalizer timeout

When GCS fault tolerance is enabled with Redis, KubeRay adds a finalizer to the RayCluster and runs a cleanup job on deletion to remove leftover data from Redis. Previously, if the cleanup job failed to complete, the finalizer was never removed and the RayCluster could be stuck in the Terminating state indefinitely, which also blocked systems like Kueue from releasing the cluster's resource quota. Starting with KubeRay v1.7, the finalizer is force-removed after a timeout, which defaults to 5 minutes and can be configured per cluster with the ray.io/gcs-ft-deletion-timeout annotation. When this happens, KubeRay emits a ForceDeletedStuckCluster warning event that includes the external storage namespace, so you know which Redis data may need manual cleanup.

LinkOthers

LinkRay platform events on Kubernetes

When a Ray workload encounters a failure caused by underlying node infrastructure – for example, a GPU XID error – the Ray application may only report a generic CUDA or task failure. The actual root cause, however, may be recorded as a Kubernetes Event associated with the node. This makes it difficult for users to distinguish between application-level failures and underlying infrastructure issues when troubleshooting from the Ray Dashboard. 

Ray 2.56.0 introduced Platform Events, which surface Kubernetes lifecycle events for KubeRay custom resources in the Ray Dashboard. See Enable Ray platform events on Kubernetes for more details.

LinkVolcano PodGroup lifecycle on RayJob completion

Previously, when a RayJob reached a terminal state (Complete or Failed), KubeRay would unconditionally delete the associated Volcano PodGroup. The behavior is fixed from this release by updating the PodGroup's MinMember and MinResources based on the live RayCluster state and deletion strategy, instead of always deleting it. RayJobs using ClusterSelector are excluded, as gang scheduling is not supported in that mode.

LinkJoin the Ray Community

KubeRay v1.7 delivers enhanced extensibility and more robust building blocks for creating scalable ML platforms with Ray on Kubernetes. As we look ahead to v1.8, we are collaborating with the Ray community to shape the roadmap. Help us build the future of KubeRay by sharing your feedback on kuberay#2999.

You can also join the Ray Slack workspace’s #kuberay channel to ask your questions. In addition, follow the Ray/KubeRay community Google calendar to join the KubeRay bi-weekly community meeting to meet KubeRay contributors.

Follow the instructions below to get started, and check Getting Started with KubeRay for more details.

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
# Install both CRDs and KubeRay operator v1.7.0.
helm install kuberay-operator kuberay/kuberay-operator --version 1.7.0