Async inference in practice: a video-indexing service on Ray Serve

In an earlier post, we introduced asynchronous inference in Ray Serve: a way to run long-running model calls off the request path, backed by a message queue, with automatic retries and queue-depth autoscaling. This post is a practical follow-up. We build a real service on top of that feature, a video-indexing pipeline, run it under a heavy load, and compare it against a common managed alternative.

LinkA quick recap: what async inference is for

In serving ML models, some applications make a single model call that runs for several seconds to minutes, e.g. - transcribing an hour of audio, indexing a video, or generating an image or a video. On a normal synchronous HTTP endpoint the request stays open for the whole computation. At millisecond latencies that coupling is invisible, but for calls this long it creates two problems. 

First, a burst of traffic piles onto a fleet that takes seconds or minutes to finish each request, so the workers fall behind and clients, or any intermediary hop, start timing out. Also, a connection held open for minutes has much more exposure to a dropped packet or a failed hop. As a result, the service and overall system gets less reliable at exactly the moments you can't predict.

Async inference solves these problems by decoupling the request scheduling from the actual computation. The client submits a job and gets an id back right away, the work goes onto a message queue,  a pool of workers, autoscaled to the queue depth, processes it in the background, and the client polls for the result.

We covered the API and the design in the earlier post, so here we focus on building a service on top of it, and benchmark it under real-world traffic. The pieces Ray Serve provides, in brief:

• @task_consumer / @task_handler - these decorators turn a deployment method into a background worker that pulls from a queue.

• TaskProcessorConfig - configuration that points your code at a message broker you run (e.g. Redis or RabbitMQ are supported) and adds an at-least-once delivery mechanism, retries, and dead-letter routing.

• enqueue_task_sync(...) - function that submits work from the producer side and returns a task id which client can poll for the result.

• AsyncInferenceAutoscalingPolicy - it is the autoscaling policy that scales the worker pool based on queue depth.

LinkThe application: a video-indexing service

Video indexing is a natural fit for async inference. As each request demands a whole video to download, decode, and embed,it can take seconds to minutes of work, and the traffic tends to arrive in bursts such as a batch upload or a nightly backfill. This video indexing service will take an S3 URI and return a task id immediately, and in the background it downloads the video, splits it into frames with ffmpeg, embeds the frames with SigLIP on a GPU, and writes the vectors back to S3, a pipeline that can take tens of minutes per request.

The application, comprises of three deployments, each scaling on its own signal:

• IndexingIngress (CPU) -  this deployment accepts POST /index, enqueues the task sent by the user, and returns the task id.

• VideoIndexConsumer (CPU) -  the @task_consumer, it consumes tasks from the queue, and downloads and chunks the video with ffmpeg, then calls the encoder. This is the deployment that autoscales on queue depth.

• VideoEncoder (GPU) - a standard deployment holding the SigLIP neural network. The video index consumer deployment calls it through a handle, so frames move between the two over Ray's RPC rather than the network, and it scales on its own GPU load.

The diagram shows the architecture of the service we deployed and tested; purple boxes denotes the deployments, and yellow ones denote the external systems the service talks to.

The producer (the deployment which is accepting the user’s request) is a very simple ingress deployment. It enqueues the task and returns an id, so the caller never waits on the work:

@serve.ingress(fastapi_app)
class IndexingIngress:
    def __init__(self, consumer):
        self.adapter = instantiate_adapter_from_config(PROCESSOR_CONFIG)

    @fastapi_app.post("/index")
    async def index(self, req: IndexRequest):
        result = self.adapter.enqueue_task_sync(
            task_name=TASK_INDEX_VIDEO,
            kwargs={"video_uri": req.video_uri, "video_id": req.video_id},
        )
        return {"task_id": result.id, "status": result.status}

The video index consumer is a simple synchronous Python code. It is idempotent on the video id, chunks the video on CPU, calls the GPU encoder through a handle, and stores the result:

@serve.deployment(ray_actor_options={"num_cpus": FFMPEG_THREADS}, ...)
@task_consumer(task_processor_config=PROCESSOR_CONFIG)
class VideoIndexConsumer:
    def __init__(self, encoder):
        self.encoder = encoder               # GPU DeploymentHandle

    @task_handler(name=TASK_INDEX_VIDEO)
    def index_video(self, video_uri, video_id=None):
        if is_done(video_id):                  # idempotent on redelivery
            return {"status": "skipped_already_indexed"}
        chunks = chunk_video(download(video_uri))         # CPU: ffmpeg
        refs = [self.encoder.remote(c.frames) for c in chunks]
        vectors = [r.result()["frame_embeddings"] for r in refs]
        write_video_embeddings(video_id, vectors)         # store to S3
        mark_done(video_id)
        return {"status": "indexed", "num_chunks": len(chunks)}

Note - For the benchmarks below, we fused the VideoIndexConsumer (CPU) and VideoEncoder (GPU) deployments into a single GPU worker, while keeping the same lightweight ingress in front. We did this to make the comparison as fair as possible and to better match the deployment model supported by the alternative, Amazon SageMaker.

SageMaker does not support deploying these two components as separate CPU and GPU services behind a single endpoint. The only alternative would be to expose them as completely separate endpoints; that route would put SageMaker at a further disadvantage: the intermediate tensors would have to travel between the two endpoints over the network, passed by value, whereas Ray Serve passes them by reference through its shared object store. Therefore, all benchmark results presented below were collected using a two-deployment architecture (the ingress and one fused worker), rather than the three-deployment architecture described above.

LinkBehavior under load

Our goal was to stress-test the service and evaluate both its reliability and performance under sustained overload. Ideally, the service should scale with incoming traffic, process requests as quickly as possible, and do so without dropping user requests. 

We evaluate reliability by measuring the number of failed requests during the load test, and performance by observing how the system responds to increasing load — particularly its ability to scale up and down in response to traffic while maintaining throughput and stability.

To exercise both aspects, we ran a 20-minute flood test at a sustained 50 RPS with periodic spikes to 100 RPS. This workload is approximately 5–10× higher than what a 4-GPU fleet can process, resulting in roughly 67,000 video requests over the duration of the test. Since the incoming request rate significantly exceeds the fleet's processing capacity, a request backlog is inevitable, making this a good test of the system's scaling behavior and resilience under sustained pressure.

As a result, we observed

1. Replicas scaled with demand - As the backlog grew, the application automatically increased the number of GPU replicas from 1 to 4, which was the configured maximum. Once the backlog was cleared, it scaled back down to 1. There was no fixed fleet and no manual tuning per burst.

2. Zero failed requests - All ~67,000 video requests were successfully accepted, queued, and eventually processed. The dead-letter queue remained empty throughout the test, indicating that no requests were dropped or failed during the sustained overload.

LinkComparing against a managed alternative: Amazon SageMaker

To put these numbers in context, we ran the same workload on Amazon SageMaker Async Inference, a common managed solution for this type of workload. Like our setup, SageMaker queues incoming requests, scales based on the backlog, and reads from and writes to S3. Both setups used the same hardware (4× NVIDIA T4 GPUs), the same 1-frame SigLIP workload, and the same 20-minute flood test, with both starting from a single cold instance. Since a SageMaker async endpoint runs a single model container, we used the same single, fused deployment on the Ray Serve side, as described above.

This means the deployment architecture, hardware, and workload were the same on both sides, making the orchestration engine the primary difference between the two systems.

* We autoscaled the SageMaker endpoint with two Application Auto Scaling step policies, each tied to a CloudWatch alarm. A backlog-fast alarm fires when ApproximateBacklogSizePerInstance hits 5 or more (on a single 60-second datapoint), triggering the fast-out policy to jump straight to the 4-instance cap, and backlog-empty alarm fires when the total ApproximateBacklogSize drops below 1, triggering the fast-in policy to drop back to a single instance.

1 - Offered RPS Graph w.r.t. time.

1 - Number of GPU replicas/instances w.r.t. time. 2 - Showing backlog queue length in Ray Serve vs Amazon SageMaker w.r.t time

Below are the few observations from the above runs:

• Scale-up is close. Both reach a full 4-GPU fleet in the same ballpark, and most of that time went to node provisioning, which both pay. The difference is in the decision to scale: Ray Serve polls the queue depth directly and reacts in seconds, while SageMaker's decision is bounded by the resolution of its CloudWatch metric.

• Scale-down differs more. Once the queue was empty, Ray Serve scaled back down within a few seconds. The fastest scale-down we saw with SageMaker was about 104 seconds.

• Ray Serve was easier to set up. The Ray Serve side is one autoscaling policy block. To get similar behavior in SageMaker, we had to configure multiple autoscaling policies and several CloudWatch alarms.

• Neither dropped anything. Both processed the full flood with an empty dead-letter queue.

A note on the SageMaker setup: autoscaling is configured to be as responsive as the platform allows. It uses a single 60-second CloudWatch alarm on the ApproximateBacklogSize and ApproximateBacklogSizePerInstance metric to trigger both scale-out and scale-in actions. Faster scaling isn't possible because these metrics are standard-resolution metric, so CloudWatch alarms cannot evaluate it more frequently than once every 60 seconds.

LinkEnd-to-end latency

Scaling speed is only part of the story. Another is - how long a user has to wait for their request to finish end to end. To measure this, we tracked the time from when a video was submitted until its embeddings were written to S3 for every request in the flood test, and below are the results:.

Each request had a base processing time of about 1–2 seconds, which included downloading the video, running FFmpeg, generating embeddings, and writing the results to S3. This part was nearly the same on both platforms. The difference came from waiting in the queue. Since there were only a limited number of GPUs available, requests had to wait for a free GPU before they could be processed. The longer a request waits in the queue, the higher its overall latency.

Ray Serve responded to the growing backlog more quickly and started new replicas sooner than SageMaker. As a result, requests spent less time waiting in the queue, so users received their results earlier.

LinkWhen async inference is the right fit

Async inference is useful for workloads that take longer to finish and where keeping an HTTP connection open is not practical. It is also a good fit for workloads with sudden traffic spikes, since requests can wait in a queue until resources are available. With Ray Serve, you get request queuing, reliable request processing, and automatic scaling based on queue size with just a small amount of configuration. Since it is built into Ray Serve, the same application can run on any cloud, on-premises, or even on a laptop without code changes.

The feature is available in Ray Serve today;  You can find more details in the Ray Serve documentation.

LinkReferences

• Ray Serve: autoscaling, async inference, and custom routing - the post that introduced the feature this one builds on.

• Ray Serve: Asynchronous Inference (documentation)

• Amazon SageMaker Asynchronous Inference (overview)

• Autoscale an asynchronous endpoint (Amazon SageMaker)

• SageMaker inference launches faster auto scaling for generative AI models

SageMaker supported-features matrix - async inference is single-container, which is why the comparison used a single collapsed deployment.