Understanding Volcano Scheduler in Kubernetes

A complete guide to advanced batch and AI/ML workload scheduling in Kubernetes

1. Introduction

Kubernetes has revolutionized how modern infrastructure teams deploy and scale containerized applications. But as workloads evolved — from simple web APIs to complex distributed AI training, large-scale data pipelines, and scientific simulations — one component quietly became a bottleneck: the scheduler.

The default Kubernetes scheduler was never designed for batch computing. It works beautifully for stateless microservices, but when data engineers started running Spark jobs and ML teams started training transformer models on Kubernetes, the cracks became impossible to ignore.

That is where Volcano Scheduler comes in.

Volcano is a production-grade, cloud-native batch scheduling system built specifically for compute-intensive workloads. It brings the scheduling intelligence of traditional HPC systems directly into Kubernetes — without abandoning the cloud-native principles that make Kubernetes powerful in the first place.

2. The Scheduling Problem in Kubernetes

To appreciate what Volcano solves, you first need to understand what the default Kubernetes scheduler cannot do.

The default scheduler treats every pod as an independent unit. It looks at available resources on each node and places pods one at a time. For a stateless web server or a REST API, this is perfectly fine. But batch and AI workloads do not behave like microservices.

Consider a distributed PyTorch training job with 16 worker pods. If the default scheduler places only 12 of them because the cluster is partially busy, those 12 pods sit idle — holding onto GPUs and memory — while waiting for the remaining 4. The job makes zero progress. Resources are wasted. This is called a partial scheduling deadlock, and it is one of the most common failure modes in ML infrastructure on Kubernetes.

Beyond that, the default scheduler has no concept of job queues, so there is no way to say that the production training pipeline should get priority over an experimental research job. There is no built-in fairness mechanism to prevent one team from consuming the entire cluster. There is no preemption logic that understands job-level priority rather than just pod-level priority.

For teams running data-intensive workloads, these are not edge cases. They are everyday operational problems. Volcano was designed to solve all of them.

3. What is Volcano Scheduler?

Volcano is an open-source, Kubernetes-native batch scheduling system originally developed by Huawei Cloud and contributed to the Cloud Native Computing Foundation (CNCF), where it is currently an incubating project.

It extends Kubernetes through Custom Resource Definitions and a pluggable scheduler framework. It introduces first-class support for jobs, queues, and batch scheduling policies that the default Kubernetes scheduler simply does not have.

Volcano does not force you to rip out your existing setup. It can coexist with the default scheduler on the same cluster, or it can replace it entirely on clusters dedicated to batch and AI workloads. Teams at companies like Huawei, Baidu, Tencent, and various research institutions run Volcano in production at scale.

The simplest way to understand Volcano is this: it brings the scheduling intelligence of systems like SLURM and PBS into the Kubernetes ecosystem — making Kubernetes a legitimate compute platform for AI, big data, and HPC, not just a platform for web services.

4. Key Concepts Behind Volcano

Volcano introduces a small set of powerful abstractions that are worth understanding before diving into how it works.

Volcano Job (vcjob) is the core workload unit. Unlike a standard Kubernetes Job, a Volcano Job supports multiple task types within a single workload, gang scheduling policies, retry strategies, and lifecycle hooks. When you submit a Volcano Job, you define how many workers you need, what the minimum number to start is, and what happens if pods fail.

Queue is a logical resource pool. Each queue has a weight and a capacity. Jobs inside a queue compete for its resources. Queues can borrow idle resources from each other when underutilized, and those resources are returned when the owning queue needs them. This enables both isolation and efficiency across teams.

PodGroup is the atomic scheduling unit for gang scheduling. Every Volcano Job creates a corresponding PodGroup that defines how many pods must be co-scheduled together. The scheduler will not bind any pods in the group to nodes until it can satisfy the minimum availability requirement.

Actions are the scheduling operations the scheduler executes each cycle — such as allocate, preempt, reclaim, and backfill. They run in a configured order and define what the scheduler does.

Plugins define how scheduling decisions are made within each action — ranking jobs, enforcing fairness, checking gang constraints, and evaluating node fitness.

5. Volcano Architecture

Volcano is composed of three tightly integrated components that extend the standard Kubernetes control plane.

The Volcano Scheduler is the decision-making core. Every scheduling cycle, it opens a session by taking a consistent snapshot of the cluster state — pending jobs, available nodes, queue resource usage, and running workloads. It then executes configured actions in sequence, making allocation and eviction decisions, and finally commits those decisions to the Kubernetes API server before closing the session and beginning the next cycle.

The Volcano Controller Manager handles the lifecycle of Volcano’s custom resources. It watches for new Volcano Jobs and creates the corresponding PodGroups. It transitions jobs through their lifecycle phases — from Pending through Running to Completed or Failed — and handles retries, task dependencies, and cleanup when jobs finish.

The Volcano Admission Webhook sits at the entry point of the system. It validates all incoming Volcano resources and rejects malformed configurations before they can cause problems. It also mutates pods belonging to a Volcano Job by injecting the necessary PodGroup annotations, ensuring the scheduler can associate individual pods with their parent job.

All three components work entirely through the standard Kubernetes API, which means Volcano requires no changes to your existing cluster infrastructure.

6. How Volcano Scheduling Works

Every Volcano scheduling cycle follows a structured pipeline that is both transparent and extensible.

The cycle begins when the scheduler opens a session and captures a consistent view of the entire cluster. This snapshot includes every node’s available resources, every queue’s current usage and deserved share, and every pending and running job. All decisions made in this cycle are based entirely on this snapshot, ensuring consistency.

Next, the scheduler executes its action pipeline in order.

The Enqueue action looks at all pending jobs and moves those that have a reasonable chance of being scheduled — based on queue capacity and cluster availability — into an active queue for consideration in this cycle.

The Allocate action is where the primary scheduling work happens. It iterates over jobs in priority order, evaluates each job’s resource requirements against the current cluster state, checks gang constraints via the PodGroup, and assigns pods to nodes for binding.

The Preempt action handles situations where a high-priority job cannot be scheduled because lower-priority jobs are occupying the needed resources. Volcano can evict those lower-priority pods to free up space, respecting queue boundaries and fairness policies in the process.

The Reclaim action enforces queue fairness. If a queue has borrowed resources from another queue and the owning queue now needs those resources back, the reclaim action identifies pods from the borrowing queue that can be evicted to return the capacity.

The Backfill action runs last and fills any remaining idle capacity in the cluster with jobs that were not scheduled during the main allocation phase. This maximizes utilization without sacrificing fairness for larger, higher-priority jobs.

Finally, the scheduler closes the session by flushing all bind decisions to the Kubernetes API server and updating job and queue status objects. The cycle completes and begins again immediately.

7. Key Features of Volcano

Gang Scheduling is the most important feature Volcano brings to Kubernetes. It guarantees that all pods in a job either start together or wait together. The scheduler will not bind a single pod to a node until the cluster can satisfy the entire PodGroup’s minimum availability requirement. This completely eliminates partial scheduling deadlocks and the wasted GPU hours that come with them.

Fair-share Scheduling with Dominant Resource Fairness (DRF) ensures that no single team or queue can monopolize cluster resources. DRF is a multi-dimensional fairness algorithm — it considers CPU, memory, GPU, and any other resource type simultaneously when calculating each queue’s fair share. Queues with higher weights receive proportionally more resources, but no queue starves another.

Queue-based Priority Management gives platform teams fine-grained control over how resources are distributed across workloads. You can define production queues with high weight and strict capacity guarantees alongside experimental queues that only use idle capacity. Jobs within each queue are further ordered by their configured priority.

Advanced Preemption and Reclamation goes far beyond what the default scheduler offers. Volcano understands job-level priority, not just pod-level priority. It can preempt an entire low-priority job to make room for a high-priority one, and it can reclaim borrowed capacity from queues that have exceeded their deserved share — all automatically and without manual intervention.

Topology-aware GPU Scheduling is critical for serious deep learning workloads. Volcano can schedule pods with awareness of NUMA topology and GPU affinity, ensuring that workers in a distributed training job are co-located on nodes where GPUs can communicate via NVLink rather than slower interconnects. The performance impact on large model training is significant.

Multi-task Job Support allows a single Volcano Job to define multiple task types with different resource requirements and explicit dependencies between them. A Spark job can define its driver and workers as separate tasks within the same job object, with the workers only starting once the driver is healthy. This eliminates the need for external orchestration logic.

Broad Framework Support makes Volcano practical without requiring custom integration work. It has built-in support for TensorFlow, PyTorch, MXNet, Apache Spark, Apache Flink, Argo Workflows, Kubeflow, and Ray. If you are already using any of these frameworks, plugging them into Volcano is straightforward.

8. Volcano vs Default Kubernetes Scheduler

The default Kubernetes scheduler and Volcano are not really competing for the same use case — they are designed for fundamentally different workload profiles.

The default scheduler is optimized for latency-sensitive, stateless workloads. It places pods quickly, one at a time, and has no concept of jobs, queues, or batch semantics. For web APIs, microservices, and event-driven applications, it is the right tool.

Volcano is optimized for throughput-sensitive, stateful batch workloads. It understands that a 16-GPU training job is a single unit of work, not 16 independent pods. It understands that a data engineering team and an ML research team should share the cluster fairly. It understands that a production pipeline should preempt an experimental run when resources are tight.

The default scheduler has no gang scheduling, no queue management, no fair-share allocation, no backfill, and no GPU topology awareness. Volcano was built specifically to provide all of these capabilities in a Kubernetes-native way.

For clusters running mixed workloads — microservices and batch jobs side by side — both schedulers can coexist. Batch workloads get routed to Volcano while standard deployments continue using the default scheduler.

9. Use Cases of Volcano Scheduler

Distributed AI and ML Training is the most common production use case. Teams training large models with PyTorch or TensorFlow use Volcano to guarantee that all GPU workers start simultaneously, to take advantage of topology-aware placement for faster inter-GPU communication, and to enforce fair resource allocation across multiple training runs competing for the same GPU pool.

Big Data Processing on Kubernetes is another major use case. Organizations migrating Spark and Flink workloads to Kubernetes use Volcano to manage driver-worker dependencies, enforce job-level resource guarantees, and prioritize production ETL pipelines over ad-hoc analytical queries in shared clusters.

High-Performance Computing on Kubernetes is a growing area. Research institutions and enterprises running MPI-based simulations, genomics pipelines, or computational fluid dynamics workloads use Volcano to get SLURM-like scheduling semantics on a modern Kubernetes platform — without maintaining a separate HPC cluster.

Multi-tenant AI Platforms are perhaps the most strategically important use case. When a platform engineering team builds an internal AI infrastructure shared across dozens of teams, Volcano’s queue system becomes the mechanism for enforcing resource governance — giving each team a guaranteed baseline, allowing burst borrowing from idle queues, and ensuring that one runaway experiment cannot starve the rest of the organization.

10. When Should You Use Volcano?

The right answer depends entirely on your workload profile.

You should seriously evaluate Volcano if your cluster runs distributed training jobs that require all workers to start simultaneously, if multiple teams share compute resources and fairness is a recurring operational concern, if you run Spark, Flink, or any other batch processing framework on Kubernetes, if GPU utilization and topology-aware placement matter to your workload performance, or if you are migrating workloads from a traditional HPC scheduler like SLURM or PBS.

You likely do not need Volcano if your cluster is dedicated to stateless microservices and web APIs, if your jobs are simple single-pod tasks with no coordination requirements, or if your team does not yet have the operational capacity to manage additional CRDs and scheduling configuration.

The honest answer is that most organizations reach a point in their Kubernetes journey where the default scheduler becomes a ceiling. Volcano is what you reach for when you hit that ceiling — and the earlier you plan for it, the less painful the transition will be.

11. Conclusion

Volcano Scheduler is one of the most important but least discussed components in the modern Kubernetes ecosystem. It quietly solves the problems that make Kubernetes frustrating for data scientists, ML engineers, and HPC teams — while staying invisible to teams running standard workloads who never needed it in the first place.

By delivering gang scheduling, fair-share resource management, advanced preemption, GPU topology awareness, and deep framework integrations, Volcano completes Kubernetes as a compute platform. It is no longer just a platform for running web services — with Volcano, it becomes a legitimate home for the most demanding computational workloads in the industry.

If you are building AI infrastructure, running big data pipelines, or managing a shared compute cluster for multiple teams, Volcano deserves a serious place in your architecture conversations.

The future of Kubernetes scheduling is batch-intelligent, workload-aware, and multi-tenant — and Volcano is already there.

---

Understanding Volcano Scheduler in Kubernetes was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.