Resize if utilisation is consistently high

Instance pools maintain a set of pre-provisioned, idle cloud VMs that clusters can draw from instantly instead of waiting for the cloud provider to launch new instances. By keeping a configurable number of warm instances ready, pools reduce cluster start-up time from minutes to seconds while capping idle costs with auto-termination. Use pools when fast cluster starts are critical and serverless compute is not an option.

  • Understand how instance pools work and why they speed up cluster starts
  • Create and configure pools with minimum idle instances and capacity limits
  • Attach clusters and job definitions to pools for faster provisioning

Who this is for: Platform administrators and data engineers who need faster cluster start times on classic compute.

Part of the Databricks Compute section of the Databricks tutorial series.

Architecture / Concept Overview: Resize if utilisation is consistently high

An instance pool is a managed set of cloud VM instances that sit idle and ready. When a cluster requests nodes, Databricks draws from the pool instead of requesting new VMs from the cloud provider — eliminating the 3-7 minute provisioning delay. When a cluster releases nodes, they return to the pool for reuse. Idle instances beyond the minimum are terminated after a configurable period.

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED Cloud[Cloud Provider]:::source --> Pool[Instance Pool]:::ingestion Pool --> C1[Cluster A]:::processing Pool --> C2[Cluster B]:::processing C1 -->|release| Pool C2 -->|release| Pool Pool --> Idle[Idle Instances]:::source

*The pool acquires VMs from the cloud, lends them to clusters on demand, and reclaims them when released.*

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED Min[Min Idle = 2]:::governance --> I1[Instance 1]:::ingestion Min --> I2[Instance 2]:::ingestion Demand[On Demand]:::source --> I3[Instance 3]:::processing Demand --> I4[Instance 4]:::processing Max[Max Capacity = 10]:::governance -.->|cap| Demand

*Pools keep a minimum number of idle instances warm and scale up to a maximum capacity on demand.*

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED NoPool[Without Pool: 3-7 min]:::source --> Start1[Cloud VM Launch]:::processing WithPool[With Pool: seconds]:::serving --> Start2[Draw from Pool]:::ingestion Start1 --> Running[Cluster Running]:::serving Start2 --> Running

*Pool-backed clusters start in seconds versus minutes for non-pool clusters.*

Key Terms

Instance Pool
A managed collection of pre-provisioned cloud VMs that clusters draw from for fast startup.
Min Idle Instances
The minimum number of warm, unattached instances the pool keeps ready at all times.
Max Capacity
The maximum total number of instances (idle plus in-use) the pool can hold.
Idle Instance Auto-Termination
The time after which idle instances beyond the minimum are released back to the cloud.
Preloaded Spark Version
A Databricks Runtime version pre-installed on pool instances to skip installation at cluster start.

Prerequisites and Setup

  • Workspace admin permissions to create instance pools
  • Knowledge of which instance types your teams use most frequently
  • Understanding of peak and off-peak usage patterns to set min/max values
  • Classic compute (instance pools do not apply to serverless)

Step-by-Step Implementation

    Configuration Reference

    Resize if utilisation is consistently high configuration options
    ParameterDescriptionRecommended Value
    instance_pool_nameHuman-readable nameDescriptive team/purpose name
    node_type_idCloud VM instance typeMatch your most common cluster type
    min_idle_instancesAlways-warm instances1-3 (balance cost vs speed)
    max_capacityHard cap on total instances2-3x peak concurrent cluster nodes
    idle_instance_autotermination_minutesTerminate excess idle instances15-30 minutes
    preloaded_spark_versionsPre-install runtimesLatest LTS
    enable_elastic_diskAuto-expand local storagetrue
    disk_specCustom disk type and sizeCloud-specific SSD config

    Monitoring, Cost, and Security Considerations

    Monitoring

    Track pool utilisation via the API (stats.idle_count, stats.used_count) or the pool UI page. Set up alerts when idle count drops to zero (pool exhausted) or stays at max (pool over-provisioned).

    Cost Optimisation

    - Min idle instances incur cloud VM costs even when no cluster uses them — keep this number small.

    - Pools eliminate per-start cloud provisioning fees and reduce time-to-first-result for developers.

    - Compare pool idle costs vs serverless startup costs to decide which approach is cheaper for your usage pattern.

    - Use Spot/Preemptible instances in pools for additional savings on fault-tolerant workloads.

    Security and Governance

    - Pools use the same VPC/VNet as your workspace, so existing network controls apply.

    - Instances are wiped between cluster uses; no data persists on pool instances after release.

    - Restrict pool creation and usage via workspace permissions to prevent uncontrolled instance proliferation.

    Common Pitfalls and Recommended Patterns

    • Setting min idle too high: each warm instance costs cloud VM charges; start with 1-2 and increase based on demand.
    • Forgetting max capacity: without a cap, a burst of cluster starts can exhaust your cloud quota.
    • Not preloading a Spark version: skipping this adds runtime installation time to each start.
    • Creating one pool per user: pools are most cost-effective when shared across a team.
    • Mixing instance types: one pool supports one node_type_id; create separate pools for different instance families.
    • Ignoring pool stats: monitor idle and used counts to right-size over time.

    Frequently Asked Questions

    Do I pay DBUs for idle pool instances?

    No. You pay only cloud VM costs for idle instances. DBU charges begin when a cluster attaches to the instance and starts running workloads.

    Can multiple clusters share the same pool?

    Yes. Pools are designed to be shared. Multiple clusters draw from and return instances to the same pool, which improves utilisation.

    Does the pool replace autoscaling?

    No. Autoscaling controls how many workers a cluster uses. The pool controls how fast those workers are provisioned. Use both together for fast-starting, right-sized clusters.

    Can I use pools with serverless compute?

    No. Pools apply only to classic compute. Serverless already starts in seconds using Databricks-managed pre-warmed infrastructure.