The notebook starts immediately with no cluster provisioning

Serverless compute on Databricks eliminates cluster management by running your workloads on Databricks-managed infrastructure that starts in seconds and scales automatically. You pay only for the compute time your code actually uses, with no idle cluster costs. Use serverless when you want instant startup, zero infrastructure overhead, and predictable per-query economics.

  • Understand how Databricks serverless compute provisions and manages resources behind the scenes
  • Learn which workloads benefit most from serverless and which still need classic compute
  • Configure serverless notebooks, jobs, and SQL warehouses

Who this is for: Data engineers, analysts, and platform teams evaluating serverless to reduce operational burden and cost.

Part of the Databricks Compute section of the Databricks tutorial series.

Architecture / Concept Overview: The notebook starts immediately with no cluster provisioning

When you run a serverless workload, Databricks provisions compute from a shared pool of pre-warmed instances in a Databricks-managed cloud account. The control plane handles all infrastructure — instance selection, autoscaling, patching, and termination. Your code runs in an isolated container with access only to the data permitted by Unity Catalog, and compute is released back to the pool within seconds of completion.

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED Request[User Query]:::source --> CP[Control Plane]:::governance CP --> Pool[Pre-Warmed Pool]:::ingestion Pool --> Exec[Serverless Executor]:::serving Exec --> Data[(Your Cloud Storage)]:::storage Exec -->|results| Request

*Serverless compute draws from pre-warmed pools, executes your query, and releases resources — all in seconds.*

The key difference between serverless and classic is who manages the infrastructure and where it runs.

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED subgraph Classic Mode YourVPC[Your Cloud VPC]:::processing --> YourVMs[Your VMs]:::processing end subgraph Serverless Mode DBXAcct[Databricks Account]:::serving --> ManagedVMs[Managed VMs]:::serving end UC[Unity Catalog]:::governance -.->|governs both| YourVMs UC -.->|governs both| ManagedVMs YourVMs --> Store[(Cloud Storage)]:::storage ManagedVMs --> Store

*Classic runs in your VPC on your VMs; serverless runs in a Databricks-managed account. Unity Catalog governs both.*

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED NB[Serverless Notebook]:::serving --> Jobs[Serverless Job]:::serving Jobs --> Pipelines[Serverless Pipeline]:::serving SQLW[Serverless SQL Warehouse]:::serving --> BI[BI Dashboards]:::source

*Serverless is available across notebooks, jobs, pipelines, and SQL warehouses.*

Key Terms

Serverless Compute
Databricks-managed compute that starts instantly, scales automatically, and bills only for active usage.
Pre-Warmed Pool
A Databricks-managed fleet of idle instances ready to serve workloads in seconds.
Classic Compute
Compute that runs on VMs in your own cloud account, requiring manual or policy-driven configuration.
Auto-Stop
The automatic suspension of serverless resources after a configurable idle period.
Serverless SQL Warehouse
A SQL warehouse running entirely on Databricks-managed infrastructure.

Prerequisites and Setup

  • A Databricks workspace with serverless enabled (account-level setting)
  • Unity Catalog enabled (required for serverless data access)
  • Workspace admin access to enable serverless features
  • No special networking configuration needed (serverless manages connectivity)

Step-by-Step Implementation

    Configuration Reference

    The notebook starts immediately with no cluster provisioning configuration options
    SettingSQL WarehouseNotebookJob
    Enable serverlessenable_serverless_compute: trueConnect → ServerlessEnvironment spec
    Auto-stop / terminationauto_stop_minsAutomaticAutomatic
    Scalingmax_num_clustersAutomaticAutomatic
    PhotonAlways enabledEnabledEnabled
    Unity Catalog requiredYesYesYes
    Custom librariesLimitedpip install in notebookRequirements file

    Monitoring, Cost, and Security Considerations

    Monitoring

    Serverless usage appears in system.billing.usage with SKU names containing SERVERLESS. Use query history in the SQL warehouse UI to see per-query duration and DBU cost.

    Cost Optimisation

    - Serverless eliminates idle cluster costs but has a higher per-DBU rate — compare total cost of ownership for steady-state workloads.

    - Set aggressive auto-stop times (5-10 minutes) on SQL warehouses.

    - Use serverless for bursty workloads and classic for sustained, predictable loads where the lower DBU rate offsets idle time.

    Security and Governance

    - Serverless enforces Unity Catalog access controls; there is no credential passthrough or instance profile option.

    - Compute runs in a Databricks-managed account with network isolation per workspace.

    - Data in transit is encrypted via TLS; at rest, you can use customer-managed keys (CMK).

    - Lakeguard provides additional isolation between users on shared serverless resources.

    Common Pitfalls and Recommended Patterns

    • Assuming serverless is always cheaper: for steady 24/7 workloads, classic with reserved instances may cost less.
    • Forgetting Unity Catalog is required: serverless cannot use legacy Hive metastore or instance profiles.
    • Installing heavy custom libraries in every notebook cell: package dependencies into a cluster-scoped init script or requirements file.
    • Not setting auto-stop on SQL warehouses: even serverless warehouses cost DBUs while they idle waiting for queries.
    • Using serverless for GPU workloads: GPU instances are not available in serverless; use classic clusters.
    • Ignoring query history for cost debugging: per-query DBU breakdowns help identify expensive queries.

    Frequently Asked Questions

    Is my data safe on serverless compute?

    Yes. Serverless compute accesses your data through Unity Catalog with the same access controls as classic. Compute runs in an isolated environment in a Databricks-managed account, and data never persists on serverless instances beyond the query lifecycle.

    Can I install custom Python packages on serverless?

    Yes. You can use %pip install in notebooks or specify a requirements file for jobs. However, packages install at session start, which adds a few seconds. Pre-built environment specs reduce this overhead.

    How fast does serverless start compared to classic?

    Serverless typically starts in under 10 seconds, compared to 3-7 minutes for classic cluster provisioning. The difference comes from drawing on pre-warmed pools rather than launching fresh cloud VMs.

    Which clouds support serverless?

    Serverless is available on AWS, Azure, and GCP, though feature availability may differ by cloud. Check the Databricks documentation for the latest regional availability.