Standard vs Dedicated Compute: When to Use Each

Databricks offers two access modes for clusters — Standard (formerly Shared) and Dedicated (formerly Single-User). Standard clusters allow multiple users to share compute with Unity Catalog-enforced isolation, while Dedicated clusters give one user exclusive access with full language support and no restrictions. Choose Standard for cost-efficient team use and Dedicated when you need unrestricted library access, custom Spark extensions, or GPU workloads.

  • Understand the differences between Standard and Dedicated access modes
  • Learn which workloads require Dedicated and which benefit from Standard
  • Configure access mode settings on clusters and policies

Who this is for: Platform administrators and data engineers deciding how to provision clusters for teams with varying isolation and access requirements.

Part of the Databricks Compute section of the Databricks tutorial series.

Architecture / Concept Overview: Standard vs Dedicated Compute: When to Use Each

Access mode controls how a cluster handles multi-tenancy and security boundaries. Standard mode uses process-level isolation to let multiple users share a cluster safely, with Unity Catalog enforcing per-user data access. Dedicated mode assigns the entire cluster to one user with no isolation overhead, enabling unrestricted access to all four languages, arbitrary JVM libraries, and FUSE mounts.

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED subgraph Standard Cluster U1[User A]:::source --> Iso1[Isolated Process A]:::processing U2[User B]:::source --> Iso2[Isolated Process B]:::processing Iso1 --> SharedJVM[Shared JVM]:::processing Iso2 --> SharedJVM end UC[Unity Catalog]:::governance -.->|per-user ACLs| SharedJVM SharedJVM --> Store[(Delta Lake)]:::storage

*Standard clusters isolate users at the process level while sharing the underlying JVM, governed by Unity Catalog.*

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED subgraph Dedicated Cluster User[Single User]:::source --> Full[Full JVM Access]:::processing Full --> GPU[GPU Support]:::serving Full --> FUSE[FUSE Mounts]:::serving Full --> CustomJAR[Custom JARs]:::serving end Full --> Store[(Delta Lake)]:::storage

*Dedicated clusters give one user unrestricted access to the full cluster, including GPUs, FUSE mounts, and arbitrary JARs.*

%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED Decision{Workload Type}:::source Decision -->|Team notebooks, SQL| Standard[Standard Cluster]:::serving Decision -->|GPU, custom JARs, FUSE| Dedicated[Dedicated Cluster]:::processing Decision -->|Automated ETL| JobCluster[Job Cluster]:::processing

*The workload type drives the access mode decision: Standard for shared work, Dedicated for unrestricted access.*

Key Terms

Standard Access Mode
A cluster mode that allows multiple users with Unity Catalog-enforced isolation and some language/library restrictions.
Dedicated Access Mode
A cluster mode that assigns the entire cluster to one user with no restrictions on languages, libraries, or system access.
Process Isolation
The mechanism Standard clusters use to separate user sessions so one user cannot access another's data or variables.
Lakeguard
The security technology that enforces user isolation on Standard clusters.

Prerequisites and Setup

  • A Databricks workspace with Unity Catalog enabled
  • Understanding of your team's workload requirements (languages, libraries, GPU needs)
  • Cluster-create permission or an appropriate cluster policy
  • Knowledge of which users need unrestricted access

Step-by-Step Implementation

    Configuration Reference

    Standard vs Dedicated Compute: When to Use Each configuration options
    CapabilityStandard (Shared)Dedicated (Single-User)
    Multi-user accessYesNo (one user)
    Unity Catalog enforcementYes, per-userYes, single identity
    PythonYesYes
    SQLYesYes
    ScalaLimited (no arbitrary JARs)Full
    RYesYes
    Custom JVM librariesNoYes
    FUSE mountsNoYes
    GPU supportNoYes
    Credential passthroughNo (uses UC)Yes
    Lakeguard isolationYesNot applicable

    Monitoring, Cost, and Security Considerations

    Monitoring

    Track cluster creation events filtered by data_security_mode to see how many Standard vs Dedicated clusters are running. Monitor per-user DBU consumption to identify whether Dedicated clusters are being fully utilised or sitting idle.

    Cost Optimisation

    - Standard clusters let multiple users share infrastructure, reducing total cluster count and cost.

    - Dedicated clusters charge the same DBU rate but serve only one user, so utilisation matters more.

    - Set shorter auto-termination on Dedicated clusters since only one person uses them.

    - Use cluster policies to default to Standard and require justification for Dedicated.

    Security and Governance

    - Standard mode enforces Unity Catalog per-user access controls, preventing users from seeing each other's data.

    - Dedicated mode runs as a single identity, so the user has full access to everything their credentials permit.

    - On Standard clusters, Lakeguard prevents users from accessing the filesystem, environment variables, or other users' sessions.

    - Use Dedicated mode only when the workload genuinely requires it (custom JARs, GPU, FUSE).

    Common Pitfalls and Recommended Patterns

    • Defaulting to Dedicated for all users: this is expensive and unnecessary when most users only need Python and SQL.
    • Running GPU workloads on Standard clusters: GPU support requires Dedicated mode.
    • Assuming Standard limits all Spark features: Standard supports Python, SQL, R, and most PySpark operations.
    • Forgetting to set single_user_name on Dedicated clusters: without it, anyone with attach permission can use the cluster.
    • Over-provisioning Dedicated clusters: since only one user connects, right-size aggressively.
    • Not using cluster policies to enforce mode: without policies, users default to the most permissive option.

    Frequently Asked Questions

    Can I change the access mode on a running cluster?

    No. Access mode is set at cluster creation time and cannot be changed. You must create a new cluster with the desired mode.

    Does Standard mode affect performance?

    There is a small overhead from process isolation, but for most workloads the difference is negligible. Photon and adaptive query execution work the same in both modes.

    Why can I not install a custom JAR on a Standard cluster?

    Custom JVM code could bypass process isolation and access other users' data. Databricks restricts this to maintain security boundaries. Use a Dedicated cluster for custom JAR workloads.

    Can job clusters use Standard mode?

    Job clusters typically use Dedicated mode since each job run has a single identity. For multi-task jobs with different user contexts, each task gets its own cluster.