Workspace Deployment: Serverless and Traditional Options
Who this is for:
Architecture / Concept Overview: Workspace Deployment: Serverless and Traditional Options
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Traditional[Traditional Deployment] --> CustomerVPC[Customer VPC]
CustomerVPC --> ClassicClusters[Classic Clusters]
CustomerVPC --> JobClusters[Job Clusters]
Serverless[Serverless Deployment] --> ManagedInfra[Databricks-Managed Infra]
ManagedInfra --> SQLWarehouse[SQL Warehouses]
ManagedInfra --> ServerlessNotebooks[Serverless Notebooks]
ManagedInfra --> ServerlessPipelines[Serverless Pipelines]
Traditional:::processing
CustomerVPC:::storage
ClassicClusters:::ingestion
JobClusters:::ingestion
Serverless:::serving
ManagedInfra:::governance
SQLWarehouse:::source
ServerlessNotebooks:::source
ServerlessPipelines:::source
*Deployment models: traditional deploys compute into customer infrastructure, while serverless abstracts infrastructure management entirely.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Plan[Plan Workspace] --> Network[Configure Network]
Network --> Storage[Configure Storage]
Storage --> Deploy[Deploy Workspace]
Deploy --> Metastore[Attach Metastore]
Metastore --> Compute[Configure Compute]
Compute --> Validate[Validate & Onboard]
Plan:::source
Network:::ingestion
Storage:::processing
Deploy:::storage
Metastore:::governance
Compute:::serving
Validate:::source
*Workspace deployment pipeline: from planning through network and storage configuration to compute enablement and validation.*
Key Terms
Prerequisites and Setup
- Account admin access for workspace creation
- Cloud provider account with sufficient quotas (vCPU, VPC, subnet limits)
- Network architecture decision (VPC peering, Private Link, or serverless)
- Storage account for DBFS root and Unity Catalog managed storage
- IAM cross-account role (AWS) or service principal (Azure) for Databricks access
- Terraform or Databricks CLI for reproducible deployments
Step-by-Step Implementation
Configuration Reference
| Feature | Traditional | Serverless |
|---|---|---|
| Infrastructure Owner | Customer | Databricks |
| Startup Time | 3-10 minutes | 5-15 seconds |
| Network Control | Full (VPC/VNet) | NCC egress rules |
| Scaling | Manual or autoscale | Automatic |
| Instance Types | Customer choice | Databricks-optimized |
| Private Link | Front-end + Back-end | Serverless Private Endpoints |
| CMK Support | Full | Managed services only |
| Cost Model | DBU + infrastructure | DBU only (bundled) |
| Maintenance | Customer-managed patches | Automatic updates |