Databricks Connect: Running Spark Locally Against a Remote Cluster
Who this is for:
Architecture / Concept Overview: Databricks Connect: Running Spark Locally Against a Remote Cluster
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
IDE[Local IDE]:::source
DC[Databricks Connect Client]:::ingestion
GRPC[gRPC / Spark Connect]:::processing
CLUSTER[Remote Cluster / Serverless]:::storage
DATA[Lakehouse Data]:::serving
UC[Unity Catalog]:::governance
IDE --> DC --> GRPC --> CLUSTER --> DATA
CLUSTER --> UC
*Databricks Connect sends Spark plans (not data) over gRPC to a remote cluster for execution, returning results to your local session.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
V2[Databricks Connect v2]:::source
SPARK[Spark Connect Protocol]:::processing
CLASSIC[Classic All-Purpose Cluster]:::storage
SERVERLESS[Serverless Compute]:::serving
LOCAL[Local Development]:::ingestion
TEST[Unit Testing]:::ingestion
V2 --> SPARK
SPARK --> CLASSIC
SPARK --> SERVERLESS
V2 --> LOCAL
V2 --> TEST
*Databricks Connect v2 uses the Spark Connect protocol and supports both classic clusters and serverless compute.*
Key Terms
Prerequisites and Setup
- Python 3.8+ with a virtual environment
- Databricks Runtime 13.3 LTS or later on the target cluster
- Network connectivity from your machine to the workspace (port 443)
- Unity Catalog enabled for the workspace (recommended)
- IDE with Python debugging support
Step-by-Step Implementation
Configuration Reference
| Parameter | Environment Variable | Description |
|---|---|---|
host | DATABRICKS_HOST | Workspace URL |
token | DATABRICKS_TOKEN | Personal access token |
cluster_id | DATABRICKS_CLUSTER_ID | Target cluster ID |
serverless | — | Use serverless compute (builder method) |
profile | DATABRICKS_CONFIG_PROFILE | Named profile from config file |