Option 3: Named profile from ~/.databrickscfg
Who this is for:
Architecture / Concept Overview: Option 3: Named profile from ~/.databrickscfg
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
APP[Python Application]:::source
SDK[Databricks SDK]:::ingestion
AUTH[Unified Auth]:::governance
RETRY[Auto Retry / Pagination]:::processing
API[REST API]:::storage
PLATFORM[Databricks Platform]:::serving
APP --> SDK --> AUTH
SDK --> RETRY --> API --> PLATFORM
*The SDK abstracts authentication, retry logic, and pagination into a single client interface.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
CLIENT[WorkspaceClient]:::source
CLUSTERS[clusters]:::processing
JOBS[jobs]:::serving
WAREHOUSES[warehouses]:::storage
CATALOG[unity catalog]:::governance
FILES[files / dbfs]:::ingestion
SERVING[model serving]:::serving
CLIENT --> CLUSTERS
CLIENT --> JOBS
CLIENT --> WAREHOUSES
CLIENT --> CATALOG
CLIENT --> FILES
CLIENT --> SERVING
*The WorkspaceClient exposes service-specific sub-clients for each Databricks domain.*
Key Terms
Prerequisites and Setup
- Python 3.8 or later
- pip or conda for package management
- A Databricks workspace with a configured authentication method
- IDE with type checking support (VS Code, PyCharm) recommended
Step-by-Step Implementation
Configuration Reference
| Parameter | Environment Variable | Description |
|---|---|---|
host | DATABRICKS_HOST | Workspace URL |
token | DATABRICKS_TOKEN | Personal access token |
client_id | DATABRICKS_CLIENT_ID | OAuth client ID |
client_secret | DATABRICKS_CLIENT_SECRET | OAuth client secret |
profile | DATABRICKS_CONFIG_PROFILE | Named profile from config file |
azure_tenant_id | ARM_TENANT_ID | Azure AD tenant for Azure auth |
cluster_id | DATABRICKS_CLUSTER_ID | Default cluster for operations |