MLflow is pre-configured with Databricks tracking
Who this is for:
Architecture / Concept Overview: MLflow is pre-configured with Databricks tracking
The Databricks Runtime (DBR) is a versioned, managed package that includes everything needed to run data engineering, data science, and ML workloads. Each version bundles a specific Spark release, Delta Lake, and hundreds of pre-installed libraries, plus proprietary optimizations like Photon and optimized I/O.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
DBR[Databricks Runtime]:::processing
DBR --> SPARK[Apache Spark - Optimized]:::processing
DBR --> DELTA[Delta Lake]:::storage
DBR --> PHOTON[Photon Engine]:::serving
DBR --> LIBS[Pre-installed Libraries]:::source
DBR --> IO[Optimized I/O]:::storage
DBR --> SEC[Security & Governance]:::governance
SPARK --> AQE[Adaptive Query Execution]:::processing
SPARK --> CACHE[Disk Caching]:::storage
PHOTON --> VEC[Vectorized C++ Engine]:::serving
IO --> SKIP[Data Skipping]:::storage
IO --> PRED[Predicate Pushdown]:::storage
*Databricks Runtime bundles Spark with proprietary optimizations, Delta Lake, and the Photon engine.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
VER[Runtime Versions]:::processing
VER --> LTS[LTS - Long Term Support]:::serving
VER --> STD[Standard - Latest Features]:::processing
VER --> ML[ML Runtime - ML Libraries]:::ingestion
VER --> GPU[GPU Runtime - CUDA/cuDNN]:::source
LTS --> L1[Supported for 2+ years]:::serving
LTS --> L2[Bug fixes and security patches]:::serving
LTS --> L3[Recommended for production]:::serving
STD --> S1[Latest Spark & Delta features]:::processing
STD --> S2[Shorter support window]:::processing
ML --> M1[PyTorch, TensorFlow, Hugging Face]:::ingestion
ML --> M2[MLflow pre-configured]:::ingestion
GPU --> G1[NVIDIA CUDA support]:::source
GPU --> G2[GPU-accelerated ML training]:::source
*Runtime variants serve different workload types.*
Key Terms
Prerequisites and Setup
- A Databricks workspace where you can create or configure clusters.
- Understanding of your workload type (data engineering, ML, or general analytics).
Step-by-Step Implementation
Configuration Reference
| Parameter | Description | Default |
|---|---|---|
spark.databricks.io.cache.enabled | Enable local SSD disk caching | false |
spark.databricks.io.cache.maxDiskUsage | Max disk space for cache | 50GB |
spark.sql.adaptive.enabled | Enable AQE | true |
spark.databricks.delta.optimizeWrite.enabled | Auto-optimize Delta write file sizes | true |
spark.databricks.delta.autoCompact.enabled | Auto-compact small Delta files | false |
spark.databricks.photon.enabled | Enable Photon engine | Depends on runtime_engine setting |
spark.databricks.preemption.enabled | Enable task preemption for fairness | true |