Deep Learning on Databricks: PyTorch and TensorFlow
Who this is for:
Architecture / Concept Overview: Deep Learning on Databricks: PyTorch and TensorFlow
Deep learning on Databricks leverages GPU clusters, optimised data loading from Delta, and integrated experiment tracking.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
DELTA[Delta Table] -->|Load| PETASTORM[Petastorm / Mosaic StreamingDataset]
PETASTORM -->|Batch| LOADER[DataLoader]
LOADER -->|Forward| GPU[GPU Training Loop]
GPU -->|Log| MLF[MLflow Tracking]
MLF -->|Register| UC[Unity Catalog Model Registry]
UC -->|Deploy| SERVE[GPU Serving Endpoint]
DELTA:::source
PETASTORM:::ingestion
LOADER:::processing
GPU:::processing
MLF:::governance
UC:::governance
SERVE:::serving
*Deep learning pipeline: Delta data flows through optimised loaders into GPU training, then into tracked and served models.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
DL[Deep Learning Stack] --> PYTORCH[PyTorch]
DL --> TF[TensorFlow]
PYTORCH --> PT_LIGHT[PyTorch Lightning]
PYTORCH --> HF_PT[Hugging Face - PyTorch]
PYTORCH --> TORCH_DIST[TorchDistributor]
TF --> KERAS[Keras]
TF --> TF_DIST[tf.distribute]
DL:::governance
PYTORCH:::processing
TF:::storage
PT_LIGHT:::ingestion
HF_PT:::ingestion
TORCH_DIST:::ingestion
KERAS:::source
TF_DIST:::source
*Supported deep learning frameworks and their distributed training extensions.*
Key Terms
Prerequisites and Setup
- Databricks Runtime for ML (GPU variant) with CUDA support.
- GPU instance types (e.g.,
g5.xlargeon AWS,Standard_NCon Azure). - MLflow autologging enabled for automatic experiment capture.
Step-by-Step Implementation
Configuration Reference
| Parameter | Default | Description |
|---|---|---|
batch_size | 32 | Number of samples per training batch |
learning_rate | 1e-3 | Optimizer step size |
epochs | varies | Number of full passes over the training data |
mixed_precision | false | Enable AMP for faster training and lower memory |
num_workers (DataLoader) | 0 | Parallel data loading workers |
pin_memory | false | Pin memory for faster GPU data transfer |