Verify runtime and key libraries
Who this is for:
Architecture / Concept Overview: Verify runtime and key libraries
The Mosaic AI stack sits on top of the lakehouse and leverages Unity Catalog for governance at every stage.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
A[Raw Data Sources] -->|Ingest| B[Feature Engineering]
B -->|Store| C[Feature Store / UC]
C -->|Train| D[AutoML / Custom Training]
D -->|Track| E[MLflow Registry]
E -->|Deploy| F[Model Serving Endpoints]
F -->|Monitor| G[Lakehouse Monitoring]
A:::source
B:::ingestion
C:::storage
D:::processing
E:::governance
F:::serving
G:::governance
*End-to-end Mosaic AI pipeline from raw data to monitored production endpoints.*
The governance layer wraps every artifact — features, models, endpoints — inside Unity Catalog.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
UC[Unity Catalog] --> FM[Feature Models]
UC --> MR[Model Registry]
UC --> SE[Serving Endpoints]
UC --> MON[Monitoring Dashboards]
FM --> ACL[Row / Column ACLs]
MR --> ACL
SE --> ACL
MON --> ACL
UC:::governance
FM:::storage
MR:::processing
SE:::serving
MON:::governance
ACL:::source
*Unity Catalog provides a single governance plane across features, models, endpoints, and monitoring.*
Key Terms
Prerequisites and Setup
- A Databricks workspace on AWS, Azure, or GCP with Unity Catalog enabled.
- Premium or Enterprise tier (Model Serving and Feature Engineering require Premium).
- A cluster running the latest Databricks Runtime for ML (select the ML variant when creating a cluster).
CREATE MODELandUSE CATALOGprivileges granted via Unity Catalog.
Step-by-Step Implementation
Configuration Reference
| Parameter | Default | Description |
|---|---|---|
workload_size | Small | Serving compute size — Small, Medium, or Large |
scale_to_zero_enabled | true | Whether the endpoint scales to zero when idle |
timeout_minutes (AutoML) | 120 | Maximum wall-clock time AutoML explores |
primary_keys (Feature Table) | — | Columns that uniquely identify a feature row |
inference_table | None | Delta table for logging prediction requests and responses |
route_optimized | false | Enable optimized routing for high-throughput serving |