models/marts/_models.yml
Who this is for:
Architecture / Concept Overview: models/marts/_models.yml
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
RAW[Raw Data in Lakehouse]:::source
DBT[dbt Project]:::ingestion
SQL_WH[SQL Warehouse]:::processing
DELTA[Delta Lake Tables]:::storage
TESTS[dbt Tests]:::governance
DOCS[dbt Docs / Lineage]:::serving
RAW --> DBT --> SQL_WH --> DELTA
DBT --> TESTS
DBT --> DOCS
*dbt submits SQL transformations to Databricks SQL Warehouses, materializing results as Delta tables with automated testing and documentation.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
LAYERS[dbt Model Layers]:::source
STAGING[staging (stg_)]:::ingestion
INTERMEDIATE[intermediate (int_)]:::processing
MARTS[marts (fct_, dim_)]:::serving
LAYERS --> STAGING --> INTERMEDIATE --> MARTS
STAGING --> S1[1:1 source mapping<br/>Renaming, casting]:::ingestion
INTERMEDIATE --> I1[Business logic<br/>Joins, filters]:::processing
MARTS --> M1[Business entities<br/>Consumption-ready]:::serving
*dbt models follow a layered architecture from staging through intermediate to mart-level tables.*
Key Terms
Prerequisites and Setup
- Databricks workspace with a SQL Warehouse
- dbt Core (CLI) or dbt Cloud account
- Python 3.8+ for dbt Core
- Unity Catalog with target schemas created
- Git repository for the dbt project
Step-by-Step Implementation
Configuration Reference
| dbt Config | Databricks Behavior | Description |
|---|---|---|
materialized: table | CREATE OR REPLACE TABLE | Full table rebuild each run |
materialized: view | CREATE OR REPLACE VIEW | Virtual view (no storage) |
materialized: incremental | MERGE INTO / INSERT | Process only new data |
materialized: ephemeral | CTE (no object created) | Inline subquery |
partition_by | PARTITIONED BY | Physical data partitioning |
liquid_clustered_by | CLUSTER BY | Liquid clustering columns |
file_format: delta | Default | Always Delta on Databricks |