Schema Evolution and Schema Enforcement in Delta Lake
Who this is for:
Architecture / Concept Overview: Schema Evolution and Schema Enforcement in Delta Lake
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
W[Writer] -->|Schema Check| ENF{Schema Enforced?}
ENF -->|Match| COMMIT[Commit Write]
ENF -->|Mismatch + mergeSchema| EVOLVE[Evolve Schema & Commit]
ENF -->|Mismatch, no flag| REJECT[Reject Write]
COMMIT --> DT[Delta Table]
EVOLVE --> DT
W:::ingestion
ENF:::governance
COMMIT:::storage
EVOLVE:::processing
REJECT:::source
DT:::storage
*Every write passes through schema enforcement. Mismatches are rejected unless the writer explicitly opts into schema evolution.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
CHANGES[Schema Change Types] --> ADD[Add Column]
CHANGES --> RENAME[Rename Column]
CHANGES --> DROP[Drop Column]
CHANGES --> TYPE[Widen Type]
CHANGES --> NEST[Add Nested Field]
ADD -->|mergeSchema| SAFE[Safe — additive]
RENAME -->|ALTER TABLE| SAFE2[Safe — column mapping]
DROP -->|ALTER TABLE| RISK[Requires column mapping]
TYPE -->|mergeSchema| SAFE3[INT → LONG, FLOAT → DOUBLE]
NEST -->|mergeSchema| SAFE4[Add field to struct]
CHANGES:::governance
ADD:::processing
RENAME:::processing
DROP:::processing
TYPE:::processing
NEST:::processing
SAFE:::serving
SAFE2:::serving
RISK:::ingestion
SAFE3:::serving
SAFE4:::serving
*Different schema changes have different safety profiles and require different API options or DDL statements.*
Key Terms
Prerequisites and Setup
- Databricks Runtime 13.3 LTS or later
- Unity Catalog with column mapping enabled (default for new tables)
ALTER TABLEprivilege for DDL-based schema changesMODIFYprivilege for write-time schema evolution
Step-by-Step Implementation
Configuration Reference
| Property | Default | Description |
|---|---|---|
delta.columnMapping.mode | name | Enables rename/drop; values: none, name, id |
mergeSchema (write option) | false | Allows additive changes during writes |
overwriteSchema (write option) | false | Replaces entire schema on overwrite |
spark.databricks.delta.schema.autoMerge.enabled | false | Session-level auto schema evolution |
delta.minReaderVersion | 2 | Required for column mapping |
delta.minWriterVersion | 5 | Required for column mapping |