Creating and Managing Delta Tables
Who this is for:
Architecture / Concept Overview: Creating and Managing Delta Tables
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
DDL[DDL Statement] -->|CREATE TABLE| CAT[Unity Catalog]
CAT -->|Register Metadata| META[Metastore]
CAT -->|Create _delta_log| STORE[Cloud Storage]
WRITE[Writer] -->|INSERT / MERGE| STORE
STORE -->|Serve Queries| READ[Readers]
DDL:::processing
CAT:::governance
META:::governance
STORE:::storage
WRITE:::ingestion
READ:::serving
*Creating a Delta table registers metadata in Unity Catalog and initialises the transaction log in cloud storage.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
TBL[Delta Table Types] --> MANAGED[Managed Table]
TBL --> EXTERNAL[External Table]
TBL --> CLONE[Cloned Table]
TBL --> TEMP[Temporary View]
MANAGED -->|Storage owned by catalog| M_DESC[DROP deletes data]
EXTERNAL -->|Storage at custom LOCATION| E_DESC[DROP removes metadata only]
CLONE -->|Deep or shallow copy| C_DESC[Independent versioning]
TBL:::storage
MANAGED:::processing
EXTERNAL:::source
CLONE:::ingestion
TEMP:::serving
M_DESC:::processing
E_DESC:::source
C_DESC:::ingestion
*Delta tables can be managed (catalog-owned storage), external (user-specified location), or cloned from existing tables.*
Key Terms
Prerequisites and Setup
- Databricks workspace with Unity Catalog enabled
CREATE TABLEandUSE SCHEMAprivileges on the target catalog and schema- Cluster running Databricks Runtime 13.3 LTS or later
Step-by-Step Implementation
Configuration Reference
| Property | Default | Description |
|---|---|---|
delta.enableChangeDataFeed | false | Record row-level changes for downstream CDC |
delta.autoOptimize.optimizeWrite | true | Coalesce small files during writes |
delta.autoOptimize.autoCompact | true | Background compaction after writes |
delta.logRetentionDuration | 30 days | Commit history retention |
delta.deletedFileRetentionDuration | 7 days | Minimum file age before VACUUM eligibility |
delta.enableDeletionVectors | true | Soft-delete rows without rewriting files |
delta.columnMapping.mode | name | Enables column rename and drop operations |
delta.minReaderVersion | 1 | Minimum protocol version for readers |
delta.minWriterVersion | 2 | Minimum protocol version for writers |