Delta Sharing: The Open Protocol for Cross-Platform Data Sharing
Who this is for:
Architecture / Concept Overview: Delta Sharing: The Open Protocol for Cross-Platform Data Sharing
Delta Sharing operates as a REST protocol where the provider runs a sharing server (built into Unity Catalog) and consumers connect using lightweight clients that understand the protocol.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
A[Provider: Unity Catalog] -->|REST API| B[Delta Sharing Server]
B -->|Pre-signed URLs| C[Cloud Storage]
D[Python Client] -->|Bearer Token| B
E[Spark Client] -->|Bearer Token| B
F[Power BI Client] -->|Bearer Token| B
G[Rust / Go Client] -->|Bearer Token| B
D -->|Read Parquet| C
E -->|Read Parquet| C
F -->|Read Parquet| C
A:::governance
B:::processing
C:::storage
D:::source
E:::source
F:::source
G:::source
*Delta Sharing servers issue pre-signed URLs to cloud storage, letting any client read shared Parquet files directly without data copies.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
A[Delta Sharing Protocol] --> B[List Shares]
A --> C[List Schemas]
A --> D[List Tables]
A --> E[Query Table Metadata]
A --> F[Read Table Data]
F --> G[Predicate Pushdown]
F --> H[Column Selection]
F --> I[Pagination]
A:::processing
B:::serving
C:::serving
D:::serving
E:::ingestion
F:::storage
G:::source
H:::source
I:::source
*The protocol supports discovery (listing shares, schemas, tables), metadata inspection, and efficient data reads with predicate pushdown.*
Key Terms
Prerequisites and Setup
- Provider: A Databricks workspace with Unity Catalog and sharing enabled
- Consumer: A Delta Sharing client installed (Python, Spark, Power BI connector, or REST)
- The sharing profile file (
.sharecredential JSON) provided by the data provider - Network connectivity from the consumer to the provider's sharing server endpoint
Step-by-Step Implementation
Configuration Reference
| Parameter | Description | Default |
|---|---|---|
share.name | Unique share identifier | Required |
recipient.type | Must be OPEN for non-Databricks recipients | Auto-detected |
recipient.token_lifetime | Bearer token expiration | 90 days |
recipient.ip_access_list | Allowed IP ranges for the recipient | None (all IPs) |
share.cdf_enabled | Enable Change Data Feed for incremental reads | false |
share.partition_columns | Columns available for predicate pushdown | All partition columns |
client.timeout | HTTP timeout for client requests | 120 seconds |
client.max_retries | Number of retries on transient failures | 3 |