Syncing Unity Catalog Tables into Lakebase for Low-Latency Reads
Who this is for:
Architecture / Concept Overview: Syncing Unity Catalog Tables into Lakebase for Low-Latency Reads
While the primary Lakebase sync direction pushes operational data to Delta Lake, reverse sync brings analytical results back into Lakebase for low-latency serving.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
A[Raw Data Sources] -->|Ingest| B[Delta Lake]
B -->|Transform| C[Databricks SQL / Spark]
C -->|Aggregated Results| D[Unity Catalog Table]
D -->|Reverse Sync| E[Lakebase]
E -->|Low Latency| F[Application API]
F -->|Serve Users| G[Web / Mobile UI]
A:::source
B:::storage
C:::processing
D:::governance
E:::serving
F:::serving
G:::source
*Reverse sync brings pre-computed analytical results from Unity Catalog into Lakebase for sub-millisecond application reads.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
A[Use Cases for Reverse Sync] --> B[ML Predictions]
A --> C[Pre-Computed Aggregations]
A --> D[Reference / Lookup Data]
A --> E[Feature Store Snapshots]
B --> F[Serve via Application]
C --> F
D --> F
E --> F
A:::processing
B:::ingestion
C:::ingestion
D:::source
E:::storage
F:::serving
*Common reverse-sync use cases include serving ML predictions, aggregated metrics, and reference data through the application layer.*
Key Terms
Prerequisites and Setup
- A Lakebase project with an existing database
- Delta tables in Unity Catalog containing the data you want to serve
- Appropriate permissions on both the Unity Catalog tables (SELECT) and the Lakebase database (CREATE TABLE)
- Unity Catalog metastore attached to your workspace
Step-by-Step Implementation
Configuration Reference
| Parameter | Description | Default |
|---|---|---|
source.catalog | Unity Catalog catalog containing the source table | Required |
source.schema | Schema within the source catalog | Required |
source.table | Table name to sync into Lakebase | Required |
sync.interval | Refresh frequency | 15 minutes |
sync.mode | full_refresh or incremental | full_refresh |
sync.conflict_on_existing | Behavior if table already exists in Lakebase | error |
sync.primary_key | Column(s) used for incremental merge (incremental mode) | None |
target.table_name | Override the table name in Lakebase | Same as source |