Lakeflow Designer: Visual Drag-and-Drop Data Preparation
Who this is for:
Architecture / Concept Overview: Lakeflow Designer: Visual Drag-and-Drop Data Preparation
Lakeflow Designer sits on top of the Declarative Pipelines runtime, generating pipeline code from a visual canvas. Users drag source, transformation, and destination nodes onto the canvas, connect them, and configure each node through a GUI. The Designer then compiles the visual graph into Declarative Pipeline definitions that execute on the standard runtime.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
USER[Designer Canvas]:::source --> SN[Source Nodes]:::ingestion
SN --> TN[Transform Nodes]:::processing
TN --> DN[Destination Nodes]:::storage
DN --> COMPILE[Compile to Pipeline Code]:::processing
COMPILE --> DLP[Declarative Pipeline Runtime]:::serving
DLP --> DT[Delta Tables in Unity Catalog]:::storage
*Visual designs are compiled into Declarative Pipeline definitions and executed by the standard runtime.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
NT[Node Types]:::processing
NT --> SRC[Source Nodes]:::ingestion
NT --> TRN[Transform Nodes]:::processing
NT --> DEST[Destination Nodes]:::storage
SRC --> S1[Table / View]:::ingestion
SRC --> S2[File Source]:::ingestion
TRN --> T1[Filter]:::processing
TRN --> T2[Aggregate]:::processing
TRN --> T3[Join]:::processing
TRN --> T4[Column Transform]:::processing
TRN --> T5[SQL Expression]:::processing
DEST --> D1[Streaming Table]:::storage
DEST --> D2[Materialized View]:::storage
*Available node types in the Lakeflow Designer canvas.*
Key Terms
Prerequisites and Setup
- Databricks workspace with Unity Catalog enabled.
- Access to the Lakeflow Designer feature (check workspace admin settings).
CREATE TABLEpermissions on the target catalog and schema.- Source tables or files already accessible in Unity Catalog.
Step-by-Step Implementation
Configuration Reference
| Setting | Description | Default |
|---|---|---|
| Target Catalog | Unity Catalog catalog for output tables | Required |
| Target Schema | Schema within the catalog | Required |
| Compute | Cluster configuration for pipeline execution | Default pipeline cluster |
| Photon | Enable Photon acceleration | Inherits from pipeline |
| Schedule | Optional cron schedule for automatic runs | None (manual) |