Lakeflow Declarative Pipelines (formerly Delta Live Tables): Introduction

    Who this is for:

    Architecture / Concept Overview: Lakeflow Declarative Pipelines (formerly Delta Live Tables): Introduction

    Declarative Pipelines shift data engineering from "write the how" to "describe the what." You define tables and views with their transformation logic and quality expectations. The pipeline runtime resolves dependencies, determines execution order, manages state, and processes data incrementally.

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED SRC[Cloud Storage / Kafka]:::source --> AL[Auto Loader / Stream]:::ingestion AL --> ST1[Streaming Table: raw_events]:::storage ST1 --> DQ[Expectations Check]:::governance DQ --> ST2[Streaming Table: clean_events]:::storage ST2 --> MV[Materialized View: daily_summary]:::serving MV --> BI[BI Dashboards]:::serving

    *A Declarative Pipeline flowing from ingestion through quality validation to analytics-ready views.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED RT[Pipeline Runtime]:::processing RT --> DEP[Dependency Resolution]:::processing RT --> INC[Incremental Processing]:::processing RT --> DQ[Data Quality Enforcement]:::governance RT --> LIN[Lineage Tracking]:::governance RT --> ERR[Error Handling & Retries]:::processing RT --> SCH[Schema Management]:::storage

    *The Declarative Pipeline runtime manages six core responsibilities automatically.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED T[Table Types]:::processing T --> STR[Streaming Table]:::storage T --> MVW[Materialized View]:::serving T --> VW[View - temporary]:::processing STR --> |Append-only incremental| DELTA1[Delta Table]:::storage MVW --> |Recomputed on change| DELTA2[Delta Table]:::storage VW --> |Not persisted| TEMP[In-memory only]:::processing

    *Three table types in Declarative Pipelines and their persistence behaviour.*

    Key Terms

    Prerequisites and Setup

    • Databricks workspace with Unity Catalog enabled.
    • CREATE TABLE and CREATE SCHEMA permissions in the target catalog.
    • Pipeline source notebooks stored in a Databricks Repo or workspace folder.
    • Databricks Runtime 13.3 LTS or later (the pipeline runtime selects it automatically).

    Step-by-Step Implementation

      Configuration Reference

      Lakeflow Declarative Pipelines (formerly Delta Live Tables): Introduction configuration options
      ParameterDescriptionDefault
      catalogUnity Catalog catalog for output tablesRequired
      targetSchema within the catalog for output tablesRequired
      continuousRun the pipeline continuously or in triggered modefalse
      photonEnable Photon accelerationfalse
      clusters.autoscale.min_workersMinimum worker count1
      clusters.autoscale.max_workersMaximum worker count5
      pipelines.maxFlowRetryAttemptsNumber of retries for failed flows2
      developmentEnable development mode (relaxed error handling, no retries)false

      Monitoring, Cost, and Security Considerations

      Common Pitfalls and Recommended Patterns

        Frequently Asked Questions