Monitoring Pipeline Runs and Handling Failures
Who this is for:
Architecture / Concept Overview: Monitoring Pipeline Runs and Handling Failures
Databricks provides multiple layers of observability: the pipeline event log for Declarative Pipelines, the Jobs run history for orchestration, system tables for workspace-wide metrics, and integration points for external alerting systems.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
PL[Declarative Pipeline]:::processing --> EL[Event Log]:::storage
JB[Lakeflow Job]:::processing --> RH[Run History]:::storage
EL --> DQ[Data Quality Metrics]:::governance
EL --> PM[Performance Metrics]:::governance
RH --> TS[Task Status & Duration]:::governance
DQ --> DASH[Monitoring Dashboard]:::serving
PM --> DASH
TS --> DASH
DASH --> ALT[Alerts: Email / Webhook / PagerDuty]:::source
*Observability stack: pipeline events and job runs feed dashboards and alerts.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
FAIL[Failure Handling]:::source
FAIL --> RETRY[Automatic Retries]:::processing
FAIL --> REPAIR[Manual Repair Run]:::processing
FAIL --> ALERT[Alert & Escalate]:::governance
FAIL --> QUARANTINE[Quarantine Bad Data]:::storage
RETRY --> R1[Max retries per task]:::processing
RETRY --> R2[Retry interval]:::processing
REPAIR --> RP1[Re-run failed tasks only]:::processing
REPAIR --> RP2[Preserve upstream results]:::processing
*Failure handling strategies available in Lakeflow.*
Key Terms
Prerequisites and Setup
- Existing Declarative Pipelines and/or Lakeflow Jobs to monitor.
- A Databricks SQL warehouse for querying event logs and system tables.
- Access to notification channels (email, Slack webhooks, PagerDuty).
Step-by-Step Implementation
Configuration Reference
| Parameter | Description | Default |
|---|---|---|
max_retries | Automatic retries per task on failure | 0 |
min_retry_interval_millis | Minimum delay between retries | 0 |
timeout_seconds | Task timeout before cancellation | 0 (no limit) |
health.rules[].metric | Health metric: RUN_DURATION_SECONDS | Required |
health.rules[].op | Comparison operator | Required |
health.rules[].value | Threshold value | Required |
email_notifications.on_failure | Email addresses for failure notifications | None |
webhook_notifications.on_failure | Webhook IDs for failure notifications | None |