System Tables: Querying Account-Level Operational Data
Who this is for:
Architecture / Concept Overview: System Tables: Querying Account-Level Operational Data
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Platform[Platform Operations] --> Billing[system.billing]
Platform --> Access[system.access]
Platform --> Compute[system.compute]
Platform --> Lakeflow[system.lakeflow]
Platform --> Lineage[system.lineage]
Billing --> Usage[usage]
Billing --> Prices[list_prices]
Access --> Audit[audit]
Compute --> Clusters[clusters]
Lakeflow --> Jobs[jobs / job_run_timeline]
Lineage --> Deps[table_lineage / column_lineage]
Platform:::governance
Billing:::processing
Access:::governance
Compute:::ingestion
Lakeflow:::storage
Lineage:::serving
Usage:::source
Prices:::source
Audit:::source
Clusters:::source
Jobs:::source
Deps:::source
*System table schemas: operational data organized into billing, access, compute, lakeflow, and lineage domains.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
SystemTables[System Tables] --> SQL[SQL Queries]
SQL --> Views[Materialized Views]
Views --> Dashboards[Lakeview Dashboards]
Dashboards --> Alerts[Scheduled Alerts]
Alerts --> Actions[Automated Actions]
Actions --> Optimization[Platform Optimization]
SystemTables:::storage
SQL:::processing
Views:::processing
Dashboards:::serving
Alerts:::governance
Actions:::ingestion
Optimization:::source
*Analytics pipeline: system tables feed SQL queries that power dashboards, trigger alerts, and drive automated platform optimization.*
Key Terms
Prerequisites and Setup
- Databricks Premium or Enterprise tier (system tables require Premium+)
- Unity Catalog enabled with metastore assigned to workspace
- Account admin or system table reader grants for accessing billing data
- SQL warehouse (serverless recommended) for querying
- Understanding of SQL and Delta table optimization for efficient querying
Step-by-Step Implementation
Configuration Reference
| System Table | Schema | Update Frequency | Retention | Key Use Cases |
|---|---|---|---|---|
| system.billing.usage | billing | ~6 hours | 365 days | Cost analysis, chargeback |
| system.billing.list_prices | billing | Daily | Current only | Price calculations |
| system.access.audit | access | 15-60 minutes | 365 days | Security, compliance |
| system.compute.clusters | compute | Near real-time | 365 days | Utilization, right-sizing |
| system.compute.warehouses | compute | Near real-time | 365 days | SQL warehouse monitoring |
| system.lakeflow.jobs | lakeflow | Near real-time | 365 days | Job inventory |
| system.lakeflow.job_run_timeline | lakeflow | Near real-time | 365 days | Reliability, performance |
| system.lineage.table_lineage | lineage | ~1 hour | 365 days | Impact analysis |
| system.lineage.column_lineage | lineage | ~1 hour | 365 days | Column-level tracking |