The Medallion Architecture: Bronze, Silver, and Gold Layers

    Who this is for:

    Architecture / Concept Overview: The Medallion Architecture: Bronze, Silver, and Gold Layers

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED SRC[Source Systems] -->|Raw Ingest| BRONZE[Bronze Layer] BRONZE -->|Cleanse, Deduplicate, Validate| SILVER[Silver Layer] SILVER -->|Aggregate, Join, Enrich| GOLD[Gold Layer] GOLD -->|Serve| BI[BI Dashboards] GOLD -->|Serve| ML[ML Features] GOLD -->|Serve| API[Data Products] SRC:::source BRONZE:::ingestion SILVER:::processing GOLD:::storage BI:::serving ML:::serving API:::serving

    *Data flows progressively through three layers, gaining quality and structure at each stage.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED BRONZE_D[Bronze Characteristics] --> RAW[Raw, append-only] BRONZE_D --> SCHEMA_FLEX[Schema-on-read] BRONZE_D --> AUDIT[Full audit trail] BRONZE_D --> REPLAY[Replayable from source] SILVER_D[Silver Characteristics] --> CLEAN[Cleansed, deduplicated] SILVER_D --> CONFORM[Conformed schema] SILVER_D --> JOIN[Joined reference data] SILVER_D --> VALID[Data quality validated] GOLD_D[Gold Characteristics] --> AGG[Aggregated metrics] GOLD_D --> BIZ[Business-level entities] GOLD_D --> SLA[SLA-governed] GOLD_D --> READY[Query-optimised] BRONZE_D:::ingestion RAW:::ingestion SCHEMA_FLEX:::ingestion AUDIT:::ingestion REPLAY:::ingestion SILVER_D:::processing CLEAN:::processing CONFORM:::processing JOIN:::processing VALID:::processing GOLD_D:::storage AGG:::storage BIZ:::storage SLA:::storage READY:::storage

    *Each layer has distinct characteristics that define its data quality guarantees and access patterns.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED UC[Unity Catalog] -->|catalog.bronze.table| B_NS[Bronze Schema] UC -->|catalog.silver.table| S_NS[Silver Schema] UC -->|catalog.gold.table| G_NS[Gold Schema] B_NS -->|Ingest team owns| OWN_B[Data Engineers] S_NS -->|Transform team owns| OWN_S[Analytics Engineers] G_NS -->|Product team owns| OWN_G[Data Product Owners] UC:::governance B_NS:::ingestion S_NS:::processing G_NS:::storage OWN_B:::ingestion OWN_S:::processing OWN_G:::serving

    *Unity Catalog schemas map naturally to medallion layers, enabling clear ownership and access control per layer.*

    Key Terms

    Prerequisites and Setup

    • Databricks workspace with Unity Catalog enabled
    • Separate schemas (or catalogs) for each layer
    • Source data landing in cloud storage
    • Service principals with appropriate grants per layer

    Step-by-Step Implementation

      Configuration Reference

      The Medallion Architecture: Bronze, Silver, and Gold Layers configuration options
      LayerTypical SchemaAccessRetentionKey Properties
      BronzeFlexible, append-onlyIngest teamLong (years)rescuedDataColumn, cloudFiles.*
      SilverStrict, SCD Type 1/2Analytics engineersMedium (months)enableChangeDataFeed, CLUSTER BY
      GoldAggregated, star schemaAnalysts, BI toolsShort (refreshed)Materialised views or scheduled refresh

      Monitoring, Cost, and Security Considerations

      Common Pitfalls and Recommended Patterns

        Frequently Asked Questions