Building a Data Mesh with Delta Sharing

    Who this is for:

    Architecture / Concept Overview: Building a Data Mesh with Delta Sharing

    A Data Mesh built on Databricks uses Unity Catalog as the self-serve data platform, Delta Sharing as the distribution mechanism, and domain teams as autonomous data product owners.

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED A[Sales Domain] -->|Publish Share| D[Federated Catalog] B[Marketing Domain] -->|Publish Share| D C[Engineering Domain] -->|Publish Share| D D -->|Discover & Query| E[Consumer: Analytics Team] D -->|Discover & Query| F[Consumer: ML Team] D -->|Governed by| G[Platform / Governance Team] A:::source B:::source C:::source D:::governance E:::serving F:::serving G:::governance

    *Each domain team publishes data products as Delta Shares, discoverable through a federated Unity Catalog governed by the platform team.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED A[Data Mesh Principles] --> B[Domain Ownership] A --> C[Data as a Product] A --> D[Self-Serve Platform] A --> E[Federated Governance] B --> F[Teams own their data end-to-end] C --> G[SLAs, docs, quality metrics] D --> H[Unity Catalog + Delta Sharing] E --> I[Global policies + local autonomy] A:::processing B:::source C:::ingestion D:::serving E:::governance F:::source G:::ingestion H:::storage I:::governance

    *The four Data Mesh principles map directly to Databricks capabilities: domain catalogs, share-backed data products, Unity Catalog as platform, and federated governance policies.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED A[Domain Workspace] -->|Own & Transform| B[Domain Catalog] B -->|Create Data Product| C[Unity Catalog Share] C -->|Register| D[Central Registry] D -->|Discover| E[Consumer Workspace] E -->|Mount Shared Catalog| F[Query Data Product] A:::source B:::storage C:::processing D:::governance E:::serving F:::serving

    *The data product lifecycle flows from domain-owned transformation through share creation, central registration, and consumer discovery.*

    Key Terms

    Prerequisites and Setup

    • Multiple Databricks workspaces (one per domain or a shared workspace with domain-specific catalogs)
    • Unity Catalog metastore spanning all participating workspaces
    • A platform or governance team to manage shared policies and the data product registry
    • Domain teams ready to own their data products end-to-end

    Step-by-Step Implementation

      Configuration Reference

      Building a Data Mesh with Delta Sharing configuration options
      ComponentDatabricks ImplementationData Mesh Role
      Domain ownershipSeparate catalogs per domainAutonomous domain teams
      Data productsCurated tables with TBLPROPERTIES metadataDiscoverable, documented assets
      DistributionDelta Sharing (D2B or open)Zero-copy cross-domain access
      DiscoveryCentral registry table + MarketplaceSelf-serve product catalog
      GovernanceUnity Catalog policies + audit logsFederated compliance
      QualityData quality rules + monitoringSLA enforcement
      ContractsTBLPROPERTIES + schema enforcementProducer-consumer agreements

      Monitoring, Cost, and Security Considerations

      Common Pitfalls and Recommended Patterns

        Frequently Asked Questions