Managing External Storage and Data Platforms via Unity Catalog

    Who this is for:

    Architecture / Concept Overview: Managing External Storage and Data Platforms via Unity Catalog

    Unity Catalog acts as a governance gateway for data that lives outside the Databricks-managed storage, including cloud object storage, external databases, and third-party data shares.

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED UC[Unity Catalog] --> MANAGED[Managed Tables<br/>UC-controlled storage] UC --> EXT_LOC[External Locations<br/>S3 · ADLS · GCS] UC --> FED[Lakehouse Federation<br/>MySQL · PostgreSQL · SQL Server] UC --> SHARE[Delta Sharing<br/>Cross-org data sharing] EXT_LOC --> EXT_TBL[External Tables] EXT_LOC --> VOLUMES[Volumes<br/>Unstructured files] FED --> FOREIGN[Foreign Catalogs] SHARE --> SHARED_DATA[Shared Tables & Views] UC:::governance MANAGED:::storage EXT_LOC:::storage FED:::source SHARE:::serving EXT_TBL:::storage VOLUMES:::storage FOREIGN:::source SHARED_DATA:::serving

    *Figure 1 — Unity Catalog governs managed storage, external locations, federated databases, and Delta Sharing connections through a single permission model.*

    Storage credentials provide the cloud-provider authentication that Unity Catalog uses to access external data.

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED SC[Storage Credential] --> EL[External Location<br/>Scoped to a storage path] EL --> TBL1[External Table A] EL --> TBL2[External Table B] EL --> VOL[External Volume] SC:::governance EL:::storage TBL1:::serving TBL2:::serving VOL:::serving

    *Figure 2 — Storage credential → external location → external tables/volumes hierarchy.*

    Key Terms

    Prerequisites and Setup

    • Account admin or metastore admin role
    • Cloud IAM credentials with access to the target storage or database
    • Network connectivity from Databricks to external databases (for federation)
    • Unity Catalog enabled on the workspace

    Step-by-Step Implementation

      Configuration Reference

      Managing External Storage and Data Platforms via Unity Catalog configuration options
      ObjectKey PropertiesPrivileges
      Storage credentialCloud IAM ARN/IDCREATE EXTERNAL LOCATION, CREATE STORAGE CREDENTIAL
      External locationURL + credential referenceCREATE EXTERNAL TABLE, READ FILES, WRITE FILES
      External tableLOCATION + USING formatSame as managed tables: SELECT, MODIFY
      ConnectionHost, port, credentialsUSE CONNECTION, CREATE FOREIGN CATALOG
      Foreign catalogConnection referenceUSE CATALOG, USE SCHEMA, SELECT
      ShareTables/views includedSELECT ON SHARE granted to recipients
      VolumeStorage locationREAD VOLUME, WRITE VOLUME

      Monitoring, Cost, and Security Considerations

      Common Pitfalls and Recommended Patterns

        Frequently Asked Questions