Integrating Databricks with AWS Services (S3, Glue, Redshift)
Who this is for:
Architecture / Concept Overview: Integrating Databricks with AWS Services (S3, Glue, Redshift)
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
SOURCES[AWS Data Sources]:::source
KINESIS[Kinesis / MSK]:::ingestion
DBX[Databricks on AWS]:::processing
S3[Amazon S3]:::storage
REDSHIFT[Amazon Redshift]:::serving
GLUE[AWS Glue Catalog]:::governance
SOURCES --> KINESIS --> DBX
SOURCES --> S3
DBX --> S3
S3 --> REDSHIFT
DBX --> GLUE
DBX --> REDSHIFT
*Databricks on AWS uses S3 as primary storage, syncs metadata with Glue Data Catalog, and federates queries to Redshift for existing warehouse workloads.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
AWS_INT[AWS Integration Layers]:::source
STORAGE_L[Storage Layer]:::storage
CATALOG_L[Catalog Layer]:::governance
COMPUTE_L[Compute Layer]:::processing
NETWORK_L[Network Layer]:::ingestion
AWS_INT --> STORAGE_L
AWS_INT --> CATALOG_L
AWS_INT --> COMPUTE_L
AWS_INT --> NETWORK_L
STORAGE_L --> S3_B[S3 Buckets]:::storage
STORAGE_L --> S3_EXPR[S3 Express One Zone]:::storage
CATALOG_L --> GLUE_C[Glue Data Catalog]:::governance
CATALOG_L --> UC_C[Unity Catalog]:::governance
COMPUTE_L --> RED[Redshift Federation]:::processing
COMPUTE_L --> ATHENA[Athena Queries]:::processing
NETWORK_L --> VPC[VPC Peering]:::ingestion
NETWORK_L --> PLINK[PrivateLink]:::ingestion
*AWS integrations span storage, catalog, compute, and network layers for a comprehensive lakehouse architecture.*
Key Terms
Prerequisites and Setup
- Databricks workspace deployed on AWS
- S3 bucket for lakehouse storage with appropriate bucket policies
- IAM roles configured for cross-account access
- VPC peering or PrivateLink for Redshift connectivity
- Unity Catalog enabled with AWS storage credentials
Step-by-Step Implementation
Configuration Reference
| Integration | Auth Method | Network | Typical Latency |
|---|---|---|---|
| S3 Read/Write | IAM Role (Instance Profile / UC) | VPC Endpoint | Low (ms) |
| Glue Catalog | IAM Role | VPC Endpoint | Low (ms) |
| Redshift Federation | Username/Password via Secrets | VPC Peering / PrivateLink | Medium (seconds) |
| Kinesis Ingestion | IAM Role | VPC Endpoint | Real-time (ms) |
| Athena Queries | IAM Role | VPC Endpoint | Medium (seconds) |