Integrating Databricks with S3, IAM, and VPC
Who this is for:
Architecture / Concept Overview: Integrating Databricks with S3, IAM, and VPC
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
CLUSTER[Databricks Cluster] -->|Instance Profile| S3_RAW[S3 - Raw Zone]
CLUSTER -->|Instance Profile| S3_CURATED[S3 - Curated Zone]
CLUSTER -->|Instance Profile| S3_ANALYTICS[S3 - Analytics Zone]
S3_RAW -->|KMS| KEY_RAW[KMS Key - Raw]
S3_CURATED -->|KMS| KEY_CUR[KMS Key - Curated]
S3_ANALYTICS -->|KMS| KEY_ANA[KMS Key - Analytics]
CLUSTER -->|VPC Endpoint| S3_EP[S3 Gateway Endpoint]
S3_EP --> S3_RAW
S3_EP --> S3_CURATED
S3_EP --> S3_ANALYTICS
CLUSTER:::processing
S3_RAW:::storage
S3_CURATED:::storage
S3_ANALYTICS:::storage
KEY_RAW:::governance
KEY_CUR:::governance
KEY_ANA:::governance
S3_EP:::serving
*Data access pattern showing how clusters use instance profiles and VPC endpoints to access S3 zones with per-zone KMS encryption.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
VPC[Customer VPC - 10.0.0.0/16] --> PRIV1[Private Subnet AZ-a - 10.0.1.0/24]
VPC --> PRIV2[Private Subnet AZ-b - 10.0.2.0/24]
VPC --> NAT_SUB[Public Subnet - NAT - 10.0.0.0/24]
PRIV1 --> SG[Security Group - Databricks]
PRIV2 --> SG
NAT_SUB --> NAT[NAT Gateway]
NAT --> IGW[Internet Gateway]
VPC --> S3EP[S3 VPC Endpoint]
VPC --> STSEP[STS VPC Endpoint]
VPC:::storage
PRIV1:::serving
PRIV2:::serving
NAT_SUB:::ingestion
SG:::governance
NAT:::ingestion
IGW:::source
S3EP:::storage
STSEP:::governance
*Customer-managed VPC architecture showing private subnets, NAT gateway, and VPC endpoints for S3 and STS.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
CROSS[Cross-Account Role] -->|Manage| EC2[EC2 Instances]
CROSS -->|Manage| SG2[Security Groups]
IP[Instance Profile] -->|Data Access| S3[S3 Buckets]
IP -->|Decrypt| KMS[KMS Keys]
EC2 -->|Assumes| IP
CROSS -.->|Cannot Access| S3
CROSS:::governance
EC2:::processing
SG2:::governance
IP:::governance
S3:::storage
KMS:::governance
*IAM role separation between the cross-account role (infrastructure management) and instance profiles (data access).*
Key Terms
Prerequisites and Setup
- Databricks workspace deployed (Databricks-managed or customer-managed VPC)
- AWS CLI configured with permissions to create IAM roles, VPCs, and S3 buckets
- Understanding of IAM role chaining and trust policies
- S3 buckets created for your data lake zones (raw, curated, analytics)
- KMS keys created for S3 encryption (optional but recommended)
Step-by-Step Implementation
Configuration Reference
| Component | Configuration | Purpose | Best Practice |
|---|---|---|---|
| Instance Profile | IAM Role + EC2 trust | Cluster data access | One per workload/security boundary |
| S3 Bucket Policy | Principal + condition | Restrict bucket access | Require VPC endpoint source |
| S3 VPC Endpoint | Gateway type + route table | Private S3 access | Always create — eliminates NAT costs for S3 |
| STS VPC Endpoint | Interface type + private DNS | Private credential calls | Required for private-only clusters |
| Security Group | Inbound: self, Outbound: all | Cluster node communication | Allow all internal, restrict egress |
| NAT Gateway | One per AZ | Outbound internet for cluster nodes | Multi-AZ for high availability |
| KMS Key | Key policy + IAM grants | S3 server-side encryption | Separate keys per data classification |