Data Encryption at Rest and in Transit
Who this is for:
Architecture / Concept Overview: Data Encryption at Rest and in Transit
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Client[Client] -->|"TLS 1.2+"| ControlPlane[Control Plane]
ControlPlane -->|"TLS 1.2+"| DataPlane[Data Plane Clusters]
DataPlane -->|"TLS 1.2+"| Storage[Cloud Storage]
Storage -->|"AES-256"| AtRest[Data at Rest]
DataPlane -->|"AES-256"| LocalDisk[Local Disk Encryption]
ControlPlane -->|"AES-256"| ManagedServices[Managed Services Data]
Client:::source
ControlPlane:::processing
DataPlane:::ingestion
Storage:::storage
AtRest:::serving
LocalDisk:::governance
ManagedServices:::governance
*Encryption coverage: TLS protects all network channels while AES-256 encrypts data at rest across storage, local disks, and managed services.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
KMS[Key Management Service] --> MasterKey[Master Key]
MasterKey --> DEK1[Data Encryption Key: Storage]
MasterKey --> DEK2[Data Encryption Key: Managed Services]
MasterKey --> DEK3[Data Encryption Key: Volumes]
DEK1 --> S3[S3/ADLS/GCS Objects]
DEK2 --> Notebooks[Notebooks & Secrets]
DEK3 --> EBS[EBS/Managed Disks]
KMS:::governance
MasterKey:::processing
DEK1:::storage
DEK2:::storage
DEK3:::storage
S3:::serving
Notebooks:::ingestion
EBS:::source
*Key hierarchy: KMS master keys generate data encryption keys (DEKs) for each encryption domain — storage, managed services, and volumes.*
Key Terms
Prerequisites and Setup
- Databricks workspace on any tier (basic encryption is always enabled)
- Premium tier for customer-managed keys and enhanced encryption controls
- Cloud provider KMS access for key creation (AWS KMS, Azure Key Vault, GCP Cloud KMS)
- Understanding of key rotation policies and compliance requirements
- IAM permissions for KMS key policy management
Step-by-Step Implementation
Configuration Reference
| Encryption Domain | Algorithm | Key Source | Configurable |
|---|---|---|---|
| Cloud storage (S3/ADLS/GCS) | AES-256 | Platform or CMK | Yes |
| Managed services (notebooks, secrets) | AES-256 | Platform or CMK | Yes (Premium) |
| Cluster volumes (EBS/Managed Disk) | AES-256 | Platform or CMK | Yes (Premium) |
| Network (API, UI, cluster comms) | TLS 1.2+ | Certificate-based | No (always on) |
| Spark shuffle | AES-256 | Per-session DEK | Yes (spark conf) |
| Spark RPC | AES-256 | Per-session DEK | Yes (spark conf) |
| DBFS root | AES-256 | Platform or CMK | Yes (Premium) |