Secret Management: Storing and Using Credentials Securely
Who this is for:
Architecture / Concept Overview: Secret Management: Storing and Using Credentials Securely
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Admin[Secret Admin] --> Scope[Secret Scope]
Scope --> Secret1[Database Password]
Scope --> Secret2[API Key]
Scope --> Secret3[Connection String]
Secret1 --> ACL1[ACL: data-engineers]
Secret2 --> ACL2[ACL: ml-team]
Secret3 --> ACL3[ACL: etl-operators]
Admin:::governance
Scope:::processing
Secret1:::storage
Secret2:::storage
Secret3:::storage
ACL1:::serving
ACL2:::serving
ACL3:::ingestion
*Secret scope hierarchy: admins create scopes containing multiple secrets, each with independent access controls.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
Notebook[Notebook Code] --> Utils[dbutils.secrets.get]
Utils --> AuthCheck[Permission Check]
AuthCheck --> Decrypt[Decrypt from Vault]
Decrypt --> Value[Secret Value in Memory]
Value --> Redacted[REDACTED in Logs]
Notebook:::source
Utils:::ingestion
AuthCheck:::governance
Decrypt:::processing
Value:::storage
Redacted:::serving
*Secret retrieval flow: code requests a secret, Databricks verifies permissions, decrypts the value, and redacts it from all output channels.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
DatabricksScope[Databricks-Backed Scope] --> InternalVault[Internal Encrypted Store]
KeyVaultScope[Key Vault-Backed Scope] --> AzureKV[Azure Key Vault]
KeyVaultScope --> AWSSM[AWS Secrets Manager]
InternalVault --> Encrypted[AES-256 Encrypted]
AzureKV --> HSM[HSM-Protected]
AWSSM --> Rotation[Auto-Rotation]
DatabricksScope:::processing
KeyVaultScope:::processing
InternalVault:::storage
AzureKV:::governance
AWSSM:::governance
Encrypted:::serving
HSM:::source
Rotation:::ingestion
*Secret backend options: Databricks-backed scopes use internal encryption, while Key Vault-backed scopes integrate with cloud-native secret stores.*
Key Terms
Prerequisites and Setup
- Databricks workspace (any tier supports Databricks-backed scopes)
- Premium tier for secret ACLs and Key Vault-backed scopes
- Databricks CLI or workspace admin access for scope creation
- For Key Vault-backed scopes: Azure Key Vault or AWS Secrets Manager with appropriate IAM
- Service principal or user credentials for initial secret population
Step-by-Step Implementation
Configuration Reference
| Setting | Databricks-Backed | Key Vault-Backed |
|---|---|---|
| Storage Location | Databricks internal | External vault |
| Encryption | AES-256 (platform-managed) | HSM-protected (vault-managed) |
| Max Secret Size | 128 KB | Vault-dependent |
| Access Control | Databricks ACLs | Databricks ACLs + Vault policies |
| Auto-Rotation | Manual | Vault-native rotation |
| Audit | Databricks audit logs | Vault + Databricks audit logs |
| Cross-Workspace | No (workspace-scoped) | Yes (via shared vault) |
| Tier Required | All | Premium+ |