Store source credentials in Databricks Secrets
Who this is for:
Architecture / Concept Overview: Store source credentials in Databricks Secrets
Lakeflow Connect abstracts away the complexity of source-system integration. Managed connectors are fully hosted by Databricks — you configure a source, map tables, and Databricks handles scheduling, schema evolution, and exactly-once delivery. Standard connectors leverage Auto Loader, JDBC, or partner integrations for sources not yet covered by managed connectors.
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
flowchart LR
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
PG[PostgreSQL]:::source --> MC[Managed Connector]:::ingestion
SF[Salesforce]:::source --> MC
MY[MySQL]:::source --> MC
MC --> |CDC Replication| DL[Delta Lake Bronze]:::storage
S3[S3 / ADLS / GCS]:::source --> AL[Auto Loader]:::ingestion
JDBC[Oracle / SQL Server]:::source --> JC[JDBC Connector]:::ingestion
AL --> DL
JC --> DL
*Managed connectors handle CDC replication automatically, while standard connectors give flexibility for file-based and JDBC sources.*
%%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%%
graph TD
classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED
LC[Lakeflow Connect]:::ingestion
LC --> MANAGED[Managed Connectors]:::ingestion
LC --> STANDARD[Standard Connectors]:::source
MANAGED --> M1[Salesforce]:::ingestion
MANAGED --> M2[PostgreSQL]:::ingestion
MANAGED --> M3[MySQL]:::ingestion
MANAGED --> M4[SQL Server]:::ingestion
STANDARD --> S1[Auto Loader]:::source
STANDARD --> S2[JDBC / ODBC]:::source
STANDARD --> S3[Partner Connect]:::source
*Connector taxonomy: managed vs standard, with representative source types.*
Key Terms
Prerequisites and Setup
- Unity Catalog enabled on the workspace.
- A catalog and schema where ingested tables will land.
- For managed connectors: source database credentials stored in Databricks Secrets.
- Network access from the Databricks control plane to the source (public endpoint, Private Link, or VPN).
Step-by-Step Implementation
Configuration Reference
| Parameter | Scope | Description | Default |
|---|---|---|---|
connection_name | Managed Connector | Unity Catalog connection to the source system | Required |
gateway_size | Managed Connector | Compute size for the ingestion gateway (SMALL, MEDIUM, LARGE) | SMALL |
snapshot_enabled | Managed Connector | Whether to perform an initial full snapshot | true |
cloudFiles.format | Auto Loader | File format (csv, json, parquet, avro, orc) | Required |
cloudFiles.schemaEvolutionMode | Auto Loader | Schema evolution strategy | addNewColumns |
cloudFiles.useNotifications | Auto Loader | Use cloud-native event notifications for file discovery | false |