resources/jobs.yml

    Who this is for:

    Architecture / Concept Overview: resources/jobs.yml

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED DEV[Developer]:::source YAML[databricks.yml]:::ingestion VALIDATE[bundle validate]:::processing DEPLOY[bundle deploy]:::storage WS[Databricks Workspace]:::serving GIT[Git Repository]:::governance DEV --> YAML --> VALIDATE --> DEPLOY --> WS YAML --> GIT

    *DABs follows a define-validate-deploy workflow, with YAML configurations stored in Git for version control.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% graph TD classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED BUNDLE[Bundle Configuration]:::source RESOURCES[Resources]:::processing TARGETS[Targets / Environments]:::governance ARTIFACTS[Artifacts]:::ingestion BUNDLE --> RESOURCES BUNDLE --> TARGETS BUNDLE --> ARTIFACTS RESOURCES --> JOBS_R[Jobs]:::serving RESOURCES --> PIPES[DLT Pipelines]:::serving RESOURCES --> MODELS[ML Models]:::serving RESOURCES --> EXP[Experiments]:::serving TARGETS --> DEV_T[development]:::storage TARGETS --> STG[staging]:::storage TARGETS --> PROD[production]:::storage

    *A bundle configuration defines resources, deployment targets, and artifacts in a single declarative structure.*

    %%{init: {"theme":"base","themeVariables":{"background":"#0B0E14","primaryTextColor":"#E0E6ED","lineColor":"#5D6470","darkMode":true,"primaryColor":"#2E4A4A","secondaryColor":"#374151","secondaryTextColor":"#E0E6ED","tertiaryColor":"#111827","tertiaryTextColor":"#E0E6ED","edgeLabelBackground":"#1f2937"}}}%% flowchart LR classDef source fill:#3F4B59,stroke:#9CA3AF,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef ingestion fill:#5A4B36,stroke:#C9A86B,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef processing fill:#535072,stroke:#8E82B4,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef storage fill:#2E4A4A,stroke:#5FAFA8,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef serving fill:#3D5550,stroke:#6BB7AA,stroke-width:2px,rx:8,ry:8,color:#E0E6ED classDef governance fill:#5A3F52,stroke:#C28BB0,stroke-width:2px,rx:8,ry:8,color:#E0E6ED PUSH[Git Push]:::source CI[CI Pipeline]:::ingestion VAL[bundle validate]:::processing DEPLOY_STG[Deploy to Staging]:::storage TEST[Integration Tests]:::serving DEPLOY_PROD[Deploy to Production]:::governance PUSH --> CI --> VAL --> DEPLOY_STG --> TEST --> DEPLOY_PROD

    *DABs integrates naturally into CI/CD pipelines with validation gates between environments.*

    Key Terms

    Prerequisites and Setup

    • Databricks CLI v0.200+ installed
    • A Databricks workspace with appropriate permissions
    • Authentication configured (PAT, OAuth, or Azure CLI)
    • Git repository for version control

    Step-by-Step Implementation

      Configuration Reference

      resources/jobs.yml configuration options
      FieldLocationDescription
      bundle.namedatabricks.ymlUnique bundle identifier
      targets.<name>databricks.ymlEnvironment-specific configuration
      targets.<name>.modedatabricks.ymldevelopment adds user prefix to resource names
      targets.<name>.run_asdatabricks.ymlIdentity for production job execution
      variables.<name>databricks.ymlParameterized values with per-target overrides
      resources.jobs.<name>YAML includesJob definitions
      resources.pipelines.<name>YAML includesDLT pipeline definitions
      artifacts.<name>databricks.ymlPython wheel or JAR build configurations

      Monitoring, Cost, and Security Considerations

      Common Pitfalls and Recommended Patterns

        Frequently Asked Questions