This should fail (exceeds max_workers)
Cluster policies are JSON-based rule sets that constrain what users can configure when creating clusters, enforcing standards for instance types, autoscaling limits, auto-termination, access modes, and Spark settings. Policies prevent cost overruns, security misconfigurations, and configuration drift by limiting choices to approved values. Every workspace should have at least one policy applied to each team or use case.
- Understand how cluster policies restrict and default cluster configurations
- Create policies that enforce cost, security, and operational standards
- Assign policies to users and groups to govern compute provisioning
Who this is for: Platform administrators and workspace managers who govern how teams provision and use compute resources.
Part of the Databricks Compute section of the Databricks tutorial series.
Architecture / Concept Overview: This should fail (exceeds max_workers)
A cluster policy defines allowable ranges, fixed values, or default values for every cluster configuration parameter. When a user creates a cluster under a policy, the UI and API enforce the constraints, hiding disallowed options and pre-filling defaults. Policies are assigned to users or groups via workspace permissions, and a user can only create clusters under policies they have access to.
*Admins define policies, users create clusters within those constraints, and only validated configurations are allowed.*
Policies use constraint types to control each parameter differently.
*Policies support fixed, range, allowlist, blocklist, and default constraint types.*
*Different teams are assigned different policies tailored to their workload requirements and cost constraints.*
Key Terms
- Cluster Policy
- A JSON rule set that constrains cluster configuration options available to users.
- Fixed Constraint
- A policy rule that locks a parameter to a specific value the user cannot change.
- Range Constraint
- A policy rule that allows a parameter value within a minimum and maximum bound.
- Allowlist Constraint
- A policy rule that limits a parameter to a set of approved values.
- Policy Permission
- A workspace ACL that grants a user or group the right to use a specific policy.
Prerequisites and Setup
- Workspace admin permissions to create and manage cluster policies
- Understanding of your teams' compute requirements (instance types, scaling, budgets)
- A list of security and cost constraints to enforce
- Unity Catalog enabled for governance integration
Step-by-Step Implementation
Configuration Reference
| Constraint Type | Syntax | Use Case |
|---|---|---|
fixed | {"type": "fixed", "value": "..."} | Lock a value (e.g., security mode) |
range | {"type": "range", "minValue": N, "maxValue": M} | Limit numeric values (e.g., workers) |
allowlist | {"type": "allowlist", "values": [...]} | Restrict to approved options (e.g., instance types) |
blocklist | {"type": "blocklist", "values": [...]} | Forbid specific options |
regex | {"type": "regex", "pattern": "..."} | Match a string pattern (e.g., runtime version) |
unlimited | {"type": "unlimited"} | Allow any value (remove inherited constraint) |
Monitoring, Cost, and Security Considerations
Monitoring
Audit cluster creation events to verify all clusters are created under a policy. Alert on clusters created by admins without a policy, which bypass governance.
Cost Optimisation
- Policies are the primary mechanism for preventing runaway compute costs.
- Cap autoscale.max_workers, enforce autotermination_minutes, and restrict expensive instance types.
- Use custom_tags to enable cost allocation and chargeback per team.
Security and Governance
- Fix data_security_mode to USER_ISOLATION to enforce Lakeguard on all shared clusters.
- Block init scripts from untrusted locations using allowlists.
- Prevent users from disabling logging or audit controls via fixed Spark configuration values.
- Assign policies through groups, not individual users, for scalable access management.
Common Pitfalls and Recommended Patterns
- Not creating any policies: without policies, every user can create unrestricted clusters.
- Making policies too restrictive: users will request exemptions or find workarounds; balance control with flexibility.
- Forgetting to assign policies to groups: a policy without permissions is unused.
- Using only fixed constraints: combine fixed (security), range (cost), and allowlist (instance types) for practical governance.
- Not versioning policy definitions: track policy JSON in source control for audit trails.
- Allowing admin bypass without monitoring: admins can create clusters without policies, so audit this regularly.
Frequently Asked Questions
Can a user have access to multiple policies?
Yes. A user sees all policies they have CAN_USE permission on and selects one when creating a cluster.
Can I change a policy on a running cluster?
No. The policy is evaluated at cluster creation time. To apply a new policy, terminate the cluster and create a new one under the updated policy.
Do policies apply to job clusters?
Yes. You can set a policy_id in a job's new_cluster configuration. This is recommended for production workloads to ensure consistency.
What happens if I update a policy definition?
Existing running clusters are not affected. New clusters created under the updated policy will use the new constraints.