Functions defined in common_functions are now available
Magic commands are special prefixes (%python, %sql, %scala, %r, %md, %run, %pip, %sh) that control cell behaviour in Databricks notebooks, letting you switch languages, run shell commands, install packages, and execute other notebooks — all within the same document. Mastering magic commands turns a notebook from a single-language script into a versatile multi-tool environment.
- Understand all available magic commands and when to use each
- Switch languages within a single notebook using
%python,%sql,%scala,%r - Install packages, run shell commands, and chain notebooks with
%pip,%sh,%run
Who this is for: Notebook users who want to leverage the full power of Databricks notebooks beyond basic cell execution.
Part of the Databricks Notebooks section of the Databricks tutorial series.
Architecture / Concept Overview: Functions defined in common_functions are now available
Each cell in a Databricks notebook executes in the context of the notebook's default language unless overridden by a magic command. Magic commands are processed by the notebook server before the code is sent to the compute layer. Some magic commands (%md, %run) are handled entirely by the notebook server, while others (%python, %sql) redirect execution to the appropriate language interpreter on the cluster.
*The notebook server parses magic commands and routes execution to the appropriate handler.*
*Magic commands fall into two categories: language switching and utility operations.*
*A typical workflow: install packages, write analysis code, validate with SQL, document with markdown.*
Key Terms
- Magic Command
- A cell-level directive prefixed with
%that controls how the cell is interpreted and executed. %run- Executes another notebook in the current notebook's context, sharing variables and state.
%pip- Installs Python packages from PyPI into the cluster's Python environment.
%sh- Executes shell commands on the cluster driver node.
%md- Renders the cell content as formatted Markdown.
%fs- Runs
dbutils.fsfilesystem commands for listing and managing files.
Prerequisites and Setup
- A Databricks notebook attached to compute
- Familiarity with the notebook's default language
- Understanding of which languages and tools your workload requires
- Cluster with appropriate permissions for shell access (
%sh)
Step-by-Step Implementation
Configuration Reference
| Magic Command | Purpose | Scope |
|---|---|---|
%python | Execute cell as Python | Cell |
%sql | Execute cell as SQL | Cell |
%scala | Execute cell as Scala | Cell |
%r | Execute cell as R | Cell |
%md | Render cell as Markdown | Cell |
%run | Execute another notebook | Session (shares state) |
%pip | Install Python packages | Cluster session |
%sh | Run shell commands | Driver node |
%fs | Filesystem operations | Driver node |
Monitoring, Cost, and Security Considerations
Monitoring
Each magic command cell shows execution time. %pip install adds time at the start of a session. %run execution time includes the entire child notebook execution. Monitor slow %sh or %pip cells that delay notebook startup.
Cost Optimisation
- Place %pip install commands at the top of the notebook so they run once per session, not repeatedly.
- Use %run to share utility code instead of duplicating functions across notebooks.
- Avoid %sh for heavy processing; use Spark for distributed work instead of single-node shell commands.
Security and Governance
- %sh runs on the driver node with the notebook owner's permissions; on Standard clusters, Lakeguard restricts shell access.
- %pip can install arbitrary packages, which may introduce security risks; use cluster policies or allowlisted package repositories.
- %run executes in the current session context, so the child notebook has access to all variables and credentials.
- %fs accesses storage paths; Unity Catalog governs which paths are accessible.
Common Pitfalls and Recommended Patterns
- Forgetting
%on the magic command:sql SELECT...is interpreted as Python code and fails. - Using
%runwith a variable path:%runrequires a literal path, not a variable — usedbutils.notebook.run()for dynamic paths. - Running
%pip installin the middle of the notebook: this restarts the Python interpreter, losing all variables. Put it at the top. - Using
%shfor heavy data processing: shell commands run only on the driver node, not distributed across workers. - Overusing magic commands: switching languages too frequently makes notebooks hard to follow.
- Not documenting with
%md: markdown cells make notebooks self-documenting and shareable.
Frequently Asked Questions
Does %run share variables with the current notebook?
Yes. %run executes the target notebook in the same Python/Scala/R context, so all variables, functions, and imports defined in the child notebook become available in the parent.
Can I use %pip on serverless compute?
Yes. %pip install works on serverless compute. Packages are installed at the start of the session and persist until the session ends.
Does %sh work on Standard clusters?
On Standard (shared) clusters with Lakeguard, %sh access is restricted to prevent security issues. Some shell commands may be blocked. On Dedicated clusters, %sh has full access.
Can I nest %run calls?
Yes. Notebook A can %run Notebook B, which %runs Notebook C. Be careful with circular references — they cause infinite loops.