Databricks Notebooks Explained: From Code Cells to Delta Lake

In this post, I want to share my understanding of Databricks notebooks—from the basics to some of the inner workings and best practices I’ve picked up along the way. Think of this as a friendly chat about how to get the most out of this incredible tool.

What Exactly is a Databricks Notebook?

At its core, a Databricks notebook is a web-based interface that lets you write and run code in small, manageable chunks. For me, the real magic isn’t in the notebook itself, but in the powerful “engine” it’s connected to: a cluster. It’s a bit like a digital lab notebook where you can combine code, explanatory text, and visualizations all in one place. What makes it special is that it’s deeply integrated with the Apache Spark environment, giving you the power to process massive datasets with ease.

Let’s break down the main components and how they work together.

1. The Notebook Itself (Your Workspace)

This is the document you see on your screen. It’s made up of individual cells, and this is where you get to tell your data story.

• Code Cells: This is where you write your code, such as Python, SQL, or Scala. My favorite part is that you can write a few lines and run just that specific cell, instead of having to run an entire script. This makes testing and iterating so much faster!
• Markdown Cells: These are for your notes, explanations, and documentation. I’ve found that using headings, lists, and bold text to make your notebook easy for others (or your future self) to understand is a real lifesaver.

2. The Cluster (The Powerful Engine)

The cluster is a collection of computers that work together to run your code. You can think of it as a factory.

• The Driver: This is the cluster’s “brain.” It receives your code from the notebook, breaks it down into small tasks, and coordinates the work.
• The Workers: These are the cluster’s “hands.” They do the heavy lifting by performing the actual computations on the data. When my code needs to process a massive dataset, the driver tells all the workers to each process a small part of it at the same time.

Databricks uses a technology called Apache Spark to manage this process. Spark is what allows the driver and workers to communicate so efficiently, making it incredibly fast for big data tasks.

For more insights read – The Powerhouse Behind Your Data: A Comprehensive Guide to Clusters in Azure Data Engineering

Behind the Scenes: How a Cell is Executed

Have you ever wondered what happens when you hit “Run Cell”? It’s not just a simple script execution. Here’s a quick walkthrough of the steps:

1. You write code in a cell and click “Run”. For example, I might write a command, like SELECT * FROM my_big_data_table; in a SQL cell.
2. The Notebook sends the code to the Cluster. The notebook sends my command to the cluster’s driver.
3. The Driver plans the work. The driver takes my code and figures out the most efficient way to execute it. For my SELECT * command, it tells the workers: “Go find a piece of this data, read it, and send me the results.”
4. The Workers do the heavy lifting. The workers get their assigned tasks. They access the data stored in the cloud (like on AWS S3 or Google Cloud Storage), read their specific part of the table, and process it.
5. The Workers send results back to the Driver. Once a worker is done with its task, it sends its partial result back to the driver. The driver gathers all the results from all the workers.
6. The Driver sends the final result back to the Notebook. The driver assembles the complete result and sends it back to my notebook. The notebook then displays the output right below the cell I just ran.

This entire process is what makes Databricks so powerful. You get to interact with a simple notebook interface, while the cluster handles the complex, high-performance computing in the background, allowing you to work with datasets that would be impossible to handle on a single computer.

Understanding Databricks Notebook Command Types

A Databricks notebook is polyglot, which means it can use multiple languages. The notebook has a default language (e.g., Python), but I’ve found it incredibly useful to switch to another language for a specific cell using a “magic command.” A magic command is a special directive that starts with a percent sign (%).

Here are the most common magic commands I use:

• %python: Executes the cell’s code using the Python interpreter.
• %sql: Executes the cell’s code as a SQL query. This is extremely useful for a data analyst who wants to query data quickly without writing a Python script.
• %scala: Executes the cell’s code using the Scala interpreter.
• %r: Executes the cell’s code using the R interpreter.
• %sh: Runs a shell command on the cluster’s driver node. This is useful for things like installing libraries or checking file systems.
• %md: Renders the cell’s content as Markdown, which is how you create explanatory text and headings.

This flexibility allows data scientists and engineers to use the best language for each task, all within a single, collaborative document.

What is a DataFrame? (The “Super Spreadsheet”)

A DataFrame is the most important concept in Databricks and Spark. I always think of it as a super-powered, distributed spreadsheet or a table in memory.

• Like a Spreadsheet: It has a clear structure with named columns and rows, and it’s easy to read and understand.
• Super-Powered: Unlike a regular spreadsheet that sits on your local computer, a DataFrame is not stored in one place. It’s distributed across the many computers (workers) in your cluster. This is what allows it to handle massive datasets that would crash a single machine.

Is it immutable? Yes! When you apply a transformation to a DataFrame (e.g., adding a new column), you’re not changing the original DataFrame. Instead, you’re creating a new DataFrame with the transformation applied. This immutability is crucial for Spark’s fault tolerance and performance.

When you perform an operation on a DataFrame, like filtering rows or joining two tables, Spark doesn’t do the work immediately. It’s “lazy.” Instead, it builds a plan for the most efficient way to execute the entire sequence of operations. The real work only happens when you ask for a result, for example, by telling it to display() the data or write() it to a file. This intelligent planning is a huge part of what makes Databricks so fast.

What is Lazy Evaluation?

In the context of Databricks and Spark, “lazy” refers to a core concept called lazy evaluation. This means that when I perform transformations on a DataFrame (like filtering, adding a new column, or joining two DataFrames), Spark doesn’t actually execute the code immediately. Instead, it just records the transformation I want to perform and builds a plan for how to do it.

The real work—the heavy computation on the data—only happens when you trigger an action. An action is a command that requires a concrete result, such as:

• display(): To show a preview of the data in your notebook.
• count(): To get the number of rows.
• write(): To save the DataFrame to a file or a Delta table.
• collect(): To bring all the data from the distributed workers back to the driver node (which I use with caution on large datasets).

This lazy approach is a key reason for Spark’s high performance. By delaying execution, Spark can:

• Optimize the plan: It can look at all the transformations I want to do and figure out the most efficient order to perform them. For example, if I filter a massive table and then join it with a smaller one, Spark will likely perform the filter first to reduce the amount of data it has to shuffle between workers for the join.
• Avoid unnecessary work: If I apply several transformations but only ever ask for the row count, Spark will only execute the code needed to get that count, skipping any transformations that are not necessary for the final result.

This intelligent planning is what makes Databricks so fast and efficient for big data processing.

Does Databricks Have Its Own Storage?

No, Databricks does not have its own proprietary storage system. This is a point that’s often misunderstood, but it’s a key part of what makes the platform so flexible. Instead, it is built to integrate seamlessly with the major cloud storage services where your data already resides.

When you use Databricks, the data is typically stored in:

• Amazon S3 (for Databricks on AWS)
• Azure Blob Storage or Azure Data Lake Storage (ADLS) (for Databricks on Azure)
• Google Cloud Storage (GCS) (for Databricks on GCP)

Databricks acts as a powerful computation layer on top of this cloud storage. The clusters read data directly from these services, process it using Apache Spark, and then write the results back to the same cloud storage. This separation of compute (Databricks) and storage (the cloud provider) is a core part of its architecture, offering flexibility, scalability, and cost-effectiveness.

What is Delta Lake?

Delta Lake is a critical open-source storage layer that brings reliability to data lakes. It’s a technology that enhances the data files stored in your cloud storage (like on S3, ADLS, or GCS) by adding powerful features that are typically found in traditional data warehouses.

Yes, Delta Lake is very closely related to Databricks. Databricks originally created and open-sourced Delta Lake. It is the foundational technology for the “lakehouse” architecture that Databricks champions, which aims to combine the best aspects of data lakes and data warehouses.

Here’s how it works: When you write code to create a Delta table in Databricks, the platform automatically creates the necessary storage structure in your cloud data lake.

1. You write a command: You run a command in a Databricks notebook, such as a CREATE TABLE SQL statement or a PySpark DataFrame.write.format(“delta”).save() command.
2. Databricks handles the underlying storage: Databricks is configured to use a specific cloud storage location. When you create a Delta table, Databricks creates a new directory within that location.
3. Data files are written: As you write data into the new Delta table, Databricks writes the data as Parquet files into this new directory.
4. The transaction log is created: In addition to the Parquet data files, a special subdirectory named _delta_log is created. This is where the Delta Lake transaction log is stored. The transaction log is a series of JSON files that record every change made to the table, providing the ACID properties, versioning, and other key features of Delta Lake.

So, while you are interacting with a logical “table” in your notebook, Databricks is managing the physical files and directories in your cloud storage in the background. This is a huge benefit because it allows me to focus on working with my data at the table level without having to worry about file-level management.

Best Practices for a Happy Databricks Life

To make your experience smooth and efficient, here are a few personal tips that I’ve found really helpful:

• Modularize Your Code: Instead of one massive notebook, I like to break my logic into smaller, focused notebooks (e.g., one for data ingestion, one for transformations, one for modeling). You can then use %run to chain them together, which makes your workflow much cleaner.
• Use Markdown: I can’t stress this enough—document everything! Use Markdown cells to explain your logic, assumptions, and results. Your future self (and your teammates) will thank you.
• Start Small: When developing, I always work with a small sample of my data. This saves me time and resources while I’m still figuring things out.
• Manage Your Clusters: Clusters are the compute power, and they cost money. I always make sure to terminate my clusters when I’m done with my work to avoid unnecessary billing.
• Leverage Databricks Repos: For serious development, I use Databricks Repos to connect my notebooks to a Git repository. This is essential for version control and collaborative development.