Skip to content

Databricks

Databricks is a unified analytics platform built on Apache Spark. This sheet covers notebooks, clusters, jobs, Delta Lake integration, Unity Catalog, and collaborative features.

Table of Contents

Installation / Setup

# Databricks notebooks have Spark pre-configured
# No explicit SparkSession creation needed

# Access notebook utilities
dbutils.notebook.run("other_notebook")
dbutils.fs.ls("dbfs:/path/to/data")

# Access widgets (parameters)
dbutils.widgets.text("input_path", "dbfs:/data/input")
input_path = dbutils.widgets.get("input_path")

Core Syntax

DBFS (Databricks File System)

# List files
dbutils.fs.ls("dbfs:/data/")

# Read from DBFS
df = spark.read.parquet("dbfs:/data/events.parquet")
df = spark.read.csv("dbfs:/data/raw/", header=True, inferSchema=True)

# Write to DBFS
df.write.mode("overwrite").parquet("dbfs:/data/clean/")

# Copy files
dbutils.fs.cp("dbfs:/data/source/", "dbfs:/data/dest/")

# Delete files
dbutils.fs.rm("dbfs:/data/old/", recurse=True)

Magic Commands

# In Databricks notebooks, use %magic commands

# SQL magic
%sql
SELECT * FROM table_name LIMIT 10

# Python magic (in a SQL cell)
%python
df = spark.sql("SELECT * FROM table_name LIMIT 10")

# Shell commands
%sh
ls -la /dbfs/data/

# Markdown
%md
# This is a title
This is a description.

Widgets (Parameters)

# Create widgets
dbutils.widgets.text("input_path", "dbfs:/data/input")
dbutils.widgets.dropdown("year", "2026", ["2024", "2025", "2026"])
dbutils.widgets.combobox("output_format", "parquet", ["parquet", "csv", "delta"])

# Get widget values
input_path = dbutils.widgets.get("input_path")
year = dbutils.widgets.get("year")

# Remove widget
dbutils.widgets.remove("input_path")

Tables (Managed and External)

# Create table from DataFrame
df.write.saveAsTable("default.my_table")

# Create Delta table
df.write.format("delta").saveAsTable("default.events_delta")

# Create external table
df.write.format("delta").option("path", "dbfs:/data/events").saveAsTable("events")

# Read table
df = spark.table("default.my_table")

# SQL
spark.sql("DESCRIBE TABLE default.my_table")
spark.sql("SHOW TABLES IN default")

Common Operations

Delta Lake Operations

# Read Delta table
df = spark.read.format("delta").load("dbfs:/data/events")

# Upsert (MERGE)
from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "dbfs:/data/events")

delta_table.alias("target").merge(
    updates.alias("source"),
    "target.id = source.id"
).whenMatchedUpdateAll() \
 .whenNotMatchedInsertAll() \
 .execute()

# Time travel
df_version0 = spark.read.format("delta").option("versionAsOf", 0).load("dbfs:/data/events")
df_timestamp = spark.read.format("delta").option("timestampAsOf", "2026-01-01").load("dbfs:/data/events")

# Optimize
delta_table.optimize().executeCompaction()

# VACUUM (remove old files)
delta_table.vacuum()

SQL Analytics

# Run SQL queries
spark.sql("""
    WITH daily_stats AS (
        SELECT date, COUNT(*) as cnt, AVG(amount) as avg_amount
        FROM events
        GROUP BY date
    )
    SELECT * FROM daily_stats WHERE cnt > 100
""").show()

Cluster Management

# In notebooks, cluster is pre-configured
# Key spark configs:
spark.conf.set("spark.sql.shuffle.partitions", "200")
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")

Practical Examples

Example 1 — ETL pipeline

# Read raw data
raw_df = spark.read.csv("dbfs:/data/raw/", header=True, inferSchema=True)

# Transform
clean_df = raw_df \
    .dropDuplicates(["id"]) \
    .filter(raw_df.amount > 0) \
    .withColumn("date", F.to_date("timestamp")) \
    .withColumn("processed_at", F.current_timestamp())

# Write to Delta
clean_df.write \
    .format("delta") \
    .mode("overwrite") \
    .partitionBy("date") \
    .saveAsTable("events_clean")

Example 2 — Streaming with Delta

# Read stream
stream_df = spark.readStream \
    .format("delta") \
    .load("dbfs:/data/events/")

# Write stream
query = stream_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "dbfs:/checkpoints/events") \
    .toTable("events_streaming")

Example 3 — Dashboard query

%sql
-- Revenue by region for current year
SELECT
    region,
    SUM(amount) as total_revenue,
    COUNT(DISTINCT customer_id) as unique_customers,
    AVG(amount) as avg_order_value
FROM events
WHERE year = 2026
GROUP BY region
ORDER BY total_revenue DESC

Common Patterns

Pattern: notebook orchestration

# Pass data between notebooks
result = dbutils.notebook.run(
    "/path/to/child_notebook",
    arguments={"input_path": "dbfs:/data/input"}
)

# Use widgets for parameterization
input_path = dbutils.widgets.get("input_path")
output_path = dbutils.widgets.get("output_path")

Pattern: incremental processing

# Read last processed timestamp
last_processed = spark.sql("SELECT MAX(processed_at) FROM events").collect()[0][0]

# Process only new data
new_df = spark.read.format("delta").load("dbfs:/data/events") \
    .filter(F.col("timestamp") > last_processed)

# Write incrementally
new_df.write.format("delta").mode("append").saveAsTable("events_processed")

Pattern: medallion architecture

# Bronze: raw data
raw_df.write.format("delta").saveAsTable("bronze_events")

# Silver: cleaned data
silver_df = spark.table("bronze_events").dropDuplicates().filter(...)
silver_df.write.format("delta").mode("overwrite").saveAsTable("silver_events")

# Gold: aggregated data
gold_df = spark.table("silver_events").groupBy("date", "region").agg(...)
gold_df.write.format("delta").mode("overwrite").saveAsTable("gold_events")

Best Practices

  • Use Delta Lake — ACID transactions, time travel, schema evolution.
  • Partition by date — optimize queries and reduce scan.
  • Use widgets — make notebooks parameterizable for reuse.
  • Implement medallion architecture — bronze/silver/gold layers.
  • Optimize Delta tables — run OPTIMIZE and VACUUM regularly.
  • Use Unity Catalog — centralized data governance and access control.
  • Cache frequently accessed tables — spark.table("events").cache().

Common Mistakes

Mistake Fix
Not using Delta format Use Delta for all production tables
Over-partitioning Partition by low-cardinality columns (date, region)
Not optimizing Delta tables Schedule OPTIMIZE regularly
Hardcoding paths Use widgets and table names
Not using VACUUM Run VACUUM to clean old files
Ignoring Unity Catalog Use UC for data governance

Troubleshooting

Symptom Cause / Fix
Slow queries Check partition strategy; run OPTIMIZE
Out of memory Increase cluster size; repartition data
File not found Check DBFS paths; verify table exists
Streaming lag Check checkpoint; increase resources
Permission denied Check Unity Catalog permissions

Official Documentation

Last Updated

2026-09-06 — Initial version.