Databricks 2: Delta Lake Internals

Every Databricks interview probes Delta Lake: not just “it adds ACID to Parquet”, but how, and what happens when two jobs write at once, why VACUUM can break time travel, or when to choose liquid clustering. This module covers the internals behind those questions.

1. Anatomy of a Delta table

s3://lake/sales/orders/
├── _delta_log/
│   ├── 00000000000000000000.json     ← commit 0: protocol, metadata (schema, partitioning), add files
│   ├── 00000000000000000001.json     ← commit 1: add/remove actions
│   ├── ...
│   ├── 00000000000000000010.checkpoint.parquet   ← snapshot of table state every N commits
│   └── _last_checkpoint
├── part-00000-…-c000.snappy.parquet  ← data files (immutable)
├── part-00001-….snappy.parquet
└── deletion_vector_….bin             ← (if deletion vectors are enabled)

• Data files are ordinary Parquet and are never modified in place. Updates write new files and mark old ones as removed.
• The transaction log (_delta_log) is an ordered sequence of JSON commits. Each commit contains actions: add (file + stats: row count, min/max/null counts per column), remove, metaData (schema, partition columns, configuration), protocol (reader/writer features), commitInfo (operation, user, metrics), and txn (streaming/idempotent writer progress).
• Checkpoints periodically (by default every 10 commits on recent versions; configurable) consolidate the log into Parquet, so readers don’t replay thousands of JSON files.
• Reading version N = load the latest checkpoint ≤ N, then apply subsequent JSON commits to get the set of active files. That set is the table snapshot.

ACID from a log:

• Atomicity: a commit is one log file write. Either it appears (all its adds/removes take effect) or it doesn’t.
• Consistency: schema enforcement and constraints are checked before commit.
• Isolation: readers always see a complete snapshot (a specific version); writers use optimistic concurrency.
• Durability: files and log live in durable object storage.

2. Optimistic concurrency control and conflicts

Writers don’t take locks. Each writer:

1. Reads the latest snapshot (version N) and records what it read.
2. Writes new data files.
3. Tries to commit version N+1 by atomically creating N+1.json (put-if-absent semantics, or a commit coordinator on stores lacking it).
4. If another writer already committed N+1, it checks for logical conflicts between its operation and the commits that won. If there are none, it retries as N+2. If there’s a conflict, it fails with a concurrency exception.

Reducing conflicts:

• Partition/cluster so concurrent jobs touch disjoint files, and add explicit predicates (MERGE … ON t.date = s.date AND …) so the conflict checker knows the scope.
• Enable row-level concurrency (Databricks, with deletion vectors and liquid clustering), which detects conflicts at row level instead of file level, so concurrent updates to different rows in the same file succeed.
• Serialise writers per table where possible (one writer job per table, many readers).
• Retry with backoff on conflict exceptions in jobs that can safely re-run (idempotent operations).

3. MERGE: what really happens

MERGE INTO silver.customers t
USING updates s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.op = 'D' THEN DELETE
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET *
WHEN NOT MATCHED AND s.op != 'D' THEN INSERT *

1. Find touched files: join the source with the target to find which target files contain matching keys (using file stats and data skipping where possible).
2. Rewrite: for copy-on-write, rewrite each touched file with updated/deleted rows applied, plus new files for inserts; commit remove for old files and add for new ones. With deletion vectors, deletes and updates mark old rows as deleted in a small bitmap file instead of rewriting the whole file (merge-on-read), so writes are much faster.

Performance rules:

• Deduplicate the source to one row per key first. Multiple source rows matching one target row is an error (non-deterministic update).
• Prune the target: add predicates on partition/clustering columns (AND t.event_date >= '2024-05-01'), otherwise MERGE may scan the whole table.
• Cluster the target on the merge key (liquid clustering), so matches touch few files.
• Enable deletion vectors and Photon; keep files reasonably sized.
• Watch the operation metrics in DESCRIBE HISTORY (numTargetFilesRemoved, numTargetRowsUpdated, executionTimeMs). If a small update rewrites thousands of files, the layout is wrong.

4. Time travel, VACUUM and retention

SELECT * FROM orders VERSION AS OF 120;
SELECT * FROM orders TIMESTAMP AS OF '2024-05-01T00:00:00Z';
RESTORE TABLE orders TO VERSION AS OF 120;     -- rollback after a bad write
DESCRIBE HISTORY orders;                        -- who did what, with metrics

• Time travel works as long as both the log entries and the data files for that version still exist.
• VACUUM physically deletes data files no longer referenced by the current version and older than the retention threshold (default 7 days, delta.deletedFileRetentionDuration). After vacuuming, older versions whose files were removed can’t be read.
• Log retention (delta.logRetentionDuration, default 30 days) controls how long commit history is kept.
• Don’t set retention below the longest-running query or streaming job lag: a reader using an old snapshot may lose files mid-read.
• GDPR: a DELETE alone leaves the deleted rows in old files until VACUUM removes them (and deletion vectors must be purged by rewriting files, e.g. REORG TABLE … APPLY (PURGE)). Erasure procedures must include the vacuum step.

5. File layout: OPTIMIZE, Z-order, liquid clustering

Small files (from streaming, frequent small batches or over-partitioning) slow reads and inflate metadata.

• OPTIMIZE compacts small files into larger ones (target ~1 GB by default; adaptive on Databricks).
• Optimized writes (shuffle before write to produce right-sized files) and auto compaction (compact after writes) prevent the problem at write time.

Data skipping: readers skip files whose min/max stats exclude the filter. Stats are collected on the first 32 columns by default (delta.dataSkippingNumIndexedCols, or delta.dataSkippingStatsColumns), so put filter columns early or configure stats columns.

Current guidance: for new tables, use liquid clustering on the columns most used in filters and joins (or CLUSTER BY AUTO to let the platform pick keys from query patterns), and skip partitioning except for very large tables with clear date-based lifecycle needs.

Predictive optimisation (Unity Catalog managed tables) automatically runs OPTIMIZE, VACUUM and statistics collection when beneficial, removing most manual maintenance jobs.

6. Other features interviewers ask about

Interview questions