Delta Lake and the Databricks Platform

A practical map of the features senior Databricks interviews probe, with the “why” behind each.

1. Delta Lake features you must know

Incremental processing with CDF

changes = (spark.readStream
           .option("readChangeFeed", "true")
           .option("startingVersion", 120)
           .table("silver.orders"))
# rows carry _change_type in (insert, update_preimage, update_postimage, delete), _commit_version

2. Structured Streaming on Delta

flowchart LR
    SRC[Kafka / AutoLoader / Delta table] --> Q[Streaming query<br/>micro-batches]
    Q -->|checkpoint: offsets + state| CK[(Checkpoint location)]
    Q -->|idempotent commit per batch| SINK[(Delta table)]
    SINK --> DOWN[Downstream stream / batch readers]

3. Lakeflow Declarative Pipelines (formerly Delta Live Tables)

Declarative pipelines: you declare tables/views and their queries; the platform handles orchestration, dependencies, retries, checkpoints, scaling and data quality.

CREATE OR REFRESH STREAMING TABLE bronze_orders
AS SELECT * FROM STREAM read_files('/landing/orders', format => 'json');

CREATE OR REFRESH STREAMING TABLE silver_orders (
  CONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW,
  CONSTRAINT has_id       EXPECT (order_id IS NOT NULL) ON VIOLATION FAIL UPDATE
) AS SELECT * FROM STREAM bronze_orders;

CREATE OR REFRESH MATERIALIZED VIEW gold_daily_revenue
AS SELECT order_date, SUM(amount) AS revenue FROM silver_orders GROUP BY order_date;

• Streaming tables (append/incremental) vs materialized views (recomputed incrementally where possible).
• AUTO CDC / APPLY CHANGES for SCD1/SCD2 from CDC feeds (handles out-of-order events with SEQUENCE BY).
• Expectations: warn / drop / fail, with metrics in the event log.
• When not to use: highly custom orchestration, cross-system workflows, or teams standardised on dbt.

4. Unity Catalog

flowchart TB
    MS[Metastore per region] --> C1[Catalog: sales_prod]
    MS --> C2[Catalog: finance_prod]
    C1 --> S1[Schema: silver]
    C1 --> S2[Schema: gold]
    S2 --> T[Tables, views, volumes,<br/>functions, models]
    UC["Governance: GRANTs, tags, ABAC policies,<br/>row filters, column masks, lineage,<br/>audit logs, system tables"] -.-> MS

• Three-level namespace catalog.schema.object; managed vs external tables; volumes for files.
• Privileges inherited downward (USE CATALOG, USE SCHEMA, SELECT, MODIFY).
• Row filters and column masks as SQL UDFs; tag-based ABAC policies for scale.
• Automatic lineage (table and column level) and system tables (audit, billing, query history, lineage) queryable with SQL.
• Delta Sharing for sharing data across orgs/platforms without copies.
• Details: Governance, security, privacy.

5. Compute and cost

Cost levers: auto-termination, autoscaling, spot workers, right-sized node types, serverless for spiky loads, predictive optimization (automatic OPTIMIZE/VACUUM), cluster policies, tagging + system billing tables for chargeback.

6. Workflows (Lakeflow Jobs)

Multi-task jobs (notebooks, Python wheels, SQL, dbt, pipelines), task dependencies, retries, repair runs (re-run only failed tasks), parameters, file-arrival and table-update triggers, and Asset Bundles (databricks.yml) for CI/CD of jobs, pipelines and their configuration as code.

7. Interview questions