Databricks 5: DevOps, Security, Serving and AI

Senior Databricks roles are judged on how you ship and operate the platform: code promotion, secure networking, serving data to BI and applications, and supporting ML/AI workloads. This module covers what to say about each.

1. Code as the source of truth: Git folders and Asset Bundles

• Git folders (formerly Repos) connect workspace folders to Git for development: branches, commits and PRs from the UI.
• Databricks Asset Bundles (DABs) describe a project’s jobs, pipelines, dashboards, models and their per-environment settings in YAML, deployed with the Databricks CLI:

# databricks.yml
bundle:
  name: sales_pipelines
targets:
  dev:
    mode: development            # prefixes resources with the developer's name, pauses schedules
    workspace: { host: https://dev.cloud.databricks.com }
  prod:
    mode: production
    workspace: { host: https://prod.cloud.databricks.com }
    run_as: { service_principal_name: sp-sales-prod }
resources:
  jobs:
    daily_sales:
      name: daily_sales
      tasks:
        - task_key: ingest
          pipeline_task: { pipeline_id: ${resources.pipelines.sales_ingest.id} }
        - task_key: gold
          depends_on: [{ task_key: ingest }]
          notebook_task: { notebook_path: ./src/gold.py, base_parameters: { catalog: "${var.catalog}" } }

CI/CD flow:

1. PR → unit tests (pytest with local Spark or Databricks Connect), linting, databricks bundle validate.
2. Merge → bundle deploy -t staging → integration tests on staging data.
3. Approval → bundle deploy -t prod running as a service principal.

Infrastructure (workspaces, metastores, networking, UC grants) is usually managed with Terraform; application resources with bundles. No manual changes in prod.

Environment isolation: separate workspaces per environment bound to environment catalogs (dev, prod), parameterised code (catalog as a job parameter), and service principals with least privilege per environment.

2. Security and networking

Interview framing: “Defence in depth: identity (SSO, SCIM, service principals), network (no public IPs, private link, egress control), data (UC permissions, ABAC, encryption with customer keys), and monitoring (audit logs in system tables, alerts).“

3. Serving data: SQL warehouses and BI

• SQL warehouses (serverless recommended) run SQL with Photon, a result cache, disk cache, intelligent workload management and autoscaling for concurrency. Sizing: the T-shirt size sets cluster size per query; min/max clusters set concurrency scaling.
• Query profile shows operator time, Photon coverage, spill, and rows/bytes per operator, which you use to tune slow dashboards.
• Materialized views and streaming tables in SQL keep aggregates fresh for BI without separate pipelines.
• AI/BI dashboards and Genie (natural-language exploration grounded in UC metadata and curated instructions); metric views define governed business metrics in UC (a semantic layer).
• External BI tools (Power BI, Tableau) connect via partner connectors; use Unity Catalog permissions end to end with OAuth/SSO pass-through where possible.
• Databricks Apps host internal data apps; Lakebase provides a managed Postgres for low-latency operational serving next to the lakehouse (useful for app backends and online features).

4. ML and AI on Databricks (what data engineers should know)

• MLflow: experiment tracking, model packaging, and the model registry in Unity Catalog (models as governed objects with versions and aliases like @champion).
• Feature engineering in Unity Catalog: feature tables are Delta tables with primary keys (and timestamp keys for point-in-time lookups); training sets are built with point-in-time joins; online serving through online tables/stores.
• Model Serving: REST endpoints for custom models, foundation models and external models (via an AI gateway with rate limits, guardrails and usage tracking); inference tables log requests/responses to Delta for monitoring.
• Vector Search: vector indexes synced from Delta tables (delta sync) for RAG retrieval, governed by UC.
• Agent framework and evaluation: building and evaluating RAG/agent applications, with tracing in MLflow.
• Data engineer’s role: reliable ingestion and chunking of documents into Delta/volumes, embedding pipelines, keeping vector indexes in sync, feature pipelines, inference table processing, and governance of all of it. See AI data architecture.

5. Operating the platform

• Observability: system tables (jobs, pipelines, billing, audit, query history) feeding dashboards and alerts; pipeline event logs; Lakehouse Monitoring for data quality and drift; SQL alerts for business rules.
• Cost management: budgets and tags, cluster policies, serverless where it fits, warehouse auto-stop, predictive optimisation, and monthly reviews of the top spenders (see module 1).
• Reliability: job retries and repair runs, idempotent tasks, table versioning (RESTORE) for rollback, deep clones or replication for DR, and tested runbooks.
• Upgrades: pin LTS runtimes in production, test new LTS versions in staging, and track deprecations (e.g. DBFS mounts, legacy Hive metastore, no-isolation clusters).

Interview questions