Reliable Data Engineering is written by a senior data engineer for people who build data platforms. Read in-depth articles on Databricks, Spark, dbt, SQL migration and AI for data, then prepare for senior interviews with a free, hands-on study platform.
đź‘‹ Welcome to Reliable Data Engineering
Glad you're here! Sign up to bookmark articles, highlight passages, leave notes other readers can see, and pick up where you left off.
Welcome back,
In 12 days, a hosted AI coding console leaked API keys and a tampered Claude Code extension put a backdoor on developer machines. Neither attack was aimed at your warehouse. Both show how close it sits.
dbt snapshots record history in the order files arrive. When one lands late, every run stays green while your SCD2 history goes wrong. Here's how to check for it, and how dbt-iterative-snapshot fixes it automatically.
Databricks shipped a SQL function that makes structured decisions on unstructured data in sub-second latency. It returns probabilities, not paragraphs. Here's why that distinction matters more than it sounds.
Both platforms now ship agents as governed objects, managed MCP servers, AI gateways and SQL extraction functions. The work moving to data engineers is context, permissions and cost control.
What Cursor, Claude Code, and Perplexity actually built -- and why the model was never the real product.
How agentic platforms are quietly making the pipeline-builder's job description obsolete -- and what survives the transition.
Org charts are lying to you. The job title says 'AI Engineer' but the work varies wildly. Here's the honest picture of who actually builds and ships agentic systems in 2026.
AI can write the code. The problem is the person approving it -- working with an atrophying skill set, a backlog that doubles every quarter, and a mounting suspicion they can't tell if what they're merging is safe.
Most people blame Claude when they hit the wall. The real culprit is how they're using it. Tokens aren't counted per message — they're counted per token.
Hardware-specific guide to running Gemma 4 locally: which model fits your Mac/GPU, Ollama vs MLX, Apple Silicon memory tuning, real tok/s numbers, and troubleshooting the things that actually break.
80% of RAG failures trace to chunking decisions. Your retrieval returns results, your LLM generates answers, and your users get confident nonsense.
DuckDB has an MCP server. Databricks bakes LLM functions into ETL. AI agents are becoming the most demanding query clients your infrastructure has ever seen.
DuckDB's new open table format ditches files-on-files metadata and replaces it with a SQL database. One query resolves everything.
The ten papers that actually come up in AI engineering interviews — what each one says, why it matters, and the exact questions interviewers ask about them.
PageIndex turns documents into navigable trees. An LLM reasons through the hierarchy to find answers — no embeddings, no similarity search, just structured retrieval.
Context Hub, Code Review Graph, and the emerging discipline of giving AI agents less to make them smarter.
Describe the task. Connect your data. Let the platform handle the rest. That is the promise of Agent Bricks — and for a specific, important set of data engineering problems, it is actually delivering on it.
We spent two years building what marketing calls the future of data architecture. It is a database with more vendors.
Why the open-source data world is buzzing about CMU’s new columnar format — and why Parquet’s decade-long reign might actually be ending
A controversial thesis on why ETL, data warehouses, and the entire modern data stack are about to become as obsolete as floppy disks — and what’s coming to replace them
Anthropic's agentic CLI reads your codebase, edits files, runs commands, and commits changes. No IDE plugin. No web UI. Just a terminal that understands what you're building.
Feeding 200K-character SQL files to an LLM is expensive and unreliable. We built TOON — a compact AST notation that gives the model structural awareness at a fraction of the token cost.
Anthropic launched its first official technical credential -- a proctored, architecture-level exam that separates people who use AI from people who build production systems with it.
