Building Blocks of Data Systems

Every data system design is assembled from roughly ten building blocks. For each one, know what it’s for, how it works inside, the knobs, and when NOT to use it.

Contents: Message log (Kafka) · Partitioning · Stream processors · Batch engines · Storage & table formats · Serving stores · Caches · Orchestrators · Schema registry · Probabilistic structures

1. The message log (Kafka)

Mental model

A Kafka topic is an append-only, partitioned, replicated log. Producers append; consumers read at their own offset. Data isn’t deleted when it’s read. It’s deleted by retention (time/size) or compaction.

flowchart LR
    P1[Producer A] -->|key=user_1| T0
    P2[Producer B] -->|key=user_2| T1
    P2 -->|key=user_3| T2
    subgraph Topic["Topic: clicks (3 partitions, RF=3)"]
        T0["P0: 0 1 2 3 4 5 →"]
        T1["P1: 0 1 2 3 →"]
        T2["P2: 0 1 2 3 4 →"]
    end
    subgraph CG1["Consumer group: flink-agg"]
        C1[consumer 1]
        C2[consumer 2]
    end
    subgraph CG2["Consumer group: lake-sink"]
        C3[consumer 1]
    end
    T0 --> C1
    T1 --> C2
    T2 --> C2
    T0 --> C3
    T1 --> C3
    T2 --> C3

Key facts interviewers probe

Kafka vs alternatives

When NOT Kafka: low volume (< a few hundred events/s) where a DB table + CDC or SQS is simpler; pure batch file drops (land in object storage + AutoLoader).

2. Partitioning strategies

Partitioning shows up in Kafka topics, Spark shuffles, table layouts and serving stores. Same trade-off everywhere: locality vs balance.

Hot-key mitigation (classic deep dive)

flowchart LR
    E[Events<br/>key = campaign_id] --> S{Hot key?}
    S -->|no| H1[hash campaign_id]
    S -->|yes| SALT[append salt 0..N-1<br/>campaign_id#7]
    H1 --> A1[Partial aggregate per key]
    SALT --> A1
    A1 --> M[Second stage:<br/>strip salt, merge partials]
    M --> OUT[(Final per campaign)]

1. Detect: per-partition lag/throughput skew; top-K key metrics.
2. Salt the hot keys (or all keys) with a random suffix → partial aggregates → merge in a second stage. Works because SUM/COUNT/MIN/MAX/HLL are mergeable.
3. Local pre-aggregation (combiner) before shuffle: Spark does this automatically for reduceByKey/groupBy().agg() partial aggregation.
4. Split hot keys onto dedicated partitions/consumers.
5. In joins: broadcast the small side, or AQE skew join (Spark splits skewed partitions automatically).

3. Stream processors

Talking point: “Our freshness SLA is under a minute and we’re a Databricks shop, so Structured Streaming with Delta sinks is the pragmatic choice. If we needed sub-100 ms fraud scoring with per-user timers, I’d pick Flink.”

Core streaming concepts (detail in streaming deep dive)

• Event time vs processing time: always aggregate on event time.
• Watermark = “I believe I’ve seen all events up to time T”. Lets the engine finalise windows and drop state.
• Windows: tumbling, sliding/hopping, session.
• Output modes: append (only finalised rows), update (changed rows), complete (whole result).
• Triggers: processing-time interval, available-now (batch-like incremental), continuous.

4. Batch engines

Key batch design rules:

1. Idempotent partitions: each run overwrites exactly one logical partition (INSERT OVERWRITE ... PARTITION (dt='2026-10-01') or MERGE / replaceWhere).
2. Incremental by default: process only new data (watermark column, Delta CDF, AutoLoader), with a full-refresh escape hatch.
3. Deterministic: no now() inside transformations. Pass the logical date in as a parameter so backfills reproduce history.

5. Storage layer

flowchart TB
    subgraph Catalog["Catalog (Unity / Glue / Polaris / Hive Metastore)"]
        CT[table name → metadata location, permissions]
    end
    subgraph Table["Open table format (Delta / Iceberg / Hudi)"]
        LOG[Transaction log / metadata tree<br/>versions, schema, file list, stats]
    end
    subgraph Files["Object storage (S3 / ADLS / GCS)"]
        F1[part-0001.parquet]
        F2[part-0002.parquet]
        F3[part-0003.parquet]
    end
    CT --> LOG
    LOG --> F1
    LOG --> F2
    LOG --> F3

• Object storage: cheap, infinitely scalable, 11 nines durable, but no transactions, no in-place updates, and listing is slow.
• Parquet: columnar, compressed, with min/max stats per row group → column pruning + predicate pushdown.
• Table format adds: ACID commits, time travel, schema evolution, MERGE/UPDATE/DELETE, file-level stats for data skipping, concurrent writers (optimistic concurrency).
• Deep dive: Storage & table formats.

File-size rule: target 128 MB–1 GB files. Thousands of tiny files are the most common lakehouse performance bug (metadata overhead, task overhead, S3 request costs) → compaction (OPTIMIZE), optimized writes, auto-compaction.

6. Serving stores

The access pattern picks the store:

flowchart LR
    Q{Query pattern?}
    Q -->|"key → value, < 10 ms"| KV[Redis / DynamoDB]
    Q -->|"aggregations on fresh data,<br/>user-facing, 1000s QPS"| OLAP[Pinot / Druid / ClickHouse]
    Q -->|"ad-hoc SQL, joins,<br/>analysts"| WH[Warehouse / Lakehouse SQL]
    Q -->|"nearest neighbours"| VEC[Vector index]
    Q -->|"full text"| ES[Elasticsearch]

Senior nuance: you often need two serving stores: raw history in the lakehouse for flexibility, and pre-aggregated hot data in an OLAP/KV store for latency. Say explicitly which one is the source of truth (usually the lakehouse) and that serving stores can be rebuilt from it.

7. Caches

• Cache-aside (app reads cache, falls back to source, populates cache): default.
• Write-through: write cache + source synchronously (consistency, slower writes).
• TTL for freshness; key design e.g. metric:{campaign}:{minute}.
• In data systems, a cache is mostly used for serving pre-computed aggregates/features, and for dimension lookups inside streaming jobs (broadcast state or async lookups to Redis) to avoid hammering the OLTP DB.
• Pitfalls: thundering herd on expiry (jitter TTLs, request coalescing), cache inconsistency on backfill (invalidate or version keys).

8. Orchestrators

Design rules: tasks idempotent, parameterised by logical date, retries with backoff, SLAs/alerts on freshness not just failure, data-aware triggers (run when upstream table updates) over cron chains. More in Orchestration & backfills.

9. Schema registry & contracts

• Central store of Avro/Protobuf/JSON schemas, versioned per subject (topic).
• Compatibility modes:
  • BACKWARD (default): new consumers can read old data → you can add fields with defaults or delete fields. Upgrade consumers first.
  • FORWARD: old consumers can read new data → add fields, delete optional fields. Upgrade producers first.
  • FULL: both.
• Producer serialization fails fast on incompatible change → bad schemas never enter the log.
• Data contract = schema + semantics + SLAs + ownership + quality rules, agreed between producer and consumer and enforced in CI.

10. Probabilistic data structures

Use in design: “Real-time unique visitors per page per minute: I’ll store HLL sketches per minute; the hourly/daily numbers are unions of the minute sketches, so I never re-scan raw data. Finance’s exact numbers come from batch.”

SQL examples: approx_count_distinct(user_id), percentile_approx(latency, 0.95), Databricks hll_sketch_agg / hll_union_agg.

Cheat table: “If they say X, reach for Y”