The Future of Data Systems

The final chapter is Kleppmann's synthesis and opinion: given all the tools and trade-offs of the previous eleven chapters, how should we build data systems? His answer centers on a single organizing idea — dataflow: treat an ordered log of immutable events as the source of truth, and build everything else (databases, indexes, caches, ML features) as derived data that is continuously, asynchronously computed from that log. The chapter ends, unusually for a technical book, with a serious discussion of ethics — because data systems now shape people's lives.

Data Integration

Because no single tool satisfies all access patterns, every nontrivial application ends up combining several — a transactional database, a search index, a cache, an analytics warehouse, maybe a recommendation system. The central practical problem is keeping them consistent with each other as data changes. The recurring answer in this book: pick a system of record (the authoritative source) and treat everything else as derived data computed from it, kept in sync by feeding the system of record's change log (via CDC or event sourcing) into each derived system in a defined order. A clear total order of writes is what makes this reliable; without it, concurrent updates to different systems race and diverge.

Batch and Stream, Together

Both batch and stream processing are tools for deriving data; they differ mainly in whether the input is bounded. The lambda architecture proposed running both — a batch layer recomputing accurate views from all history, and a speed layer giving low-latency approximate results — but maintaining two code paths is painful. Kleppmann argues the trend is toward unifying them: a powerful stream processor can also reprocess historical data (replay the log from the beginning), so you can rebuild a derived dataset from scratch when you change the code, without a separate batch system.

Unbundling the Database

A traditional database bundles many features: a storage engine, secondary indexes, replication, a query optimizer, materialized views. Kleppmann reframes a whole application as if it were a database turned inside out: rather than one monolith doing everything internally, you unbundle these features into separate, specialized systems, and use an event log as the integration point that wires them together. The index, the cache, and the materialized view become independent systems, each subscribing to the log and maintaining its own derived state asynchronously.

Dataflow: Application Code as Derivation Functions

In this view, application code becomes a set of derivation functions that react to events and produce derived outputs — much like a spreadsheet, where changing one cell automatically recomputes everything downstream. Stream processors maintain the derived state; when a new event arrives, the relevant views are updated. This shifts work from the read path to the write path: a materialized view does the expensive computation once when data changes (write time), so reads become cheap lookups. The trade-off is the classic one — more write-time work and storage in exchange for faster, simpler reads.

Aiming for Correctness

If derived data is maintained asynchronously and is only eventually consistent, how do we keep applications correct? Kleppmann is skeptical of relying on any single mechanism (even transactions) and pushes a more robust mindset.

The End-to-End Argument

Many correctness guarantees can only be achieved end-to-end, not by any single layer. The canonical example is deduplication: TCP retransmits, a stream processor retries, a transaction may be re-submitted — so the same operation can reach the database more than once. No lower layer can tell that two payment requests are "the same" if the user clicked twice. The fix is an end-to-end idempotency key: the client generates a unique request ID, and the operation is made idempotent against it, so duplicates anywhere in the stack are absorbed.

# client generates the ID once, sends it with every retry
POST /payments   request_id = "a1b2-c3d4"   amount = 50

server:
  if seen(request_id):        # duplicate from a retry anywhere
      return stored_result     # absorbed — no double charge
  else:
      result = charge(50)
      record(request_id, result)

  retries from TCP, the app, or a stream processor all collapse
  to one effect — correctness enforced END TO END, not per layer.

Timeliness vs Integrity

Kleppmann separates two things people lump together as "consistency." Timeliness means users see the system in an up-to-date state (no stale reads); a temporary violation is just lag, and usually self-corrects. Integrity means the absence of corruption — no lost or contradictory data, no money created or destroyed; a violation is permanent and serious. In practice, integrity matters far more than timeliness. Many applications happily tolerate a few seconds of inconsistency (the search index lags the database) as long as integrity is never violated. Log-based derived-data systems are appealing precisely because they make integrity easy: deterministic derivation from an immutable, ordered log can't lose or corrupt data, even if it lags.

Trust, but Verify

Finally, don't assume your systems are correct — software has bugs, disks corrupt data silently, and "it hasn't broken yet" is not a guarantee. Kleppmann advocates building systems that continuously audit themselves: check invariants, verify derived data against the source, and make it possible to detect (and recover from) corruption rather than discovering it years later. An immutable event log helps here too — you can always re-derive and compare.

Doing the Right Thing

The book closes on ethics, insisting that engineers can't treat the human consequences of data systems as someone else's problem. The concerns:

• Predictive analytics and bias. Algorithmic decisions about credit, employment, insurance, and policing can encode and amplify existing discrimination; biased training data produces biased outcomes, and feedback loops can make them self-reinforcing.
• Privacy and surveillance. Pervasive data collection amounts to surveillance; "consent" is often meaningless when using a service requires accepting opaque terms. Data hoarded today is a liability — it leaks, and it can be abused by future owners or governments.
• Data as a liability, not just an asset. The instinct to collect everything should be balanced against the responsibility (and risk) of holding it.

The call to action: treat respect for users — their dignity, privacy, and autonomy — as a first-class engineering requirement, not an afterthought.