# Table of Content LLMS index: [llms.txt](/llms.txt) --- ![](/title.jpg) ## [Preface](/preface) - [Who Should Read This Book?](/preface#who-should-read-this-book) - [Scope of This Book](/preface#scope-of-this-book) - [Outline of This Book](/preface#outline-of-this-book) - [References and Further Reading](/preface#references-and-further-reading) - [O'Reilly Safari](/preface#oreilly-safari) - [How to Contact Us](/preface#how-to-contact-us) - [Acknowledgments](/preface#acknowledgments) ## [1. Trade-offs in Data Systems Architecture](/ch1) - [Operational versus Analytical Systems](/ch1#sec_introduction_analytics) - [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - [Data Warehousing](/ch1#sec_introduction_dwh) - [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - [Beyond the data lake](/ch1#beyond-the-data-lake) - [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - [Cloud versus Self-Hosting](/ch1#sec_introduction_cloud) - [Pros and Cons of Cloud Services](/ch1#sec_introduction_cloud_tradeoffs) - [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - [Layering of cloud services](/ch1#layering-of-cloud-services) - [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - [Operations in the Cloud Era](/ch1#sec_introduction_operations) - [Distributed versus Single-Node Systems](/ch1#sec_introduction_distributed) - [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems) - [Microservices and Serverless](/ch1#sec_introduction_microservices) - [Cloud Computing versus Supercomputing](/ch1#id17) - [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - [Summary](/ch1#summary) - [References](/ch1#references) ## [2. Defining Nonfunctional Requirements](/ch2) - [Case Study: Social Network Home Timelines](/ch2#sec_introduction_twitter) - [Representing Users, Posts, and Follows](/ch2#id20) - [Materializing and Updating Timelines](/ch2#sec_introduction_materializing) - [Describing Performance](/ch2#sec_introduction_percentiles) - [Latency and Response Time](/ch2#id23) - [Average, Median, and Percentiles](/ch2#id24) - [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability) - [Fault Tolerance](/ch2#id27) - [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy) - [Software faults](/ch2#software-faults) - [Humans and Reliability](/ch2#id31) - [Scalability](/ch2#sec_introduction_scalability) - [Describing Load](/ch2#id33) - [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing) - [Principles for Scalability](/ch2#id35) - [Maintainability](/ch2#sec_introduction_maintainability) - [Operability: Making Life Easy for Operations](/ch2#id37) - [Simplicity: Managing Complexity](/ch2#id38) - [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability) - [Summary](/ch2#summary) - [References](/ch2#references) ## [3. Data Models and Query Languages](/ch3) - [Relational Model versus Document Model](/ch3#sec_datamodels_history) - [The Object-Relational Mismatch](/ch3#sec_datamodels_document) - [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm) - [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization) - [Trade-offs of normalization](/ch3#trade-offs-of-normalization) - [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many) - [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - [When to Use Which Model](/ch3#sec_datamodels_document_summary) - [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - [Query languages for documents](/ch3#query-languages-for-documents) - [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - [Graph-Like Data Models](/ch3#sec_datamodels_graph) - [Property Graphs](/ch3#id56) - [The Cypher Query Language](/ch3#id57) - [Graph Queries in SQL](/ch3#id58) - [Triple-Stores and SPARQL](/ch3#id59) - [The RDF data model](/ch3#the-rdf-data-model) - [The SPARQL query language](/ch3#the-sparql-query-language) - [Datalog: Recursive Relational Queries](/ch3#id62) - [GraphQL](/ch3#id63) - [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - [Dataframes, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - [Summary](/ch3#summary) - [References](/ch3#references) ## [4. Storage and Retrieval](/ch4) - [Storage and Indexing for OLTP](/ch4#sec_storage_oltp) - [Log-Structured Storage](/ch4#sec_storage_log_structured) - [The SSTable file format](/ch4#the-sstable-file-format) - [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - [Bloom filters](/ch4#bloom-filters) - [Compaction strategies](/ch4#sec_storage_lsm_compaction) - [B-Trees](/ch4#sec_storage_b_trees) - [Making B-trees reliable](/ch4#sec_storage_btree_wal) - [B-tree variants](/ch4#b-tree-variants) - [Comparing B-Trees and LSM-Trees](/ch4#sec_storage_btree_lsm_comparison) - [Read performance](/ch4#read-performance) - [Sequential vs. random writes](/ch4#sidebar_sequential) - [Write amplification](/ch4#write-amplification) - [Disk space usage](/ch4#disk-space-usage) - [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn) - [Storing values within the index](/ch4#sec_storage_index_heap) - [Keeping everything in memory](/ch4#sec_storage_inmemory) - [Data Storage for Analytics](/ch4#sec_storage_analytics) - [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - [Column-Oriented Storage](/ch4#sec_storage_column) - [Column Compression](/ch4#sec_storage_column_compression) - [Sort Order in Column Storage](/ch4#sort-order-in-column-storage) - [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - [Full-Text Search](/ch4#sec_storage_full_text) - [Vector Embeddings](/ch4#id92) - [Summary](/ch4#summary) - [References](/ch4#references) ## [5. Encoding and Evolution](/ch5) - [Formats for Encoding Data](/ch5#sec_encoding_formats) - [Language-Specific Formats](/ch5#id96) - [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - [JSON Schema](/ch5#json-schema) - [Binary encoding](/ch5#binary-encoding) - [Protocol Buffers](/ch5#sec_encoding_protobuf) - [Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - [Avro](/ch5#sec_encoding_avro) - [The writer’s schema and the reader’s schema](/ch5#the-writers-schema-and-the-readers-schema) - [Schema evolution rules](/ch5#schema-evolution-rules) - [But what is the writer’s schema?](/ch5#but-what-is-the-writers-schema) - [Dynamically generated schemas](/ch5#dynamically-generated-schemas) - [The Merits of Schemas](/ch5#sec_encoding_schemas) - [Modes of Dataflow](/ch5#sec_encoding_dataflow) - [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db) - [Different values written at different times](/ch5#different-values-written-at-different-times) - [Archival storage](/ch5#archival-storage) - [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc) - [Web services](/ch5#sec_web_services) - [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - [Durable execution](/ch5#durable-execution) - [Event-Driven Architectures](/ch5#sec_encoding_dataflow_msg) - [Message brokers](/ch5#message-brokers) - [Distributed actor frameworks](/ch5#distributed-actor-frameworks) - [Summary](/ch5#summary) - [References](/ch5#references) ## [6. Replication](/ch6) - [Single-Leader Replication](/ch6#sec_replication_leader) - [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async) - [Setting Up New Followers](/ch6#sec_replication_new_replica) - [Handling Node Outages](/ch6#sec_replication_failover) - [Follower failure: Catch-up recovery](/ch6#follower-failure-catch-up-recovery) - [Leader failure: Failover](/ch6#leader-failure-failover) - [Implementation of Replication Logs](/ch6#sec_replication_implementation) - [Statement-based replication](/ch6#statement-based-replication) - [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - [Problems with Replication Lag](/ch6#sec_replication_lag) - [Reading Your Own Writes](/ch6#sec_replication_ryw) - [Monotonic Reads](/ch6#sec_replication_monotonic_reads) - [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix) - [Solutions for Replication Lag](/ch6#id131) - [Multi-Leader Replication](/ch6#sec_replication_multi_leader) - [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - [Multi-leader replication topologies](/ch6#sec_replication_topologies) - [Problems with different topologies](/ch6#problems-with-different-topologies) - [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients) - [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts) - [Conflict avoidance](/ch6#conflict-avoidance) - [Last write wins (discarding concurrent writes)](/ch6#sec_replication_lww) - [Manual conflict resolution](/ch6#manual-conflict-resolution) - [Automatic conflict resolution](/ch6#automatic-conflict-resolution) - [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - [What is a conflict?](/ch6#what-is-a-conflict) - [Leaderless Replication](/ch6#sec_replication_leaderless) - [Writing to the Database When a Node Is Down](/ch6#id287) - [Catching up on missed writes](/ch6#sec_replication_read_repair) - [Quorums for reading and writing](/ch6#sec_replication_quorum_condition) - [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations) - [Monitoring staleness](/ch6#monitoring-staleness) - [Single-Leader vs. Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - [Multi-region operation](/ch6#multi-region-operation) - [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - [The “happens-before” relation and concurrency](/ch6#sec_replication_happens_before) - [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship) - [Version vectors](/ch6#version-vectors) - [Summary](/ch6#summary) - [References](/ch6#references) ## [7. Sharding](/ch7) - [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - [Sharding by Key Range](/ch7#sec_sharding_key_range) - [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - [Sharding by Hash of Key](/ch7#sec_sharding_hash) - [Hash modulo number of nodes](/ch7#hash-modulo-number-of-nodes) - [Fixed number of shards](/ch7#fixed-number-of-shards) - [Sharding by hash range](/ch7#sharding-by-hash-range) - [Consistent hashing](/ch7#sec_sharding_consistent_hashing) - [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - [Request Routing](/ch7#sec_sharding_routing) - [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes) - [Local Secondary Indexes](/ch7#id166) - [Global Secondary Indexes](/ch7#id167) - [Summary](/ch7#summary) - [References](/ch7#references) ## [8. Transactions](/ch8) - [What Exactly Is a Transaction?](/ch8#sec_transactions_overview) - [The Meaning of ACID](/ch8#sec_transactions_acid) - [Atomicity](/ch8#sec_transactions_acid_atomicity) - [Consistency](/ch8#sec_transactions_acid_consistency) - [Isolation](/ch8#sec_transactions_acid_isolation) - [Durability](/ch8#durability) - [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - [Single-object writes](/ch8#sec_transactions_single_object) - [The need for multi-object transactions](/ch8#sec_transactions_need) - [Handling errors and aborts](/ch8#handling-errors-and-aborts) - [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - [Read Committed](/ch8#sec_transactions_read_committed) - [No dirty reads](/ch8#no-dirty-reads) - [No dirty writes](/ch8#sec_transactions_dirty_write) - [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl) - [Visibility rules for observing a consistent snapshot](/ch8#sec_transactions_mvcc_visibility) - [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - [Preventing Lost Updates](/ch8#sec_transactions_lost_update) - [Atomic write operations](/ch8#atomic-write-operations) - [Explicit locking](/ch8#explicit-locking) - [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set) - [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - [Write Skew and Phantoms](/ch8#sec_transactions_write_skew) - [Characterizing write skew](/ch8#characterizing-write-skew) - [More examples of write skew](/ch8#more-examples-of-write-skew) - [Phantoms causing write skew](/ch8#sec_transactions_phantom) - [Materializing conflicts](/ch8#materializing-conflicts) - [Serializability](/ch8#sec_transactions_serializability) - [Actual Serial Execution](/ch8#sec_transactions_serial) - [Encapsulating transactions in stored procedures](/ch8#encapsulating-transactions-in-stored-procedures) - [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - [Sharding](/ch8#sharding) - [Summary of serial execution](/ch8#summary-of-serial-execution) - [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl) - [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - [Predicate locks](/ch8#predicate-locks) - [Index-range locks](/ch8#sec_transactions_2pl_range) - [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi) - [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise) - [Detecting stale MVCC reads](/ch8#detecting-stale-mvcc-reads) - [Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - [Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - [Distributed Transactions](/ch8#sec_transactions_distributed) - [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - [A system of promises](/ch8#a-system-of-promises) - [Coordinator failure](/ch8#coordinator-failure) - [Three-phase commit](/ch8#three-phase-commit) - [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa) - [Exactly-once message processing](/ch8#sec_transactions_exactly_once) - [XA transactions](/ch8#xa-transactions) - [Holding locks while in doubt](/ch8#holding-locks-while-in-doubt) - [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - [Problems with XA transactions](/ch8#problems-with-xa-transactions) - [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - [Summary](/ch8#summary) - [References](/ch8#references) ## [9. The Trouble with Distributed Systems](/ch9) - [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - [Unreliable Networks](/ch9#sec_distributed_networks) - [The Limitations of TCP](/ch9#sec_distributed_tcp) - [Network Faults in Practice](/ch9#sec_distributed_network_faults) - [Detecting Faults](/ch9#id307) - [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - [Network congestion and queueing](/ch9#network-congestion-and-queueing) - [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - [Unreliable Clocks](/ch9#sec_distributed_clocks) - [Monotonic Versus Time-of-Day Clocks](/ch9#sec_distributed_monotonic_timeofday) - [Time-of-day clocks](/ch9#time-of-day-clocks) - [Monotonic clocks](/ch9#monotonic-clocks) - [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - [Relying on Synchronized Clocks](/ch9#sec_distributed_clocks_relying) - [Timestamps for ordering events](/ch9#sec_distributed_lww) - [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval) - [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - [Process Pauses](/ch9#sec_distributed_clocks_pauses) - [Response time guarantees](/ch9#sec_distributed_clocks_realtime) - [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - [Knowledge, Truth, and Lies](/ch9#sec_distributed_truth) - [The Majority Rules](/ch9#sec_distributed_majority) - [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing) - [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - [Fencing with multiple replicas](/ch9#fencing-with-multiple-replicas) - [Byzantine Faults](/ch9#sec_distributed_byzantine) - [Weak forms of lying](/ch9#weak-forms-of-lying) - [System Model and Reality](/ch9#sec_distributed_system_model) - [Defining the correctness of an algorithm](/ch9#defining-the-correctness-of-an-algorithm) - [Safety and liveness](/ch9#sec_distributed_safety_liveness) - [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - [Formal Methods and Randomized Testing](/ch9#sec_distributed_formal) - [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - [Fault injection](/ch9#sec_fault_injection) - [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - [Summary](/ch9#summary) - [References](/ch9#references) ## [10. Consistency and Consensus](/ch10) - [Linearizability](/ch10#sec_consistency_linearizability) - [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - [Relying on Linearizability](/ch10#sec_consistency_linearizability_usage) - [Locking and leader election](/ch10#locking-and-leader-election) - [Constraints and uniqueness guarantees](/ch10#sec_consistency_uniqueness) - [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - [The Cost of Linearizability](/ch10#sec_linearizability_cost) - [The CAP theorem](/ch10#the-cap-theorem) - [Linearizability and network delays](/ch10#linearizability-and-network-delays) - [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - [Logical Clocks](/ch10#sec_consistency_timestamps) - [Lamport timestamps](/ch10#lamport-timestamps) - [Hybrid logical clocks](/ch10#hybrid-logical-clocks) - [Lamport/hybrid logical clocks vs. vector clocks](/ch10#lamporthybrid-logical-clocks-vs-vector-clocks) - [Linearizable ID Generators](/ch10#sec_consistency_linearizable_id) - [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - [Enforcing constraints using logical clocks](/ch10#enforcing-constraints-using-logical-clocks) - [Consensus](/ch10#sec_consistency_consensus) - [The Many Faces of Consensus](/ch10#sec_consistency_faces) - [Single-value consensus](/ch10#single-value-consensus) - [Compare-and-set as consensus](/ch10#compare-and-set-as-consensus) - [Shared logs as consensus](/ch10#sec_consistency_shared_logs) - [Fetch-and-add as consensus](/ch10#fetch-and-add-as-consensus) - [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - [Consensus in Practice](/ch10#sec_consistency_total_order) - [Using shared logs](/ch10#sec_consistency_smr) - [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - [Subtleties of consensus](/ch10#subtleties-of-consensus) - [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - [Coordination Services](/ch10#sec_consistency_coordination) - [Allocating work to nodes](/ch10#allocating-work-to-nodes) - [Service discovery](/ch10#service-discovery) - [Summary](/ch10#summary) - [References](/ch10#references) ## [11. Batch Processing](/ch11) - [Batch Processing with Unix Tools](/ch11#sec_batch_unix) - [Simple Log Analysis](/ch11#sec_batch_log_analysis) - [Chain of Commands Versus Custom Program](/ch11#sec_batch_custom_program) - [Sorting Versus In-memory Aggregation](/ch11#id275) - [Batch Processing in Distributed Systems](/ch11#sec_batch_distributed) - [Distributed Filesystems](/ch11#sec_batch_dfs) - [Object Stores](/ch11#id277) - [Distributed Job Orchestration](/ch11#id278) - [Batch Processing Models](/ch11#id431) - [MapReduce](/ch11#sec_batch_mapreduce) - [Dataflow Engines](/ch11#sec_batch_dataflow) - [Shuffling Data](/ch11#sec_shuffle) - [Query languages](/ch11#sec_batch_query_lanauges) - [DataFrames](/ch11#id287) - [Batch Use Cases](/ch11#sec_batch_output) - [Extract–Transform–Load (ETL)](/ch11#sec_batch_etl_usage) - [Analytics](/ch11#sec_batch_olap) - [Machine Learning](/ch11#id290) - [Serving Derived Data](/ch11#sec_batch_serving_derived) - [Summary](/ch11#id292) - [References](/ch11#references) ## [12. Stream Processing](/ch12) - [Transmitting Event Streams](/ch12#sec_stream_transmit) - [Messaging Systems](/ch12#sec_stream_messaging) - [Log-based Message Brokers](/ch12#sec_stream_log) - [Databases and Streams](/ch12#sec_stream_databases) - [Keeping Systems in Sync](/ch12#sec_stream_sync) - [Change Data Capture](/ch12#sec_stream_cdc) - [State, Streams, and Immutability](/ch12#sec_stream_immutability) - [Processing Streams](/ch12#sec_stream_processing) - [Uses of Stream Processing](/ch12#sec_stream_uses) - [Reasoning About Time](/ch12#sec_stream_time) - [Stream Joins](/ch12#sec_stream_joins) - [Fault Tolerance](/ch12#sec_stream_fault_tolerance) - [Summary](/ch12#id332) - [References](/ch12#references) ## [13. A Philosophy of Streaming Systems](/ch13) - [Data Integration](/ch13#sec_future_integration) - [Combining Specialized Tools by Deriving Data](/ch13#id442) - [Batch and Stream Processing](/ch13#sec_future_batch_streaming) - [Unbundling Databases](/ch13#sec_future_unbundling) - [Composing Data Storage Technologies](/ch13#id447) - [Designing Applications Around Dataflow](/ch13#sec_future_dataflow) - [Observing Derived State](/ch13#sec_future_observing) - [Aiming for Correctness](/ch13#sec_future_correctness) - [The End-to-End Argument for Databases](/ch13#sec_future_end_to_end) - [Enforcing Constraints](/ch13#sec_future_constraints) - [Timeliness and Integrity](/ch13#sec_future_integrity) - [Trust, but Verify](/ch13#sec_future_verification) - [Summary](/ch13#id367) - [References](/ch13#references) ## [14. Doing the Right Thing](/ch14) - [Predictive Analytics](/ch14#id369) - [Bias and Discrimination](/ch14#id370) - [Responsibility and Accountability](/ch14#id371) - [Feedback Loops](/ch14#id372) - [Privacy and Tracking](/ch14#id373) - [Surveillance](/ch14#id374) - [Consent and Freedom of Choice](/ch14#id375) - [Privacy and Use of Data](/ch14#id457) - [Data as Assets and Power](/ch14#id376) - [Remembering the Industrial Revolution](/ch14#id377) - [Legislation and Self-Regulation](/ch14#sec_future_legislation) - [Summary](/ch14#id594) - [References](/ch14#references) ## [Glossary](/glossary) ## [Colophon](/colophon) - [About the Author](/colophon#about-the-author) - [Colophon](/colophon#colophon)