# Indexes LLMS index: [llms.txt](/llms.txt) --- ### Symbols - 3FS (distributed filesystem, [Distributed Filesystems](/ch11#sec_batch_dfs) ### A - aborts (transactions), [Transactions](/ch8#ch_transactions), [Atomicity](/ch8#sec_transactions_acid_atomicity) - cascading, [No dirty reads](/ch8#no-dirty-reads) - in two-phase commit, [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - performance of optimistic concurrency control, [Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - retrying aborted transactions, [Handling errors and aborts](/ch8#handling-errors-and-aborts) - abstraction, [Layering of cloud services](/ch1#layering-of-cloud-services), [Simplicity: Managing Complexity](/ch2#id38), [Data Models and Query Languages](/ch3#ch_datamodels), [Transactions](/ch8#ch_transactions), [Summary](/ch8#summary) - accidental complexity, [Simplicity: Managing Complexity](/ch2#id38) - accountability, [Responsibility and Accountability](/ch14#id371) - accounting (financial data), [Summary](/ch3#summary), [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - Accumulo (database) - wide-column data model, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality), [Column Compression](/ch4#sec_storage_column_compression) - ACID properties (transactions), [The Meaning of ACID](/ch8#sec_transactions_acid) - atomicity, [Atomicity](/ch8#sec_transactions_acid_atomicity), [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - consistency, [Consistency](/ch8#sec_transactions_acid_consistency), [Maintaining integrity in the face of software bugs](/ch13#id455) - durability, [Making B-trees reliable](/ch4#sec_storage_btree_wal), [Durability](/ch8#durability) - isolation, [Isolation](/ch8#sec_transactions_acid_isolation), [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - acknowledgements (messaging), [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - active/active replication (see multi-leader replication) - active/passive replication (see leader-based replication) - ActiveMQ (messaging), [Message brokers](/ch5#message-brokers), [Message brokers compared to databases](/ch12#id297) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - ActiveRecord (object-relational mapper), [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm), [Handling errors and aborts](/ch8#handling-errors-and-aborts) - activity (workflows) (see workflow engines) - actor model, [Distributed actor frameworks](/ch5#distributed-actor-frameworks) - (see also event-driven architecture) - comparison to stream processing, [Event-Driven Architectures and RPC](/ch12#sec_stream_actors_drpc) - adaptive capacity, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - Advanced Message Queuing Protocol (see AMQP) - aerospace systems, [Byzantine Faults](/ch9#sec_distributed_byzantine) - Aerospike (database) - strong consistency mode, [Single-object writes](/ch8#sec_transactions_single_object) - AGE (graph database), [The Cypher Query Language](/ch3#id57) - aggregation - data cubes and materialized views, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - in batch processes, [Sorting Versus In-memory Aggregation](/ch11#id275) - in stream processes, [Stream analytics](/ch12#id318) - aggregation pipeline (MongoDB), [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization), [Query languages for documents](/ch3#query-languages-for-documents) - Agile, [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability) - minimizing irreversibility, [Batch Processing](/ch11#ch_batch), [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - moving faster with confidence, [The end-to-end argument again](/ch13#id456) - agreement, [Single-value consensus](/ch10#single-value-consensus), [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - (see also consensus) - AI (artificial intelligence) (see machine learning) - AI Act (European Union), [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - AirByte, [Data Warehousing](/ch1#sec_introduction_dwh) - Airflow (workflow scheduler), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows), [Batch Processing](/ch11#ch_batch), [Scheduling Workflows](/ch11#sec_batch_workflows) - cloud data warehouse integration, [Query languages](/ch11#sec_batch_query_lanauges) - use for ETL, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - Akamai - response time study, [Average, Median, and Percentiles](/ch2#id24) - algorithms - algorithm correctness, [Defining the correctness of an algorithm](/ch9#defining-the-correctness-of-an-algorithm) - B-trees, [B-Trees](/ch4#sec_storage_b_trees)-[B-tree variants](/ch4#b-tree-variants) - for distributed systems, [System Model and Reality](/ch9#sec_distributed_system_model) - mergesort, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Shuffling Data](/ch11#sec_shuffle) - scheduling, [Resource Allocation](/ch11#id279) - SSTables and LSM-trees, [The SSTable file format](/ch4#the-sstable-file-format)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - all-to-all replication topologies, [Multi-leader replication topologies](/ch6#sec_replication_topologies) - AllegroGraph (database), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - SPARQL query language, [The SPARQL query language](/ch3#the-sparql-query-language) - ALTER TABLE statement (SQL), [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility), [Encoding and Evolution](/ch5#ch_encoding) - Amazon - Dynamo (see Dynamo (database)) - response time study, [Average, Median, and Percentiles](/ch2#id24) - Amazon Web Services (AWS) - Aurora (see Aurora (cloud database)) - ClockBound (see ClockBound (time sync)) - correctness testing, [Formal Methods and Randomized Testing](/ch9#sec_distributed_formal) - DynamoDB (see DynamoDB (database)) - EBS (see EBS (virtual block device)) - Kinesis (see Kinesis (messaging)) - Neptune (see Neptune (graph database)) - network reliability, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - S3 (see S3 (object storage)) - amplification - of bias, [Bias and Discrimination](/ch14#id370) - of failures, [Maintaining derived state](/ch13#id446) - of tail latency, [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla), [Local Secondary Indexes](/ch7#id166) - write amplification, [Write amplification](/ch4#write-amplification) - AMQP (Advanced Message Queuing Protocol), [Message brokers compared to databases](/ch12#id297) - (see also messaging systems) - comparison to log-based messaging, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Replaying old messages](/ch12#sec_stream_replay) - message ordering, [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - analytical systems, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - as derived data systems, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - ETL from operational systems, [Data Warehousing](/ch1#sec_introduction_dwh) - governance, [Beyond the data lake](/ch1#beyond-the-data-lake) - analytics, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics)-[Systems of Record and Derived Data](/ch1#sec_introduction_derived) - comparison to transaction processing, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - data normalization, [Trade-offs of normalization](/ch3#trade-offs-of-normalization) - data warehousing (see data warehousing) - predictive (see predictive analytics) - relation to batch processing, [Analytics](/ch11#sec_batch_olap)-[Analytics](/ch11#sec_batch_olap) - schemas for, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics)-[Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - snapshot isolation for queries, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - stream analytics, [Stream analytics](/ch12#id318) - analytics engineering, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - anti-entropy, [Catching up on missed writes](/ch6#sec_replication_read_repair) - Antithesis (deterministic simulation testing), [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - Apache Accumulo (see Accumulo) - Apache ActiveMQ (see ActiveMQ) - Apache AGE (see AGE) - Apache Arrow (see Arrow (data format)) - Apache Avro (see Avro) - Apache Beam (see Beam) - Apache BookKeeper (see BookKeeper) - Apache Cassandra (see Cassandra) - Apache Curator (see Curator) - Apache DataFusion (see DataFusion (query engine)) - Apache Druid (see Druid (database)) - Apache Flink (see Flink (processing framework)) - Apache HBase (see HBase) - Apache Iceberg (see Iceberg (table format)) - Apache Jena (see Jena) - Apache Kafka (see Kafka) - Apache Lucene (see Lucene) - Apache Oozie (see Oozie (workflow scheduler)) - Apache ORC (see ORC (data format)) - Apache Parquet (see Parquet (data format)) - Apache Pig (query language), [Query languages](/ch11#sec_batch_query_lanauges) - Apache Pinot (see Pinot (database)) - Apache Pulsar (see Pulsar) - Apache Qpid (see Qpid) - Apache Samza (see Samza) - Apache Solr (see Solr) - Apache Spark (see Spark) (see Spark (processing framework)) - Apache Storm (see Storm) - Apache Superset (see Superset (data visualization software)) - Apache Thrift (see Thrift) - Apache ZooKeeper (see ZooKeeper) - Apama (stream analytics), [Complex event processing](/ch12#id317) - append-only files (see logs) - Application Programming Interfaces (APIs), [Data Models and Query Languages](/ch3#ch_datamodels) - for change streams, [API support for change streams](/ch12#sec_stream_change_api) - for distributed transactions, [XA transactions](/ch8#xa-transactions) - for services, [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc)-[Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - (see also services) - evolvability, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - RESTful, [Web services](/ch5#sec_web_services) - application state (see state) - approximate search (see similarity search) - archival storage, data from databases, [Archival storage](/ch5#archival-storage) - arcs (see edges) - ArcticDB (database), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - arithmetic mean, [Average, Median, and Percentiles](/ch2#id24) - arrays - array databases, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - multidimensional, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - Arrow (data format), [Column-Oriented Storage](/ch4#sec_storage_column), [DataFrames](/ch11#id287) - artificial intelligence (see machine learning) - ASCII text, [Protocol Buffers](/ch5#sec_encoding_protobuf) - ASN.1 (schema language), [The Merits of Schemas](/ch5#sec_encoding_schemas) - associative table, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many), [Property Graphs](/ch3#id56) - asynchronous networks, [Unreliable Networks](/ch9#sec_distributed_networks), [Glossary](/glossary) - comparison to synchronous networks, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - system model, [System Model and Reality](/ch9#sec_distributed_system_model) - asynchronous replication, [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async), [Glossary](/glossary) - data loss on failover, [Leader failure: Failover](/ch6#leader-failure-failover) - reads from asynchronous follower, [Problems with Replication Lag](/ch6#sec_replication_lag) - with multiple leaders, [Multi-Leader Replication](/ch6#sec_replication_multi_leader) - Asynchronous Transfer Mode (ATM), [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - atomic broadcast, [Shared logs as consensus](/ch10#sec_consistency_shared_logs) - atomic clocks, [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - (see also clocks) - atomicity (concurrency), [Glossary](/glossary) - atomic increment, [Single-object writes](/ch8#sec_transactions_single_object) - compare-and-set (CAS), [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set), [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - (see also compare-and-set (CAS)) - denormalized data, [Trade-offs of normalization](/ch3#trade-offs-of-normalization) - fetch-and-add/increment, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical), [Consensus](/ch10#sec_consistency_consensus), [Fetch-and-add as consensus](/ch10#fetch-and-add-as-consensus) - write operations, [Atomic write operations](/ch8#atomic-write-operations) - atomicity (transactions), [Atomicity](/ch8#sec_transactions_acid_atomicity), [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object), [Glossary](/glossary) - atomic commit - avoiding, [Multi-shard request processing](/ch13#id360), [Coordination-avoiding data systems](/ch13#id454) - blocking and nonblocking, [Three-phase commit](/ch8#three-phase-commit) - in stream processing, [Exactly-once message processing](/ch8#sec_transactions_exactly_once), [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited), [Atomic commit revisited](/ch12#sec_stream_atomic_commit) - maintaining derived data, [Keeping Systems in Sync](/ch12#sec_stream_sync) - distributed transactions, [Distributed Transactions](/ch8#sec_transactions_distributed)-[Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - for multi-object transactions, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - for single-object writes, [Single-object writes](/ch8#sec_transactions_single_object) - relation to consensus, [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - auditability, [Trust, but Verify](/ch13#sec_future_verification)-[Tools for auditable data systems](/ch13#id366) - designing for, [Designing for auditability](/ch13#id365) - self-auditing systems, [Don't just blindly trust what they promise](/ch13#id364) - through immutability, [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - tools for auditable data systems, [Tools for auditable data systems](/ch13#id366) - Aurora (cloud database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - Aurora DSQL (database) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - auto-scaling, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - Automerge (CRDT library), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - availability, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability) - (see also fault tolerance) - in CAP theorem, [The CAP theorem](/ch10#the-cap-theorem) - in leader election, [Subtleties of consensus](/ch10#subtleties-of-consensus) - in service level agreements (SLAs), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - availability zones, [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy), [Reading Your Own Writes](/ch6#sec_replication_ryw) - Avro (data format), [Avro](/ch5#sec_encoding_avro)-[Dynamically generated schemas](/ch5#dynamically-generated-schemas) - dynamically generated schemas, [Dynamically generated schemas](/ch5#dynamically-generated-schemas) - object container files, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema), [Archival storage](/ch5#archival-storage) - reader determining writer's schema, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - schema evolution, [The writer's schema and the reader's schema](/ch5#the-writers-schema-and-the-readers-schema) - use in batch processing, [MapReduce](/ch11#sec_batch_mapreduce) - awk (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Distributed Job Orchestration](/ch11#id278) - Axon Framework, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - Azkaban (workflow scheduler), [Batch Processing](/ch11#ch_batch) - Azure Blob Storage (object storage), [Layering of cloud services](/ch1#layering-of-cloud-services), [Setting Up New Followers](/ch6#sec_replication_new_replica) - conditional headers, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - Azure managed disks, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - Azure SQL DB (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - Azure Storage, [Object Stores](/ch11#id277) - Azure Synapse Analytics (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - Azure Virtual Machines - spot virtual machines, [Handling Faults](/ch11#id281) ### B - B-trees (indexes), [B-Trees](/ch4#sec_storage_b_trees)-[B-tree variants](/ch4#b-tree-variants) - B+ trees, [B-tree variants](/ch4#b-tree-variants) - branching factor, [B-Trees](/ch4#sec_storage_b_trees) - comparison to LSM-trees, [Comparing B-Trees and LSM-Trees](/ch4#sec_storage_btree_lsm_comparison)-[Disk space usage](/ch4#disk-space-usage) - crash recovery, [Making B-trees reliable](/ch4#sec_storage_btree_wal) - growing by splitting a page, [B-Trees](/ch4#sec_storage_b_trees) - immutable variants, [B-tree variants](/ch4#b-tree-variants), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - similarity to shard splitting, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - variants, [B-tree variants](/ch4#b-tree-variants) - B2 (object storage), [Distributed Filesystems](/ch11#sec_batch_dfs) - Backblaze B2 (see B2 (object storage)) - backend, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - backoff, exponential, [Describing Performance](/ch2#sec_introduction_percentiles), [Handling errors and aborts](/ch8#handling-errors-and-aborts) - backpressure, [Describing Performance](/ch2#sec_introduction_percentiles), [Read performance](/ch4#read-performance), [Messaging Systems](/ch12#sec_stream_messaging), [Glossary](/glossary) - in batch processing, [Scheduling Workflows](/ch11#sec_batch_workflows) - in TCP, [The Limitations of TCP](/ch9#sec_distributed_tcp) - backups - database snapshot for replication, [Setting Up New Followers](/ch6#sec_replication_new_replica) - in multitenant systems, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - integrity of, [Don't just blindly trust what they promise](/ch13#id364) - snapshot isolation for, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - using object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - versus replication, [Replication](/ch6#ch_replication) - backward compatibility, [Encoding and Evolution](/ch5#ch_encoding) - BadgerDB (database) - serializable transactions, [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi) - BASE, contrast to ACID, [The Meaning of ACID](/ch8#sec_transactions_acid) - bash shell (Unix), [Storage and Indexing for OLTP](/ch4#sec_storage_oltp) - batch processing, [Batch Processing](/ch11#ch_batch)-[Summary](/ch11#id292), [Glossary](/glossary) - and functional programming, [MapReduce](/ch11#sec_batch_mapreduce) - benefits of, [Batch Processing](/ch11#ch_batch) - combining with stream processing, [Unifying batch and stream processing](/ch13#id338) - comparison to stream processing, [Processing Streams](/ch12#sec_stream_processing) - dataflow engines, [Dataflow Engines](/ch11#sec_batch_dataflow)-[Dataflow Engines](/ch11#sec_batch_dataflow) - fault tolerance, [Handling Faults](/ch11#id281), [Messaging Systems](/ch12#sec_stream_messaging) - for data integration, [Batch and Stream Processing](/ch13#sec_future_batch_streaming)-[Unifying batch and stream processing](/ch13#id338) - graphs and iterative processing, [Machine Learning](/ch11#id290) - high-level APIs and languages, [Query languages](/ch11#sec_batch_query_lanauges)-[Query languages](/ch11#sec_batch_query_lanauges) - in cloud data warehouses, [Query languages](/ch11#sec_batch_query_lanauges) - in distributed systems, [Batch Processing in Distributed Systems](/ch11#sec_batch_distributed) - join and group by, [JOIN and GROUP BY](/ch11#sec_batch_join)-[JOIN and GROUP BY](/ch11#sec_batch_join) - limitations, [Batch Processing](/ch11#ch_batch) - log-based messaging and, [Replaying old messages](/ch12#sec_stream_replay) - maintaining derived state, [Maintaining derived state](/ch13#id446) - measuring performance, [Batch Processing](/ch11#ch_batch) - models of, [Batch Processing Models](/ch11#id431) - resource allocation, [Resource Allocation](/ch11#id279)-[Resource Allocation](/ch11#id279) - resource managers, [Distributed Job Orchestration](/ch11#id278) - schedulers, [Distributed Job Orchestration](/ch11#id278) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived)-[Serving Derived Data](/ch11#sec_batch_serving_derived) - shuffling data, [Shuffling Data](/ch11#sec_shuffle)-[Shuffling Data](/ch11#sec_shuffle) - task execution, [Distributed Job Orchestration](/ch11#id278) - use cases, [Batch Use Cases](/ch11#sec_batch_output)-[Serving Derived Data](/ch11#sec_batch_serving_derived) - using Unix tools (example), [Batch Processing with Unix Tools](/ch11#sec_batch_unix)-[Sorting Versus In-memory Aggregation](/ch11#id275) - batch processing frameworks - comparison to operating systems, [Batch Processing in Distributed Systems](/ch11#sec_batch_distributed) - Beam (dataflow library), [Unifying batch and stream processing](/ch13#id338) - BERT (language model), [Vector Embeddings](/ch4#id92) - bias, [Bias and Discrimination](/ch14#id370) - bidirectional replication (see multi-leader replication) - big ball of mud, [Simplicity: Managing Complexity](/ch2#id38) - big data - versus data minimization, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Legislation and Self-Regulation](/ch14#sec_future_legislation) - BigQuery (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Batch Processing](/ch11#ch_batch) - DataFrames, [Query languages](/ch11#sec_batch_query_lanauges) - sharding and clustering, [Sharding by hash range](/ch7#sharding-by-hash-range) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - Bigtable (database) - sharding scheme, [Sharding by Key Range](/ch7#sec_sharding_key_range) - storage layout, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - tablets (sharding), [Sharding](/ch7#ch_sharding) - wide-column data model, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality), [Column Compression](/ch4#sec_storage_column_compression) - binary data encodings, [Binary encoding](/ch5#binary-encoding)-[The Merits of Schemas](/ch5#sec_encoding_schemas) - Avro, [Avro](/ch5#sec_encoding_avro)-[Dynamically generated schemas](/ch5#dynamically-generated-schemas) - MessagePack, [Binary encoding](/ch5#binary-encoding)-[Binary encoding](/ch5#binary-encoding) - Protocol Buffers, [Protocol Buffers](/ch5#sec_encoding_protobuf)-[Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - binary encoding - based on schemas, [The Merits of Schemas](/ch5#sec_encoding_schemas) - by network drivers, [The Merits of Schemas](/ch5#sec_encoding_schemas) - binary strings, lack of support in JSON and XML, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - Bitcoin (cryptocurrency), [Tools for auditable data systems](/ch13#id366) - Byzantine fault tolerance, [Byzantine Faults](/ch9#sec_distributed_byzantine) - concurrency bugs in exchanges, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - bitmap indexes, [Column Compression](/ch4#sec_storage_column_compression) - BitTorrent uTP protocol, [The Limitations of TCP](/ch9#sec_distributed_tcp) - Bkd-trees (indexes), [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - blameless postmortems, [Humans and Reliability](/ch2#id31) - Blazegraph (database), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - SPARQL query language, [The SPARQL query language](/ch3#the-sparql-query-language) - blob storage (see object storage) - block (file system), [Distributed Filesystems](/ch11#sec_batch_dfs) - block device (disk), [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - blockchains, [Summary](/ch3#summary) - Byzantine fault tolerance, [Byzantine Faults](/ch9#sec_distributed_byzantine), [Consensus](/ch10#sec_consistency_consensus), [Tools for auditable data systems](/ch13#id366) - blocking atomic commit, [Three-phase commit](/ch8#three-phase-commit) - Bloom filter (algorithm), [Bloom filters](/ch4#bloom-filters), [Read performance](/ch4#read-performance), [Stream analytics](/ch12#id318) - BookKeeper (replicated log), [Allocating work to nodes](/ch10#allocating-work-to-nodes) - bounded datasets, [Stream Processing](/ch12#ch_stream), [Glossary](/glossary) - (see also batch processing) - bounded delays, [Glossary](/glossary) - in networks, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - process pauses, [Response time guarantees](/ch9#sec_distributed_clocks_realtime) - broadcast - total order broadcast (see shared logs) - brokerless messaging, [Direct messaging from producers to consumers](/ch12#id296) - Brubeck (metrics aggregator), [Direct messaging from producers to consumers](/ch12#id296) - BTM (transaction coordinator), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - Buf - Bufstream (messaging), [Setting Up New Followers](/ch6#sec_replication_new_replica) - Bufstream (messaging), [Disk space usage](/ch12#sec_stream_disk_usage) - build or buy, [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud) - bursty network traffic patterns, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - business analyst, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - business data processing, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - business intelligence, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics)-[Data Warehousing](/ch1#sec_introduction_dwh) - Business Process Execution Language (BPEL), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - Business Process Model and Notation (BPMN), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - example, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - byte sequence, encoding data in, [Formats for Encoding Data](/ch5#sec_encoding_formats) - Byzantine faults, [Byzantine Faults](/ch9#sec_distributed_byzantine)-[Weak forms of lying](/ch9#weak-forms-of-lying), [System Model and Reality](/ch9#sec_distributed_system_model), [Glossary](/glossary) - Byzantine fault-tolerant systems, [Byzantine Faults](/ch9#sec_distributed_byzantine) - Byzantine Generals Problem, [Byzantine Faults](/ch9#sec_distributed_byzantine) - consensus algorithms and, [Consensus](/ch10#sec_consistency_consensus), [Tools for auditable data systems](/ch13#id366) ### C - caches, [Keeping everything in memory](/ch4#sec_storage_inmemory), [Glossary](/glossary) - and materialized views, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - as derived data, [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - in CPUs, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized), [Linearizability and network delays](/ch10#linearizability-and-network-delays) - invalidation and maintenance, [Keeping Systems in Sync](/ch12#sec_stream_sync), [Maintaining materialized views](/ch12#sec_stream_mat_view) - linearizability, [Linearizability](/ch10#sec_consistency_linearizability) - local disks in the cloud, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - calendar sync, [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - California Consumer Privacy Act (CCPA), [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - Camunda (workflow engine), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - canonical version (of data), [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - CAP theorem, [The CAP theorem](/ch10#the-cap-theorem)-[The CAP theorem](/ch10#the-cap-theorem), [Glossary](/glossary) - capacity planning, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - Cap'n Proto (data format), [Formats for Encoding Data](/ch5#sec_encoding_formats) - carbon emissions, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - cascading aborts, [No dirty reads](/ch8#no-dirty-reads) - cascading failures, [Software faults](/ch2#software-faults), [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations), [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - Cassandra (database) - change data capture, [Implementing change data capture](/ch12#id307), [API support for change streams](/ch12#sec_stream_change_api) - compaction strategy, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - consistency level ANY, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - hash-range sharding, [Sharding by Hash of Key](/ch7#sec_sharding_hash), [Sharding by hash range](/ch7#sharding-by-hash-range) - last-write-wins conflict resolution, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - leaderless replication, [Leaderless Replication](/ch6#sec_replication_leaderless) - lightweight transactions, [Single-object writes](/ch8#sec_transactions_single_object) - linearizability, lack of, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - log-structured storage, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - multi-region support, [Multi-region operation](/ch6#multi-region-operation) - secondary indexes, [Local Secondary Indexes](/ch7#id166) - use of clocks, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations), [Timestamps for ordering events](/ch9#sec_distributed_lww) - vnodes (sharding), [Sharding](/ch7#ch_sharding) - cat (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis) - catalog, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - causal context, [Version vectors](/ch6#version-vectors) - (see also causal dependencies) - causal dependencies, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before)-[Version vectors](/ch6#version-vectors) - capturing, [Version vectors](/ch6#version-vectors), [Ordering events to capture causality](/ch13#sec_future_capture_causality), [Reads are events too](/ch13#sec_future_read_events) - by total ordering, [The limits of total ordering](/ch13#id335) - in transactions, [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise) - sending message to friends (example), [Ordering events to capture causality](/ch13#sec_future_capture_causality) - causality, [Glossary](/glossary) - causal ordering - total order consistent with, [Logical Clocks](/ch10#sec_consistency_timestamps) - consistency with, [Logical Clocks](/ch10#sec_consistency_timestamps)-[Enforcing constraints using logical clocks](/ch10#enforcing-constraints-using-logical-clocks) - happens-before relation, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - in serializable transactions, [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise)-[Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - mismatch with clocks, [Timestamps for ordering events](/ch9#sec_distributed_lww) - ordering events to capture, [Ordering events to capture causality](/ch13#sec_future_capture_causality) - violations of, [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix), [Problems with different topologies](/ch6#problems-with-different-topologies), [Timestamps for ordering events](/ch9#sec_distributed_lww) - with synchronized clocks, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - cell-based architecture, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - CEP (see complex event processing) - CephFS (distributed filesystem), [Batch Processing](/ch11#ch_batch), [Object Stores](/ch11#id277) - certificate transparency, [Tools for auditable data systems](/ch13#id366) - cgroups, [Distributed Job Orchestration](/ch11#id278) - change data capture, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication), [Change Data Capture](/ch12#sec_stream_cdc) - API support for change streams, [API support for change streams](/ch12#sec_stream_change_api) - comparison to event sourcing, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - implementing, [Implementing change data capture](/ch12#id307) - initial snapshot, [Initial snapshot](/ch12#sec_stream_cdc_snapshot) - log compaction, [Log compaction](/ch12#sec_stream_log_compaction) - changelogs, [State, Streams, and Immutability](/ch12#sec_stream_immutability) - change data capture, [Change Data Capture](/ch12#sec_stream_cdc) - for operator state, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - in stream joins, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - log compaction, [Log compaction](/ch12#sec_stream_log_compaction) - maintaining derived state, [Databases and Streams](/ch12#sec_stream_databases) - chaos engineering, [Fault Tolerance](/ch2#id27), [Fault injection](/ch9#sec_fault_injection) - checkpointing - in high-performance computing, [Cloud Computing Versus Supercomputing](/ch1#id17) - in stream processors, [Microbatching and checkpointing](/ch12#id329) - circuit breaker (limiting retries), [Describing Performance](/ch2#sec_introduction_percentiles) - circuit-switched networks, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - circular buffers, [Disk space usage](/ch12#sec_stream_disk_usage) - circular replication topologies, [Multi-leader replication topologies](/ch6#sec_replication_topologies) - Citus (database) - hash sharding, [Fixed number of shards](/ch7#fixed-number-of-shards) - ClickHouse (database), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - incremental view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - clickstream data, analysis of, [JOIN and GROUP BY](/ch11#sec_batch_join) - clients - calling services, [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc) - offline-capable, [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients), [Stateful, offline-capable clients](/ch13#id347) - pushing state changes to, [Pushing state changes to clients](/ch13#id348) - request routing, [Request Routing](/ch7#sec_sharding_routing) - ClockBound (time sync), [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval) - use in YugabyteDB, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - clocks, [Unreliable Clocks](/ch9#sec_distributed_clocks)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - atomic clocks, [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - confidence interval, [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval)-[Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - for global snapshots, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - hybrid logical clocks, [Hybrid logical clocks](/ch10#hybrid-logical-clocks) - logical (see logical clocks) - skew, [Last write wins (discarding concurrent writes)](/ch6#sec_replication_lww), [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations), [Relying on Synchronized Clocks](/ch9#sec_distributed_clocks_relying)-[Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - slewing, [Monotonic clocks](/ch9#monotonic-clocks) - synchronization and accuracy, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy)-[Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - synchronization using GPS, [Unreliable Clocks](/ch9#sec_distributed_clocks), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy), [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - time-of-day versus monotonic clocks, [Monotonic Versus Time-of-Day Clocks](/ch9#sec_distributed_monotonic_timeofday) - timestamping events, [Whose clock are you using, anyway?](/ch12#id438) - cloud services, [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud)-[Cloud Computing Versus Supercomputing](/ch1#id17) - availability zones, [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy), [Reading Your Own Writes](/ch6#sec_replication_ryw) - data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - need for service discovery, [Service discovery](/ch10#service-discovery) - network glitches, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - pros and cons, [Pros and Cons of Cloud Services](/ch1#sec_introduction_cloud_tradeoffs)-[Pros and Cons of Cloud Services](/ch1#sec_introduction_cloud_tradeoffs) - quotas, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - regions (see regions (geographic distribution)) - serverless, [Microservices and Serverless](/ch1#sec_introduction_microservices) - shared resources, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - versus supercomputing, [Cloud Computing Versus Supercomputing](/ch1#id17) - cloud-native, [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native)-[Operations in the Cloud Era](/ch1#sec_introduction_operations) - Cloudflare - R2 (see R2 (object storage)) - clustered indexes, [Storing values within the index](/ch4#sec_storage_index_heap) - clustering (record ordering), [Sharding by hash range](/ch7#sharding-by-hash-range) - CockroachDB (database) - consensus-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - consistency model, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - key-range sharding, [Sharding](/ch7#ch_sharding), [Sharding by Key Range](/ch7#sec_sharding_key_range) - serializable transactions, [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi) - sharded secondary indexes, [Global Secondary Indexes](/ch7#id167) - transactions, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - use of model-checking, [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - code generation - for query execution, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - with Protocol Buffers, [Protocol Buffers](/ch5#sec_encoding_protobuf) - collaborative editing, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - column families (Bigtable), [Data locality for reads and writes](/ch3#sec_datamodels_document_locality), [Column Compression](/ch4#sec_storage_column_compression) - column-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column)-[Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - column compression, [Column Compression](/ch4#sec_storage_column_compression) - Parquet, [Column-Oriented Storage](/ch4#sec_storage_column), [Archival storage](/ch5#archival-storage) - sort order in, [Sort Order in Column Storage](/ch4#sort-order-in-column-storage)-[Sort Order in Column Storage](/ch4#sort-order-in-column-storage) - vectorized processing, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - versus wide-column model, [Column Compression](/ch4#sec_storage_column_compression) - writing to, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - comma-separated values (see CSV) - command query responsibility segregation (CQRS), [Event Sourcing and CQRS](/ch3#sec_datamodels_events)-[Event Sourcing and CQRS](/ch3#sec_datamodels_events), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - commands (event sourcing), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - commits (transactions), [Transactions](/ch8#ch_transactions) - atomic commit, [Distributed Transactions](/ch8#sec_transactions_distributed)-[Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - (see also atomicity; transactions) - read committed isolation, [Read Committed](/ch8#sec_transactions_read_committed) - three-phase commit (3PC), [Three-phase commit](/ch8#three-phase-commit) - two-phase commit (2PC), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc)-[Coordinator failure](/ch8#coordinator-failure) - commutative operations, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - compaction - of changelogs, [Log compaction](/ch12#sec_stream_log_compaction) - (see also log compaction) - for stream operator state, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - of log-structured storage, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - issues with, [Read performance](/ch4#read-performance) - size-tiered and leveled approaches, [Compaction strategies](/ch4#sec_storage_lsm_compaction), [Disk space usage](/ch4#disk-space-usage) - compare-and-set (CAS), [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set), [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - implementing locks, [Coordination Services](/ch10#sec_consistency_coordination) - implementing uniqueness constraints, [Constraints and uniqueness guarantees](/ch10#sec_consistency_uniqueness) - on object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - relation to consensus, [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable), [Consensus](/ch10#sec_consistency_consensus), [Compare-and-set as consensus](/ch10#compare-and-set-as-consensus) - relation to fencing tokens, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - relation to transactions, [Single-object writes](/ch8#sec_transactions_single_object) - compatibility, [Encoding and Evolution](/ch5#ch_encoding), [Modes of Dataflow](/ch5#sec_encoding_dataflow) - calling services, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - properties of encoding formats, [Summary](/ch5#summary) - using databases, [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db)-[Archival storage](/ch5#archival-storage) - compensating transactions, [Advantages of immutable events](/ch12#sec_stream_immutability_pros), [Loosely interpreted constraints](/ch13#id362) - compilation, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - complex event processing (CEP), [Complex event processing](/ch12#id317) - complexity - distilling in theoretical models, [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - essential and accidental, [Simplicity: Managing Complexity](/ch2#id38) - hiding using abstraction, [Data Models and Query Languages](/ch3#ch_datamodels) - managing, [Simplicity: Managing Complexity](/ch2#id38) - composing data systems (see unbundling databases) - compression - in SSTables, [The SSTable file format](/ch4#the-sstable-file-format) - compute-intensive applications, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - computer games, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - concatenated indexes, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - in hash-sharded systems, [Sharding by hash range](/ch7#sharding-by-hash-range) - concurrency - actor programming model, [Distributed actor frameworks](/ch5#distributed-actor-frameworks), [Event-Driven Architectures and RPC](/ch12#sec_stream_actors_drpc) - (see also event-driven architecture) - bugs from weak transaction isolation, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - conflict resolution, [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts)-[Types of conflict](/ch6#sec_replication_write_conflicts) - definition, [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts) - detecting concurrent writes, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent)-[Version vectors](/ch6#version-vectors) - dual writes, problems with, [Keeping Systems in Sync](/ch12#sec_stream_sync) - happens-before relation, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - in replicated systems, [Problems with Replication Lag](/ch6#sec_replication_lag)-[Version vectors](/ch6#version-vectors), [Linearizability](/ch10#sec_consistency_linearizability)-[Linearizability and network delays](/ch10#linearizability-and-network-delays) - lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update) - multi-version concurrency control (MVCC), [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - optimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - ordering of operations, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - reducing, through event logs, [Concurrency control](/ch12#sec_stream_concurrency), [Dataflow: Interplay between state changes and application code](/ch13#id450) - time and relativity, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - transaction isolation, [Isolation](/ch8#sec_transactions_acid_isolation) - write skew (transaction isolation), [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts) - conditional write, [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set) - in transactions, [Single-object writes](/ch8#sec_transactions_single_object) - on object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - conference management system (example), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - conflict-free replicated datatypes (CRDTs), [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - for leaderless replication, [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship) - preventing lost updates, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - conflicts - avoidance, [Conflict avoidance](/ch6#conflict-avoidance) - causal dependencies, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - conflict detection - in distributed transactions, [Problems with XA transactions](/ch8#problems-with-xa-transactions) - in log-based systems, [Uniqueness constraints require consensus](/ch13#id452) - in serializable snapshot isolation (SSI), [Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - in two-phase commit, [A system of promises](/ch8#a-system-of-promises) - conflict resolution - by aborting transactions, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - by apologizing, [Loosely interpreted constraints](/ch13#id362) - last write wins (LWW), [Timestamps for ordering events](/ch9#sec_distributed_lww) - using atomic operations, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - determining what is a conflict, [Types of conflict](/ch6#sec_replication_write_conflicts), [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - in leaderless replication, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update)-[Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - materializing, [Materializing conflicts](/ch8#materializing-conflicts) - resolution, [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts)-[Types of conflict](/ch6#sec_replication_write_conflicts) - automatic, [Automatic conflict resolution](/ch6#automatic-conflict-resolution) - in leaderless systems, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - last write wins (LWW), [Last write wins (discarding concurrent writes)](/ch6#sec_replication_lww) - using custom logic, [Manual conflict resolution](/ch6#manual-conflict-resolution), [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship) - siblings, [Manual conflict resolution](/ch6#manual-conflict-resolution), [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship) - merging, [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship) - write skew (transaction isolation), [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts) - Confluent - Freight (messaging), [Setting Up New Followers](/ch6#sec_replication_new_replica), [Disk space usage](/ch12#sec_stream_disk_usage) - schema registry, [JSON Schema](/ch5#json-schema), [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - congestion (networks) - avoidance, [The Limitations of TCP](/ch9#sec_distributed_tcp) - limiting accuracy of clocks, [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval) - queueing delays, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - consensus, [Consensus](/ch10#sec_consistency_consensus)-[Summary](/ch10#summary), [Glossary](/glossary) - algorithms, [Consensus](/ch10#sec_consistency_consensus), [Consensus in Practice](/ch10#sec_consistency_total_order) - consensus numbers, [Fetch-and-add as consensus](/ch10#fetch-and-add-as-consensus) - coordination services, [Coordination Services](/ch10#sec_consistency_coordination)-[Service discovery](/ch10#service-discovery) - cost of, [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - impossibility of, [Consensus](/ch10#sec_consistency_consensus) - preventing split brain, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - reconfiguration, [Subtleties of consensus](/ch10#subtleties-of-consensus) - relation to atomic commitment, [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - relation to compare-and-set (CAS), [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable), [Compare-and-set as consensus](/ch10#compare-and-set-as-consensus) - relation to fetch-and-add, [Fetch-and-add as consensus](/ch10#fetch-and-add-as-consensus) - relation to replication, [Using shared logs](/ch10#sec_consistency_smr) - relation to shared logs, [Shared logs as consensus](/ch10#sec_consistency_shared_logs) - relation to uniqueness constraints, [Uniqueness constraints require consensus](/ch13#id452) - safety and liveness properties, [Single-value consensus](/ch10#single-value-consensus) - single-value consensus, [Single-value consensus](/ch10#single-value-consensus) - consent (GDPR), [Consent and Freedom of Choice](/ch14#id375) - consistency, [Consistency](/ch8#sec_transactions_acid_consistency), [Timeliness and Integrity](/ch13#sec_future_integrity) - across different databases, [Leader failure: Failover](/ch6#leader-failure-failover), [Keeping Systems in Sync](/ch12#sec_stream_sync), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views), [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions) - causal, [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix), [Problems with different topologies](/ch6#problems-with-different-topologies), [Ordering events to capture causality](/ch13#sec_future_capture_causality) - consistent prefix reads, [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix)-[Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix) - consistent snapshots, [Setting Up New Followers](/ch6#sec_replication_new_replica), [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation)-[Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner), [Initial snapshot](/ch12#sec_stream_cdc_snapshot), [Creating an index](/ch13#id340) - (see also snapshots) - crash recovery, [Making B-trees reliable](/ch4#sec_storage_btree_wal) - enforcing constraints (see constraints) - eventual, [Problems with Replication Lag](/ch6#sec_replication_lag) - (see also eventual consistency) - in ACID transactions, [Consistency](/ch8#sec_transactions_acid_consistency), [Maintaining integrity in the face of software bugs](/ch13#id455) - in CAP theorem, [The CAP theorem](/ch10#the-cap-theorem) - in leader election, [Subtleties of consensus](/ch10#subtleties-of-consensus) - in microservices, [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems) - linearizability, [Solutions for Replication Lag](/ch6#id131), [Linearizability](/ch10#sec_consistency_linearizability)-[Linearizability and network delays](/ch10#linearizability-and-network-delays) - meanings of, [Consistency](/ch8#sec_transactions_acid_consistency) - monotonic reads, [Monotonic Reads](/ch6#sec_replication_monotonic_reads)-[Monotonic Reads](/ch6#sec_replication_monotonic_reads) - of secondary indexes, [The need for multi-object transactions](/ch8#sec_transactions_need), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation), [Reasoning about dataflows](/ch13#id443), [Creating an index](/ch13#id340) - read-after-write, [Reading Your Own Writes](/ch6#sec_replication_ryw)-[Reading Your Own Writes](/ch6#sec_replication_ryw) - in derived data systems, [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions) - strong (see linearizability) - timeliness and integrity, [Timeliness and Integrity](/ch13#sec_future_integrity) - using quorums, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations), [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - consistent hashing, [Consistent hashing](/ch7#sec_sharding_consistent_hashing) - consistent prefix reads, [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix) - constraints (databases), [Consistency](/ch8#sec_transactions_acid_consistency), [Characterizing write skew](/ch8#characterizing-write-skew) - asynchronously checked, [Loosely interpreted constraints](/ch13#id362) - coordination avoidance, [Coordination-avoiding data systems](/ch13#id454) - ensuring idempotence, [Uniquely identifying requests](/ch13#id355) - in log-based systems, [Enforcing Constraints](/ch13#sec_future_constraints)-[Multi-shard request processing](/ch13#id360) - across multiple shards, [Multi-shard request processing](/ch13#id360) - in two-phase commit, [Distributed Transactions](/ch8#sec_transactions_distributed), [A system of promises](/ch8#a-system-of-promises) - relation to consensus, [Uniqueness constraints require consensus](/ch13#id452) - requiring linearizability, [Constraints and uniqueness guarantees](/ch10#sec_consistency_uniqueness) - Consul (coordination service), [Coordination Services](/ch10#sec_consistency_coordination) - use for service discovery, [Service discovery](/ch10#service-discovery) - consumers (message streams), [Message brokers](/ch5#message-brokers), [Transmitting Event Streams](/ch12#sec_stream_transmit) - backpressure, [Messaging Systems](/ch12#sec_stream_messaging) - consumer groups, [Multiple consumers](/ch12#id298) - consumer offsets in logs, [Consumer offsets](/ch12#sec_stream_log_offsets) - failures, [Acknowledgments and redelivery](/ch12#sec_stream_reordering), [Consumer offsets](/ch12#sec_stream_log_offsets) - fan-out, [Materializing and Updating Timelines](/ch2#sec_introduction_materializing), [Multiple consumers](/ch12#id298), [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging) - load balancing, [Multiple consumers](/ch12#id298), [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging) - not keeping up with producers, [Messaging Systems](/ch12#sec_stream_messaging), [Disk space usage](/ch12#sec_stream_disk_usage), [Making unbundling work](/ch13#sec_future_unbundling_favor) - content models (JSON Schema), [JSON Schema](/ch5#json-schema) - contention - between transactions, [Handling errors and aborts](/ch8#handling-errors-and-aborts) - blocking threads, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - performance of optimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - under two-phase locking, [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - context switches, [Latency and Response Time](/ch2#id23), [Process Pauses](/ch9#sec_distributed_clocks_pauses) - convergence (conflict resolution), [Automatic conflict resolution](/ch6#automatic-conflict-resolution)-[CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - coordination - avoidance, [Coordination-avoiding data systems](/ch13#id454) - cross-datacenter, [The limits of total ordering](/ch13#id335) - cross-region, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - cross-shard ordering, [Sharding](/ch8#sharding), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner), [Using shared logs](/ch10#sec_consistency_smr), [Multi-shard request processing](/ch13#id360) - routing requests to shards, [Request Routing](/ch7#sec_sharding_routing) - services, [Locking and leader election](/ch10#locking-and-leader-election), [Coordination Services](/ch10#sec_consistency_coordination)-[Service discovery](/ch10#service-discovery) - coordinator (in 2PC), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - failure, [Coordinator failure](/ch8#coordinator-failure) - in XA transactions, [XA transactions](/ch8#xa-transactions)-[Problems with XA transactions](/ch8#problems-with-xa-transactions) - recovery, [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - copy-on-write (B-trees), [B-tree variants](/ch4#b-tree-variants), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - CORBA (Common Object Request Broker Architecture), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - coronal mass ejection (see solar storm) - correctness - auditability, [Trust, but Verify](/ch13#sec_future_verification)-[Tools for auditable data systems](/ch13#id366) - Byzantine fault tolerance, [Byzantine Faults](/ch9#sec_distributed_byzantine) - dealing with partial failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - in log-based systems, [Enforcing Constraints](/ch13#sec_future_constraints)-[Multi-shard request processing](/ch13#id360) - of algorithm within system model, [Defining the correctness of an algorithm](/ch9#defining-the-correctness-of-an-algorithm) - of derived data, [Designing for auditability](/ch13#id365) - of immutable data, [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - of personal data, [Responsibility and Accountability](/ch14#id371), [Privacy and Use of Data](/ch14#id457) - of time, [Problems with different topologies](/ch6#problems-with-different-topologies), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy)-[Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - of transactions, [Consistency](/ch8#sec_transactions_acid_consistency), [Aiming for Correctness](/ch13#sec_future_correctness), [Maintaining integrity in the face of software bugs](/ch13#id455) - timeliness and integrity, [Timeliness and Integrity](/ch13#sec_future_integrity)-[Coordination-avoiding data systems](/ch13#id454) - corruption of data - detecting, [The end-to-end argument](/ch13#sec_future_e2e_argument), [Don't just blindly trust what they promise](/ch13#id364)-[Tools for auditable data systems](/ch13#id366) - due to pathological memory access, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - due to radiation, [Byzantine Faults](/ch9#sec_distributed_byzantine) - due to split brain, [Leader failure: Failover](/ch6#leader-failure-failover), [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing) - due to weak transaction isolation, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - integrity as absence of, [Timeliness and Integrity](/ch13#sec_future_integrity) - network packets, [Weak forms of lying](/ch9#weak-forms-of-lying) - on disks, [Durability](/ch8#durability) - preventing using write-ahead logs, [Making B-trees reliable](/ch4#sec_storage_btree_wal) - recovering from, [Batch Processing](/ch11#ch_batch), [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - cosine similarity (semantic search), [Vector Embeddings](/ch4#id92) - Couchbase (database) - document data model, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - durability, [Keeping everything in memory](/ch4#sec_storage_inmemory) - hash sharding, [Fixed number of shards](/ch7#fixed-number-of-shards) - join support, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - rebalancing, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - vBuckets (sharding), [Sharding](/ch7#ch_sharding) - CouchDB (database) - as sync engine, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - B-tree storage, [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - conflict resolution, [Manual conflict resolution](/ch6#manual-conflict-resolution) - coupling (loose and tight), [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability) - covering indexes, [Storing values within the index](/ch4#sec_storage_index_heap) - CozoDB (database), [Datalog: Recursive Relational Queries](/ch3#id62) - CPUs - cache coherence and memory barriers, [Linearizability and network delays](/ch10#linearizability-and-network-delays) - caching and pipelining, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - computing the wrong result, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - SIMD instructions, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - crash-stop and crash-recovery faults, [System Model and Reality](/ch9#sec_distributed_system_model) - CRDTs (see conflict-free replicated datatypes) - CREATE INDEX statement (SQL), [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn), [Creating an index](/ch13#id340) - credit rating agencies, [Responsibility and Accountability](/ch14#id371) - crypto-shredding, [Event Sourcing and CQRS](/ch3#sec_datamodels_events), [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - cryptocurrencies, [Summary](/ch3#summary) - cryptography - defense against attackers, [Byzantine Faults](/ch9#sec_distributed_byzantine) - end-to-end encryption and authentication, [The end-to-end argument](/ch13#sec_future_e2e_argument) - CSV (comma-separated values), [Storage and Indexing for OLTP](/ch4#sec_storage_oltp), [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - Curator (ZooKeeper recipes), [Locking and leader election](/ch10#locking-and-leader-election), [Allocating work to nodes](/ch10#allocating-work-to-nodes) - Cypher (query language), [The Cypher Query Language](/ch3#id57) - comparison to SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language) ### D - Daft (processing framework) - DataFrames, [DataFrames](/ch11#id287) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - Dagster (workflow scheduler), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows), [Batch Processing](/ch11#ch_batch), [Scheduling Workflows](/ch11#sec_batch_workflows) - cloud data warehouse integration, [Query languages](/ch11#sec_batch_query_lanauges) - dashboard (business intelligence), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - Dask (processing framework), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - data catalog, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - data connectors, [Data Warehousing](/ch1#sec_introduction_dwh) - data contracts, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - change data capture, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - data corruption (see corruption of data) - data cubes, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - data engineering, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - data fabric, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - data formats (see encoding) - data infrastructure, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - data integration, [Data Integration](/ch13#sec_future_integration)-[Unifying batch and stream processing](/ch13#id338), [Summary](/ch13#id367) - batch and stream processing, [Batch and Stream Processing](/ch13#sec_future_batch_streaming)-[Unifying batch and stream processing](/ch13#id338) - maintaining derived state, [Maintaining derived state](/ch13#id446) - reprocessing data, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - unifying, [Unifying batch and stream processing](/ch13#id338) - by unbundling databases, [Unbundling Databases](/ch13#sec_future_unbundling)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - comparison to federated databases, [The meta-database of everything](/ch13#id341) - combining tools by deriving data, [Combining Specialized Tools by Deriving Data](/ch13#id442)-[Ordering events to capture causality](/ch13#sec_future_capture_causality) - derived data versus distributed transactions, [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions) - limits of total ordering, [The limits of total ordering](/ch13#id335) - ordering events to capture causality, [Ordering events to capture causality](/ch13#sec_future_capture_causality) - reasoning about dataflows, [Reasoning about dataflows](/ch13#id443) - need for, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - using batch processing, [Batch Processing](/ch11#ch_batch), [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - data lake, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - data lakehouse, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Analytics](/ch11#sec_batch_olap) - data locality (see locality) - data mesh, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - data minimization, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Legislation and Self-Regulation](/ch14#sec_future_legislation) - data models, [Data Models and Query Languages](/ch3#ch_datamodels)-[Summary](/ch3#summary) - DataFrames and arrays, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - graph-like models, [Graph-Like Data Models](/ch3#sec_datamodels_graph)-[GraphQL](/ch3#id63) - Datalog language, [Datalog: Recursive Relational Queries](/ch3#id62)-[Datalog: Recursive Relational Queries](/ch3#id62) - property graphs, [Property Graphs](/ch3#id56) - RDF and triple-stores, [Triple-Stores and SPARQL](/ch3#id59)-[The SPARQL query language](/ch3#the-sparql-query-language) - relational model versus document model, [Relational Model versus Document Model](/ch3#sec_datamodels_history)-[Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - supporting multiple, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - data pipelines, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - data products, [Beyond the data lake](/ch1#beyond-the-data-lake) - data protection regulations (see GDPR) - data residence laws, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - data science, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - data silo, [Data Warehousing](/ch1#sec_introduction_dwh) - data systems - correctness, constraints, and integrity, [Aiming for Correctness](/ch13#sec_future_correctness)-[Tools for auditable data systems](/ch13#id366) - data integration, [Data Integration](/ch13#sec_future_integration)-[Unifying batch and stream processing](/ch13#id338) - goals for using, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - heterogeneous, keeping in sync, [Keeping Systems in Sync](/ch12#sec_stream_sync) - maintainability, [Maintainability](/ch2#sec_introduction_maintainability)-[Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability) - possible faults in, [Transactions](/ch8#ch_transactions) - reliability, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability)-[Humans and Reliability](/ch2#id31) - hardware faults, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - human errors, [Humans and Reliability](/ch2#id31) - importance of, [Humans and Reliability](/ch2#id31) - software faults, [Software faults](/ch2#software-faults) - scalability, [Scalability](/ch2#sec_introduction_scalability)-[Principles for Scalability](/ch2#id35) - unbundling databases, [Unbundling Databases](/ch13#sec_future_unbundling)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - unreliable clocks, [Unreliable Clocks](/ch9#sec_distributed_clocks)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - data warehousing, [Data Warehousing](/ch1#sec_introduction_dwh), [Glossary](/glossary) - cloud-based solutions, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - ETL (extract-transform-load), [Data Warehousing](/ch1#sec_introduction_dwh), [Keeping Systems in Sync](/ch12#sec_stream_sync) - for batch processing, [Batch Processing](/ch11#ch_batch) - keeping data systems in sync, [Keeping Systems in Sync](/ch12#sec_stream_sync) - schema design, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - sharding and clustering, [Sharding by hash range](/ch7#sharding-by-hash-range) - slowly changing dimension (SCD), [Time-dependence of joins](/ch12#sec_stream_join_time) - data-intensive applications, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - database administrator, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - database-internal distributed transactions, [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal), [Atomic commit revisited](/ch12#sec_stream_atomic_commit) - databases - archival storage, [Archival storage](/ch5#archival-storage) - comparison of message brokers to, [Message brokers compared to databases](/ch12#id297) - dataflow through, [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db) - end-to-end argument for, [The end-to-end argument](/ch13#sec_future_e2e_argument)-[Applying end-to-end thinking in data systems](/ch13#id357) - checking integrity, [The end-to-end argument again](/ch13#id456) - relation to event streams, [Databases and Streams](/ch12#sec_stream_databases)-[Limitations of immutability](/ch12#sec_stream_immutability_limitations) - (see also changelogs) - API support for change streams, [API support for change streams](/ch12#sec_stream_change_api), [Separation of application code and state](/ch13#id344) - change data capture, [Change Data Capture](/ch12#sec_stream_cdc)-[API support for change streams](/ch12#sec_stream_change_api) - event sourcing, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - keeping systems in sync, [Keeping Systems in Sync](/ch12#sec_stream_sync)-[Keeping Systems in Sync](/ch12#sec_stream_sync) - philosophy of immutable events, [State, Streams, and Immutability](/ch12#sec_stream_immutability)-[Limitations of immutability](/ch12#sec_stream_immutability_limitations) - unbundling, [Unbundling Databases](/ch13#sec_future_unbundling)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - composing data storage technologies, [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - designing applications around dataflow, [Designing Applications Around Dataflow](/ch13#sec_future_dataflow)-[Stream processors and services](/ch13#id345) - observing derived state, [Observing Derived State](/ch13#sec_future_observing)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - datacenters - failures of, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - geographically distributed (see regions (geographic distribution)) - multitenancy and shared resources, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - network architecture, [Cloud Computing Versus Supercomputing](/ch1#id17) - network faults, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - dataflow, [Modes of Dataflow](/ch5#sec_encoding_dataflow)-[Distributed actor frameworks](/ch5#distributed-actor-frameworks), [Designing Applications Around Dataflow](/ch13#sec_future_dataflow)-[Stream processors and services](/ch13#id345) - correctness of dataflow systems, [Correctness of dataflow systems](/ch13#id453) - dataflow engines, [Dataflow Engines](/ch11#sec_batch_dataflow) - comparison to stream processing, [Processing Streams](/ch12#sec_stream_processing) - DataFrames, [DataFrames](/ch11#id287) - support in batch processing frameworks, [Batch Processing](/ch11#ch_batch) - event-driven, [Event-Driven Architectures](/ch5#sec_encoding_dataflow_msg)-[Distributed actor frameworks](/ch5#distributed-actor-frameworks) - reasoning about, [Reasoning about dataflows](/ch13#id443) - through databases, [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db) - through services, [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc)-[Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - workflow engines (see workflow engines) - DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - implementation, [DataFrames](/ch11#id287) - in batch processing, [DataFrames](/ch11#id287) - in notebooks, [Machine Learning](/ch11#id290) - support in batch processing frameworks, [Batch Processing](/ch11#ch_batch) - DataFusion (query engine), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - Datalog (query language), [Datalog: Recursive Relational Queries](/ch3#id62)-[Datalog: Recursive Relational Queries](/ch3#id62) - Datastream (change data capture), [API support for change streams](/ch12#sec_stream_change_api) - datatypes - binary strings in XML and JSON, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - conflict-free, [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - in Avro encodings, [Avro](/ch5#sec_encoding_avro) - in Protocol Buffers, [Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - numbers in XML and JSON, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - Datensparsamkeit, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - Datomic (database) - B-tree storage, [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - data model, [Graph-Like Data Models](/ch3#sec_datamodels_graph), [Triple-Stores and SPARQL](/ch3#id59) - Datalog query language, [Datalog: Recursive Relational Queries](/ch3#id62) - excision (deleting data), [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - languages for transactions, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - serial execution of transactions, [Actual Serial Execution](/ch8#sec_transactions_serial) - Daylight Saving Time (DST), [Time-of-day clocks](/ch9#time-of-day-clocks) - Db2 (database) - change data capture, [Implementing change data capture](/ch12#id307) - DBA (database administrator), [Operations in the Cloud Era](/ch1#sec_introduction_operations) - deadlocks, [Explicit locking](/ch8#explicit-locking) - detection, in distributed transaction, [Problems with XA transactions](/ch8#problems-with-xa-transactions) - in two-phase locking (2PL), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - Debezium (change data capture), [Implementing change data capture](/ch12#id307) - Cassandra, [API support for change streams](/ch12#sec_stream_change_api) - for data integration, [Unbundled versus integrated systems](/ch13#id448) - declarative languages, [Data Models and Query Languages](/ch3#ch_datamodels), [Glossary](/glossary) - and sync engines, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - Datalog, [Datalog: Recursive Relational Queries](/ch3#id62) - in document databases, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - recursive SQL queries, [Graph Queries in SQL](/ch3#id58) - SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language) - DeepSeek - 3FS (see 3FS) - delays - bounded network delays, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - bounded process pauses, [Response time guarantees](/ch9#sec_distributed_clocks_realtime) - unbounded network delays, [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - unbounded process pauses, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - deleting data, [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - in LSM storage, [Disk space usage](/ch4#disk-space-usage) - legal basis, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - Delta Lake (table format), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - sharding and clustering, [Sharding by hash range](/ch7#sharding-by-hash-range) - demilitarized zone (networking), [Serving Derived Data](/ch11#sec_batch_serving_derived) - denormalization (data representation), [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization)-[Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many), [Glossary](/glossary) - in derived data systems, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - in event sourcing/CQRS, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - in social network case study, [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - materialized views, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - updating derived data, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object), [The need for multi-object transactions](/ch8#sec_transactions_need), [Combining Specialized Tools by Deriving Data](/ch13#id442) - versus normalization, [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - derived data, [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Stream Processing](/ch12#ch_stream), [Glossary](/glossary) - batch processing, [Batch Processing](/ch11#ch_batch) - event sourcing and CQRS, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - from change data capture, [Implementing change data capture](/ch12#id307) - maintaining derived state through logs, [Databases and Streams](/ch12#sec_stream_databases)-[API support for change streams](/ch12#sec_stream_change_api), [State, Streams, and Immutability](/ch12#sec_stream_immutability)-[Concurrency control](/ch12#sec_stream_concurrency) - observing, by subscribing to streams, [End-to-end event streams](/ch13#id349) - outputs of batch and stream processing, [Batch and Stream Processing](/ch13#sec_future_batch_streaming) - through application code, [Application code as a derivation function](/ch13#sec_future_dataflow_derivation) - versus distributed transactions, [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions) - design patterns, [Simplicity: Managing Complexity](/ch2#id38) - deterministic operations, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs), [Faults and Partial Failures](/ch9#sec_distributed_partial_failure), [Glossary](/glossary) - and idempotence, [Idempotence](/ch12#sec_stream_idempotence), [Reasoning about dataflows](/ch13#id443) - computing derived data, [Maintaining derived state](/ch13#id446), [Correctness of dataflow systems](/ch13#id453), [Designing for auditability](/ch13#id365) - in event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - in state machine replication, [Using shared logs](/ch10#sec_consistency_smr), [Databases and Streams](/ch12#sec_stream_databases) - in statement-based replication, [Statement-based replication](/ch6#statement-based-replication) - in testing, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - joins, [Time-dependence of joins](/ch12#sec_stream_join_time) - making code deterministic, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - overview, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - deterministic simulation testing (DST), [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - DevOps, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - dimension tables, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - dimensional modeling (see star schemas) - directed acyclic graphs (DAG) - workflows, [Scheduling Workflows](/ch11#sec_batch_workflows) - (see also workflow engines) - dirty reads (transaction isolation), [No dirty reads](/ch8#no-dirty-reads) - dirty writes (transaction isolation), [No dirty writes](/ch8#sec_transactions_dirty_write) - disaggregation - of storage and compute, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - Discord (group chat) - GraphQL example, [GraphQL](/ch3#id63) - discrimination, [Bias and Discrimination](/ch14#id370) - disks (see hard disks) - distributed actor frameworks, [Distributed actor frameworks](/ch5#distributed-actor-frameworks) - distributed filesystems, [Distributed Filesystems](/ch11#sec_batch_dfs)-[Distributed Filesystems](/ch11#sec_batch_dfs) - comparison to object storage, [Object Stores](/ch11#id277) - use by Flink, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - distributed ledgers, [Summary](/ch3#summary) - distributed systems, [The Trouble with Distributed Systems](/ch9#ch_distributed)-[Summary](/ch9#summary), [Glossary](/glossary) - Byzantine faults, [Byzantine Faults](/ch9#sec_distributed_byzantine)-[Weak forms of lying](/ch9#weak-forms-of-lying) - detecting network faults, [Detecting Faults](/ch9#id307) - faults and partial failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - formalization of consensus, [Single-value consensus](/ch10#single-value-consensus) - impossibility results, [The CAP theorem](/ch10#the-cap-theorem), [Consensus](/ch10#sec_consistency_consensus) - issues with failover, [Leader failure: Failover](/ch6#leader-failure-failover) - multi-region (see regions (geographic distribution)) - network problems, [Unreliable Networks](/ch9#sec_distributed_networks)-[Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - problems with, [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems) - quorums, relying on, [The Majority Rules](/ch9#sec_distributed_majority) - reasons for using, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Replication](/ch6#ch_replication) - synchronized clocks, relying on, [Relying on Synchronized Clocks](/ch9#sec_distributed_clocks_relying)-[Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - system models, [System Model and Reality](/ch9#sec_distributed_system_model)-[Deterministic simulation testing](/ch9#deterministic-simulation-testing) - use of clocks and time, [Unreliable Clocks](/ch9#sec_distributed_clocks) - distributed transactions (see transactions) - Django (web framework), [Handling errors and aborts](/ch8#handling-errors-and-aborts) - DMZ (demilitarized zone), [Serving Derived Data](/ch11#sec_batch_serving_derived) - DNS (Domain Name System), [Request Routing](/ch7#sec_sharding_routing), [Service discovery](/ch10#service-discovery) - for load balancing, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - Docker (container manager), [Separation of application code and state](/ch13#id344) - document data model, [Relational Model versus Document Model](/ch3#sec_datamodels_history)-[Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - comparison to relational model, [When to Use Which Model](/ch3#sec_datamodels_document_summary)-[Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - multi-object transactions, need for, [The need for multi-object transactions](/ch8#sec_transactions_need) - sharded secondary indexes, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes) - versus relational model - convergence of models, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - data locality, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - document-partitioned indexes (see local secondary indexes) - domain-driven design (DDD), [Simplicity: Managing Complexity](/ch2#id38), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - dotted version vectors, [Version vectors](/ch6#version-vectors) - double-entry bookkeeping, [Summary](/ch3#summary) - DRBD (Distributed Replicated Block Device), [Single-Leader Replication](/ch6#sec_replication_leader) - drift (clocks), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - Druid (database), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Column-Oriented Storage](/ch4#sec_storage_column), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - handling writes, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - pre-aggregation, [Analytics](/ch11#sec_batch_olap) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - Dryad (dataflow engine), [Dataflow Engines](/ch11#sec_batch_dataflow) - dual writes, problems with, [Keeping Systems in Sync](/ch12#sec_stream_sync) - DuckDB (database), [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems), [Compaction strategies](/ch4#sec_storage_lsm_compaction) - column-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column) - use for ETL, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - duplicates, suppression of, [Duplicate suppression](/ch13#id354) - (see also idempotence) - using a unique ID, [Uniquely identifying requests](/ch13#id355), [Multi-shard request processing](/ch13#id360) - durability (transactions), [Making B-trees reliable](/ch4#sec_storage_btree_wal), [Durability](/ch8#durability), [Glossary](/glossary) - durable execution, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - Restate (see Restate (workflow engine)) - Temporal (see Temporal (workflow engine)) - durable functions (see workflow engines) - duration (time), [Unreliable Clocks](/ch9#sec_distributed_clocks) - measurement with monotonic clocks, [Monotonic clocks](/ch9#monotonic-clocks) - dynamically typed languages - analogy to schema-on-read, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - Dynamo (database), [Leaderless Replication](/ch6#sec_replication_leaderless) - Dynamo-style databases (see leaderless replication) - DynamoDB (database) - auto-scaling, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - hash-range sharding, [Sharding by hash range](/ch7#sharding-by-hash-range) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - sharded secondary indexes, [Global Secondary Indexes](/ch7#id167) ### E - EBS (virtual block device), [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - compared to object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - ECC (see error-correcting codes) - EDB Postgres Distributed (database), [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - edges (in graphs), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - property graph model, [Property Graphs](/ch3#id56) - edit distance (full-text search), [Full-Text Search](/ch4#sec_storage_full_text) - effectively-once semantics, [Fault Tolerance](/ch12#sec_stream_fault_tolerance), [Exactly-once execution of an operation](/ch13#id353) - (see also exactly-once semantics) - preservation of integrity, [Correctness of dataflow systems](/ch13#id453) - Elastic Compute Cloud (EC2) - spot instances, [Handling Faults](/ch11#id281) - elasticity, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - cloud data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Query languages](/ch11#sec_batch_query_lanauges) - Elasticsearch (search server) - local secondary indexes, [Local Secondary Indexes](/ch7#id166) - percolator (stream search), [Search on streams](/ch12#id320) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - shard rebalancing, [Fixed number of shards](/ch7#fixed-number-of-shards) - use of Lucene, [Full-Text Search](/ch4#sec_storage_full_text) - Elm (programming language), [End-to-end event streams](/ch13#id349) - ELT (extract-load-transform), [Data Warehousing](/ch1#sec_introduction_dwh) - relation to batch processing, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - embarassingly parallel (algorithms) - ETL (see ETL (extract-transform-load)) - MapReduce, [MapReduce](/ch11#sec_batch_mapreduce) - (see also MapReduce) - embedded storage engines, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - embedding (vector), [Vector Embeddings](/ch4#id92) - encodings (data formats), [Encoding and Evolution](/ch5#ch_encoding)-[The Merits of Schemas](/ch5#sec_encoding_schemas) - Avro, [Avro](/ch5#sec_encoding_avro)-[Dynamically generated schemas](/ch5#dynamically-generated-schemas) - binary variants of JSON and XML, [Binary encoding](/ch5#binary-encoding) - compatibility, [Encoding and Evolution](/ch5#ch_encoding) - calling services, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - using databases, [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db)-[Archival storage](/ch5#archival-storage) - defined, [Formats for Encoding Data](/ch5#sec_encoding_formats) - JSON, XML, and CSV, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - language-specific formats, [Language-Specific Formats](/ch5#id96) - merits of schemas, [The Merits of Schemas](/ch5#sec_encoding_schemas) - Protocol Buffers, [Protocol Buffers](/ch5#sec_encoding_protobuf)-[Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - representations of data, [Formats for Encoding Data](/ch5#sec_encoding_formats) - end-to-end argument, [The end-to-end argument](/ch13#sec_future_e2e_argument)-[Applying end-to-end thinking in data systems](/ch13#id357) - checking integrity, [The end-to-end argument again](/ch13#id456) - publish/subscribe streams, [End-to-end event streams](/ch13#id349) - enrichment (stream), [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - Enterprise JavaBeans (EJB), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - enterprise software, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - entities (see vertices) - ephemeral storage, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - epoch (consensus algorithms), [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - epoch (Unix timestamps), [Time-of-day clocks](/ch9#time-of-day-clocks) - erasure coding (error correction), [Distributed Filesystems](/ch11#sec_batch_dfs) - error handling - for network faults, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - in transactions, [Handling errors and aborts](/ch8#handling-errors-and-aborts) - error-correcting codes, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults), [Distributed Filesystems](/ch11#sec_batch_dfs) - Esper (CEP engine), [Complex event processing](/ch12#id317) - essential complexity, [Simplicity: Managing Complexity](/ch2#id38) - etcd (coordination service), [Coordination Services](/ch10#sec_consistency_coordination)-[Service discovery](/ch10#service-discovery) - generating fencing tokens, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens), [Coordination Services](/ch10#sec_consistency_coordination) - linearizable operations, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable), [Subtleties of consensus](/ch10#subtleties-of-consensus) - locks and leader election, [Locking and leader election](/ch10#locking-and-leader-election) - use for service discovery, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery), [Service discovery](/ch10#service-discovery) - use for shard assignment, [Request Routing](/ch7#sec_sharding_routing) - use of Raft algorithm, [Single-Leader Replication](/ch6#sec_replication_leader) - Ethereum (blockchain), [Tools for auditable data systems](/ch13#id366) - Ethernet (networks), [Cloud Computing Versus Supercomputing](/ch1#id17), [Unreliable Networks](/ch9#sec_distributed_networks), [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - packet checksums, [Weak forms of lying](/ch9#weak-forms-of-lying), [The end-to-end argument](/ch13#sec_future_e2e_argument) - ethics, [Doing the Right Thing](/ch14)-[Legislation and Self-Regulation](/ch14#sec_future_legislation) - code of ethics and professional practice, [Doing the Right Thing](/ch14) - legislation and self-regulation, [Legislation and Self-Regulation](/ch14#sec_future_legislation) - predictive analytics, [Predictive Analytics](/ch14#id369)-[Feedback Loops](/ch14#id372) - amplifying bias, [Bias and Discrimination](/ch14#id370) - feedback loops, [Feedback Loops](/ch14#id372) - privacy and tracking, [Privacy and Tracking](/ch14#id373)-[Legislation and Self-Regulation](/ch14#sec_future_legislation) - consent and freedom of choice, [Consent and Freedom of Choice](/ch14#id375) - data as assets and power, [Data as Assets and Power](/ch14#id376) - meaning of privacy, [Privacy and Use of Data](/ch14#id457) - surveillance, [Surveillance](/ch14#id374) - respect, dignity, and agency, [Legislation and Self-Regulation](/ch14#sec_future_legislation) - unintended consequences, [Doing the Right Thing](/ch14), [Feedback Loops](/ch14#id372) - ETL (extract-transform-load), [Data Warehousing](/ch1#sec_introduction_dwh), [Keeping Systems in Sync](/ch12#sec_stream_sync), [Glossary](/glossary) - relation to batch processing, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage)-[Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - using batch processing, [Batch Processing](/ch11#ch_batch) - Euclidean distance (semantic search), [Vector Embeddings](/ch4#id92) - European Union - AI Act (see AI Act) - GDPR (see GDPR) - event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events)-[Event Sourcing and CQRS](/ch3#sec_datamodels_events) - and change data capture, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - comparison to change data capture, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - immutability and auditability, [State, Streams, and Immutability](/ch12#sec_stream_immutability), [Designing for auditability](/ch13#id365) - large, reliable data systems, [Uniquely identifying requests](/ch13#id355), [Correctness of dataflow systems](/ch13#id453) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - event streams (see streams) - event-driven architecture, [Event-Driven Architectures](/ch5#sec_encoding_dataflow_msg)-[Distributed actor frameworks](/ch5#distributed-actor-frameworks) - distributed actor frameworks, [Distributed actor frameworks](/ch5#distributed-actor-frameworks) - events, [Transmitting Event Streams](/ch12#sec_stream_transmit) - deciding on total order of, [The limits of total ordering](/ch13#id335) - deriving views from event log, [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - event time versus processing time, [Event time versus processing time](/ch12#id322), [Microbatching and checkpointing](/ch12#id329), [Unifying batch and stream processing](/ch13#id338) - immutable, advantages of, [Advantages of immutable events](/ch12#sec_stream_immutability_pros), [Designing for auditability](/ch13#id365) - ordering to capture causality, [Ordering events to capture causality](/ch13#sec_future_capture_causality) - reads as, [Reads are events too](/ch13#sec_future_read_events) - stragglers, [Handling straggler events](/ch12#id323) - timestamp of, in stream processing, [Whose clock are you using, anyway?](/ch12#id438) - EventSource (browser API), [Pushing state changes to clients](/ch13#id348) - EventStoreDB (database), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - eventual consistency, [Replication](/ch6#ch_replication), [Problems with Replication Lag](/ch6#sec_replication_lag), [Safety and liveness](/ch9#sec_distributed_safety_liveness) - (see also conflicts) - and perpetual inconsistency, [Timeliness and Integrity](/ch13#sec_future_integrity) - strong eventual consistency, [Automatic conflict resolution](/ch6#automatic-conflict-resolution) - evidence - data used as, [Humans and Reliability](/ch2#id31) - evolvability, [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability), [Encoding and Evolution](/ch5#ch_encoding) - calling services, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - graph-structured data, [Property Graphs](/ch3#id56) - of databases, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility), [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db)-[Archival storage](/ch5#archival-storage), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views), [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - reprocessing data, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing), [Unifying batch and stream processing](/ch13#id338) - schema evolution in Avro, [The writer's schema and the reader's schema](/ch5#the-writers-schema-and-the-readers-schema) - schema evolution in Protocol Buffers, [Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - schema-on-read, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility), [Encoding and Evolution](/ch5#ch_encoding), [The Merits of Schemas](/ch5#sec_encoding_schemas) - exactly-once semantics, [Exactly-once message processing](/ch8#sec_transactions_exactly_once), [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited), [Fault Tolerance](/ch12#sec_stream_fault_tolerance), [Exactly-once execution of an operation](/ch13#id353) - parity with batch processors, [Unifying batch and stream processing](/ch13#id338) - preservation of integrity, [Correctness of dataflow systems](/ch13#id453) - using durable execution, [Durable execution](/ch5#durable-execution) - exclusive mode (locks), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - exponential backoff, [Describing Performance](/ch2#sec_introduction_percentiles), [Handling errors and aborts](/ch8#handling-errors-and-aborts) - ext4 (file system), [Distributed Filesystems](/ch11#sec_batch_dfs) - eXtended Architecture transactions (see XA transactions) - extract-transform-load (see ETL) ### F - Facebook - Faiss (vector index), [Vector Embeddings](/ch4#id92) - React (user interface library), [End-to-end event streams](/ch13#id349) - social graphs, [Graph-Like Data Models](/ch3#sec_datamodels_graph) - facts - fact table (star schema), [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - in Datalog, [Datalog: Recursive Relational Queries](/ch3#id62) - in event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - fail-slow faults, [System Model and Reality](/ch9#sec_distributed_system_model) - fail-stop model, [System Model and Reality](/ch9#sec_distributed_system_model) - failover, [Leader failure: Failover](/ch6#leader-failure-failover), [Glossary](/glossary) - (see also leader-based replication) - in leaderless replication, absence of, [Writing to the Database When a Node Is Down](/ch6#id287) - leader election, [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing), [Consensus](/ch10#sec_consistency_consensus), [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - potential problems, [Leader failure: Failover](/ch6#leader-failure-failover) - failures - amplification by distributed transactions, [Maintaining derived state](/ch13#id446) - failure detection, [Detecting Faults](/ch9#id307) - automatic rebalancing causing cascading failures, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - timeouts and unbounded delays, [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing), [Network congestion and queueing](/ch9#network-congestion-and-queueing) - using a coordination service, [Coordination Services](/ch10#sec_consistency_coordination) - faults versus, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability) - partial failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure), [Summary](/ch9#summary) - Faiss (vector index), [Vector Embeddings](/ch4#id92) - false positive (Bloom filters), [Bloom filters](/ch4#bloom-filters) - fan-out (messaging systems), [Materializing and Updating Timelines](/ch2#sec_introduction_materializing), [Multiple consumers](/ch12#id298) - fault injection, [Fault Tolerance](/ch2#id27), [Network Faults in Practice](/ch9#sec_distributed_network_faults), [Fault injection](/ch9#sec_fault_injection) - fault isolation, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - fault tolerance, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability)-[Humans and Reliability](/ch2#id31), [Glossary](/glossary) - formalization in consensus, [Single-value consensus](/ch10#single-value-consensus) - human fault tolerance, [Batch Processing](/ch11#ch_batch) - in batch processing, [Handling Faults](/ch11#id281) - in log-based systems, [Applying end-to-end thinking in data systems](/ch13#id357), [Timeliness and Integrity](/ch13#sec_future_integrity)-[Correctness of dataflow systems](/ch13#id453) - in stream processing, [Fault Tolerance](/ch12#sec_stream_fault_tolerance)-[Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - atomic commit, [Atomic commit revisited](/ch12#sec_stream_atomic_commit) - idempotence, [Idempotence](/ch12#sec_stream_idempotence) - maintaining derived state, [Maintaining derived state](/ch13#id446) - microbatching and checkpointing, [Microbatching and checkpointing](/ch12#id329) - rebuilding state after a failure, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - of distributed transactions, [XA transactions](/ch8#xa-transactions)-[Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - of leader-based and leaderless replication, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - transaction atomicity, [Atomicity](/ch8#sec_transactions_acid_atomicity), [Distributed Transactions](/ch8#sec_transactions_distributed)-[Exactly-once message processing](/ch8#sec_transactions_exactly_once) - faults - Byzantine faults, [Byzantine Faults](/ch9#sec_distributed_byzantine)-[Weak forms of lying](/ch9#weak-forms-of-lying) - failures versus, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability) - handled by transactions, [Transactions](/ch8#ch_transactions) - handling in supercomputers and cloud computing, [Cloud Computing Versus Supercomputing](/ch1#id17) - hardware, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - in distributed systems, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - introducing deliberately (see fault injection) - network faults, [Network Faults in Practice](/ch9#sec_distributed_network_faults)-[Detecting Faults](/ch9#id307) - asymmetric faults, [The Majority Rules](/ch9#sec_distributed_majority) - detecting, [Detecting Faults](/ch9#id307) - tolerance of, in multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - software faults, [Software faults](/ch2#software-faults) - tolerating (see fault tolerance) - feature engineering (machine learning), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - federated databases, [The meta-database of everything](/ch13#id341) - Feldera (database) - incremental view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - fence (CPU instruction), [Linearizability and network delays](/ch10#linearizability-and-network-delays) - fencing (preventing split brain), [Leader failure: Failover](/ch6#leader-failure-failover), [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens)-[Fencing with multiple replicas](/ch9#fencing-with-multiple-replicas) - generating fencing tokens, [Using shared logs](/ch10#sec_consistency_smr), [Coordination Services](/ch10#sec_consistency_coordination) - properties of fencing tokens, [Defining the correctness of an algorithm](/ch9#defining-the-correctness-of-an-algorithm) - stream processors writing to databases, [Idempotence](/ch12#sec_stream_idempotence), [Exactly-once execution of an operation](/ch13#id353) - fetch-and-add - relation to consensus, [Fetch-and-add as consensus](/ch10#fetch-and-add-as-consensus) - Fibre Channel (networks), [Distributed Filesystems](/ch11#sec_batch_dfs) - field tags (Protocol Buffers), [Protocol Buffers](/ch5#sec_encoding_protobuf)-[Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - Figma (graphics software), [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - filesystem in userspace (FUSE), [Setting Up New Followers](/ch6#sec_replication_new_replica), [Distributed Filesystems](/ch11#sec_batch_dfs) - on object storage, [Object Stores](/ch11#id277) - financial data - accounting ledgers, [Summary](/ch3#summary) - immutability, [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - time series data, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - Fivetran, [Data Warehousing](/ch1#sec_introduction_dwh) - FizzBee (specification language), [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - flat index (vector index), [Vector Embeddings](/ch4#id92) - FlatBuffers (data format), [Formats for Encoding Data](/ch5#sec_encoding_formats) - Flink (processing framework), [Batch Processing](/ch11#ch_batch), [Dataflow Engines](/ch11#sec_batch_dataflow) - cost efficiency, [Query languages](/ch11#sec_batch_query_lanauges) - DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes), [DataFrames](/ch11#id287) - fault tolerance, [Handling Faults](/ch11#id281), [Microbatching and checkpointing](/ch12#id329), [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - FlinkML, [Machine Learning](/ch11#id290) - for data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - high availability using ZooKeeper, [Coordination Services](/ch10#sec_consistency_coordination) - integration of batch and stream processing, [Unifying batch and stream processing](/ch13#id338) - query optimizer, [Query languages](/ch11#sec_batch_query_lanauges) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - stream processing, [Stream analytics](/ch12#id318) - streaming SQL support, [Complex event processing](/ch12#id317) - flow control, [The Limitations of TCP](/ch9#sec_distributed_tcp), [Messaging Systems](/ch12#sec_stream_messaging), [Glossary](/glossary) - FLP result (on consensus), [Consensus](/ch10#sec_consistency_consensus) - Flyte (workflow scheduler), [Machine Learning](/ch11#id290) - followers, [Single-Leader Replication](/ch6#sec_replication_leader), [Glossary](/glossary) - (see also leader-based replication) - formal methods, [Formal Methods and Randomized Testing](/ch9#sec_distributed_formal)-[Deterministic simulation testing](/ch9#deterministic-simulation-testing) - forward compatibility, [Encoding and Evolution](/ch5#ch_encoding) - forward decay (algorithm), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - Fossil (version control system), [Concurrency control](/ch12#sec_stream_concurrency) - shunning (deleting data), [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - FoundationDB (database) - consistency model, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - deterministic simulation testing, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - key-range sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - process-per-core model, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - serializable transactions, [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi), [Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - transactions, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - fractional indexing, [When to Use Which Model](/ch3#sec_datamodels_document_summary) - fragmentation (of B-trees), [Disk space usage](/ch4#disk-space-usage) - frame (computer graphics), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - frontend (web development), [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - FrostDB (database) - deterministic simulation testing (DST), [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - fsync (system call), [Making B-trees reliable](/ch4#sec_storage_btree_wal), [Durability](/ch8#durability) - full-text search, [Full-Text Search](/ch4#sec_storage_full_text), [Glossary](/glossary) - and fuzzy indexes, [Full-Text Search](/ch4#sec_storage_full_text) - Lucene storage engine, [Full-Text Search](/ch4#sec_storage_full_text) - sharded indexes, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes) - Function as a Service (FaaS), [Microservices and Serverless](/ch1#sec_introduction_microservices) - functional programming - inspiration for MapReduce, [MapReduce](/ch11#sec_batch_mapreduce) - functional requirements, [Defining Nonfunctional Requirements](/ch2#ch_nonfunctional) - FUSE (see filesystem in userspace (FUSE)) - fuzzing, [Formal Methods and Randomized Testing](/ch9#sec_distributed_formal) - fuzzy search (see similarity search) ### G - Gallina (specification language), [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - game development, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - garbage collection - immutability and, [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - process pauses for, [Latency and Response Time](/ch2#id23), [Process Pauses](/ch9#sec_distributed_clocks_pauses)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact), [The Majority Rules](/ch9#sec_distributed_majority) - (see also process pauses) - gas stations algorithmic pricing, [Feedback Loops](/ch14#id372) - GDPR (regulation), [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - consent, [Consent and Freedom of Choice](/ch14#id375) - data minimization, [Legislation and Self-Regulation](/ch14#sec_future_legislation) - legitimate interest, [Consent and Freedom of Choice](/ch14#id375) - right of access, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - right to erasure, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Disk space usage](/ch4#disk-space-usage), [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - GenBank (genome database), [Summary](/ch3#summary) - General Data Protection Regulation (see GDPR (regulation)) - genome analysis, [Summary](/ch3#summary) - geographic distribution (see regions (geographic distribution)) - geospatial indexes, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - Git (version control system), [Concurrency control](/ch12#sec_stream_concurrency) - local-first software, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - merge conflicts, [Manual conflict resolution](/ch6#manual-conflict-resolution) - GitHub, postmortems, [Leader failure: Failover](/ch6#leader-failure-failover), [Leader failure: Failover](/ch6#leader-failure-failover), [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - global secondary indexes, [Global Secondary Indexes](/ch7#id167), [Summary](/ch7#summary) - globally unique identifiers (see UUIDs) - GlusterFS (distributed filesystem), [Batch Processing](/ch11#ch_batch), [Distributed Filesystems](/ch11#sec_batch_dfs), [Object Stores](/ch11#id277) - GNU Coreutils (Linux), [Sorting Versus In-memory Aggregation](/ch11#id275) - Go (programming language) - garbage collection, [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - GoldenGate (change data capture), [Implementing change data capture](/ch12#id307) - (see also Oracle) - Google - BigQuery (see BigQuery (database)) - Bigtable (see Bigtable (database)) - Chubby (lock service), [Coordination Services](/ch10#sec_consistency_coordination) - Cloud Storage (object storage), [Setting Up New Followers](/ch6#sec_replication_new_replica), [Object Stores](/ch11#id277) - request preconditions, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - Compute Engine - preemptible instances, [Handling Faults](/ch11#id281) - Dataflow (stream processing) - data warehouse integration, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - Dataflow (stream processor), [Stream analytics](/ch12#id318), [Atomic commit revisited](/ch12#sec_stream_atomic_commit), [Unifying batch and stream processing](/ch13#id338) - (see also Beam) - Datastream (change data capture), [API support for change streams](/ch12#sec_stream_change_api) - Docs (collaborative editor), [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps), [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - operational transformation, [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - Dremel (query engine), [Column-Oriented Storage](/ch4#sec_storage_column) - Firestore (database), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - MapReduce (batch processing), [Batch Processing](/ch11#ch_batch) - (see also MapReduce) - Percolator (transaction system), [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - persistent disks (cloud service), [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - Pub/Sub (messaging), [Message brokers](/ch5#message-brokers), [Message brokers compared to databases](/ch12#id297), [Using logs for message storage](/ch12#id300) - response time study, [Average, Median, and Percentiles](/ch2#id24) - Sheets (collaborative spreadsheet), [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps), [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - Spanner (see Spanner (database)) - TrueTime (clock API), [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval) - gossip protocol, [Request Routing](/ch7#sec_sharding_routing) - governance, [Beyond the data lake](/ch1#beyond-the-data-lake) - government use of data, [Data as Assets and Power](/ch14#id376) - GPS (Global Positioning System) - use for clock synchronization, [Unreliable Clocks](/ch9#sec_distributed_clocks), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy), [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - GPT (language model), [Vector Embeddings](/ch4#id92) - GPU (graphics processing unit), [Layering of cloud services](/ch1#layering-of-cloud-services), [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - gradual rollout (see rolling upgrades) - GraphQL (query language), [GraphQL](/ch3#id63) - validation, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - graphs, [Glossary](/glossary) - as data models, [Graph-Like Data Models](/ch3#sec_datamodels_graph)-[GraphQL](/ch3#id63) - property graphs, [Property Graphs](/ch3#id56) - RDF and triple-stores, [Triple-Stores and SPARQL](/ch3#id59)-[The SPARQL query language](/ch3#the-sparql-query-language) - DAGs (see directed acyclic graphs) - processing and analysis, [Machine Learning](/ch11#id290) - query languages - Cypher, [The Cypher Query Language](/ch3#id57) - Datalog, [Datalog: Recursive Relational Queries](/ch3#id62)-[Datalog: Recursive Relational Queries](/ch3#id62) - GraphQL, [GraphQL](/ch3#id63) - Gremlin, [Graph-Like Data Models](/ch3#sec_datamodels_graph) - recursive SQL queries, [Graph Queries in SQL](/ch3#id58) - SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language)-[The SPARQL query language](/ch3#the-sparql-query-language) - traversal, [Property Graphs](/ch3#id56) - gray failures, [System Model and Reality](/ch9#sec_distributed_system_model) - in leaderless replication, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - Gremlin (graph query language), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - grep (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis) - gRPC (service calls), [Microservices and Serverless](/ch1#sec_introduction_microservices), [Web services](/ch5#sec_web_services) - forward and backward compatibility, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - GUIDs (see UUIDs) ### H - Hadoop (data infrastructure) - comparison to distributed databases, [Batch Processing](/ch11#ch_batch) - MapReduce (see MapReduce) - NodeManager, [Distributed Job Orchestration](/ch11#id278) - YARN (see YARN (job scheduler)) - HANA (see SAP HANA (database)) - happens-before relation, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - hard disks - access patterns, [Sequential versus random writes](/ch4#sidebar_sequential) - detecting corruption, [The end-to-end argument](/ch13#sec_future_e2e_argument), [Don't just blindly trust what they promise](/ch13#id364) - faults in, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults), [Durability](/ch8#durability) - sequential vs. random writes, [Sequential versus random writes](/ch4#sidebar_sequential) - sequential write throughput, [Disk space usage](/ch12#sec_stream_disk_usage) - hardware faults, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - hash function - in Bloom filters, [Bloom filters](/ch4#bloom-filters) - hash join - in stream processing, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - hash sharding, [Sharding by Hash of Key](/ch7#sec_sharding_hash)-[Consistent hashing](/ch7#sec_sharding_consistent_hashing), [Summary](/ch7#summary) - consistent hashing, [Consistent hashing](/ch7#sec_sharding_consistent_hashing) - problems with hash mod N, [Hash modulo number of nodes](/ch7#hash-modulo-number-of-nodes) - range queries, [Sharding by hash range](/ch7#sharding-by-hash-range) - suitable hash functions, [Sharding by Hash of Key](/ch7#sec_sharding_hash) - with fixed number of shards, [Fixed number of shards](/ch7#fixed-number-of-shards) - hash tables, [Log-Structured Storage](/ch4#sec_storage_log_structured) - Hazelcast (in-memory data grid) - FencedLock, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - Flake ID Generator, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - HBase (database) - bug due to lack of fencing, [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing) - key-range sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - log-structured storage, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - regions (sharding), [Sharding](/ch7#ch_sharding) - request routing, [Request Routing](/ch7#sec_sharding_routing) - size-tiered compaction, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - wide-column data model, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality), [Column Compression](/ch4#sec_storage_column_compression) - HDFS (Hadoop Distributed File System), [Batch Processing](/ch11#ch_batch), [Distributed Filesystems](/ch11#sec_batch_dfs) - (see also distributed filesystems) - checking data integrity, [Don't just blindly trust what they promise](/ch13#id364) - DataNode, [Distributed Filesystems](/ch11#sec_batch_dfs) - NameNode, [Distributed Filesystems](/ch11#sec_batch_dfs) - use in MapReduce, [MapReduce](/ch11#sec_batch_mapreduce) - workflow example, [Scheduling Workflows](/ch11#sec_batch_workflows) - HdrHistogram (numerical library), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - head (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Distributed Job Orchestration](/ch11#id278) - head vertex (property graphs), [Property Graphs](/ch3#id56) - head-of-line blocking, [Latency and Response Time](/ch2#id23) - heap files (databases), [Storing values within the index](/ch4#sec_storage_index_heap) - in multiversion concurrency control, [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl) - heat management, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - hedged requests, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - heterogeneous distributed transactions, [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa), [Problems with XA transactions](/ch8#problems-with-xa-transactions) - heuristic decisions (in 2PC), [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - Hex (notebook), [Machine Learning](/ch11#id290) - hexagons - for geospatial indexing, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - Hibernate (object-relational mapper), [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm) - hierarchical model, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - hierarchical navigable small world (vector index), [Vector Embeddings](/ch4#id92) - hierarchical queries (see recursive common table expressions) - high availability (see fault tolerance) - high-frequency trading, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - high-performance computing (HPC), [Cloud Computing Versus Supercomputing](/ch1#id17) - hinted handoff (leaderless replication), [Catching up on missed writes](/ch6#sec_replication_read_repair) - histograms, [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - Hive (data warehouse), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - query optimizer, [Query languages](/ch11#sec_batch_query_lanauges) - HNSW (vector index), [Vector Embeddings](/ch4#id92) - hopping windows (stream processing), [Types of windows](/ch12#id324) - (see also windows) - Hoptimator (query engine), [The meta-database of everything](/ch13#id341) - Horizon scandal, [Humans and Reliability](/ch2#id31) - lack of transactions, [Transactions](/ch8#ch_transactions) - horizontal scaling (see scaling out) - by sharding, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - HornetQ (messaging), [Message brokers](/ch5#message-brokers), [Message brokers compared to databases](/ch12#id297) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - hot keys, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - hot spots, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - due to celebrities, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - for time-series data, [Sharding by Key Range](/ch7#sec_sharding_key_range) - relieving, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - hot standbys (see leader-based replication) - HTAP (see hybrid transactional/analytic processing) - HTTP, use in APIs (see services) - human errors, [Humans and Reliability](/ch2#id31), [Network Faults in Practice](/ch9#sec_distributed_network_faults), [Batch Processing](/ch11#ch_batch) - hybrid logical clocks, [Hybrid logical clocks](/ch10#hybrid-logical-clocks) - hybrid transactional/analytic processing, [Data Warehousing](/ch1#sec_introduction_dwh), [Data Storage for Analytics](/ch4#sec_storage_analytics) - hydrating IDs (join), [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - hypergraph, [Property Graphs](/ch3#id56) - HyperLogLog (algorithm), [Stream analytics](/ch12#id318) ### I - I/O operations, waiting for, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - IaaS (see infrastructure as a service (IaaS)) - IBM - Db2 (database) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - serializable isolation, [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - MQ (messaging), [Message brokers compared to databases](/ch12#id297) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - System R (database), [What Exactly Is a Transaction?](/ch8#sec_transactions_overview) - WebSphere (messaging), [Message brokers](/ch5#message-brokers) - Iceberg (table format), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - databases on object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - log-based message broker storage, [Disk space usage](/ch12#sec_stream_disk_usage) - idempotence, [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc), [Idempotence](/ch12#sec_stream_idempotence), [Glossary](/glossary) - by giving operations unique IDs, [Multi-shard request processing](/ch13#id360) - by giving requests unique IDs, [Uniquely identifying requests](/ch13#id355) - for exactly-once semantics, [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - idempotent operations, [Exactly-once execution of an operation](/ch13#id353) - in workflow engines, [Durable execution](/ch5#durable-execution) - immutability - advantages of, [Advantages of immutable events](/ch12#sec_stream_immutability_pros), [Designing for auditability](/ch13#id365) - and right to erasure, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Disk space usage](/ch4#disk-space-usage) - crypto-shredding for deletion, [Event Sourcing and CQRS](/ch3#sec_datamodels_events), [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - deriving state from event log, [State, Streams, and Immutability](/ch12#sec_stream_immutability)-[Limitations of immutability](/ch12#sec_stream_immutability_limitations) - for crash recovery, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - in B-trees, [B-tree variants](/ch4#b-tree-variants), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - in event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events), [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - limitations of, [Concurrency control](/ch12#sec_stream_concurrency) - impedance mismatch, [The Object-Relational Mismatch](/ch3#sec_datamodels_document) - in doubt (transaction status), [Coordinator failure](/ch8#coordinator-failure) - holding locks, [Holding locks while in doubt](/ch8#holding-locks-while-in-doubt) - orphaned transactions, [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - in-memory databases, [Keeping everything in memory](/ch4#sec_storage_inmemory) - durability, [Durability](/ch8#durability) - serial transaction execution, [Actual Serial Execution](/ch8#sec_transactions_serial) - incidents - accounting software bugs leading to wrongful convictions, [Humans and Reliability](/ch2#id31) - blameless postmortems, [Humans and Reliability](/ch2#id31) - crashes due to leap seconds, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - data corruption and financial losses due to concurrency bugs, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - data corruption on hard disks, [Durability](/ch8#durability) - data loss due to last-write-wins, [Timestamps for ordering events](/ch9#sec_distributed_lww) - data on disks unreadable, [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - disclosure of sensitive data due to primary key reuse, [Leader failure: Failover](/ch6#leader-failure-failover) - errors in transaction serializability, [Maintaining integrity in the face of software bugs](/ch13#id455) - gigabit network interface with 1 Kb/s throughput, [System Model and Reality](/ch9#sec_distributed_system_model) - leap second crash, [Software faults](/ch2#software-faults) - network faults, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - network interface dropping only inbound packets, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - network partitions and whole-datacenter failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - poor handling of network faults, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - sending message to ex-partner, [Ordering events to capture causality](/ch13#sec_future_capture_causality) - sharks biting undersea cables, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - split brain due to 1-minute packet delay, [Leader failure: Failover](/ch6#leader-failure-failover), [Network Faults in Practice](/ch9#sec_distributed_network_faults) - SSD failure after 32,768 hours, [Software faults](/ch2#software-faults) - thread contention bringing down a service, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - vibrations in server rack, [Latency and Response Time](/ch2#id23) - violation of uniqueness constraint, [Maintaining integrity in the face of software bugs](/ch13#id455) - incremental view maintenance (IVM), [Maintaining materialized views](/ch12#sec_stream_mat_view) - for data integration, [Unbundled versus integrated systems](/ch13#id448) - indexes, [Storage and Indexing for OLTP](/ch4#sec_storage_oltp), [Glossary](/glossary) - and snapshot isolation, [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - as derived data, [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - B-trees, [B-Trees](/ch4#sec_storage_b_trees)-[B-tree variants](/ch4#b-tree-variants) - clustered, [Storing values within the index](/ch4#sec_storage_index_heap) - comparison of B-trees and LSM-trees, [Comparing B-Trees and LSM-Trees](/ch4#sec_storage_btree_lsm_comparison)-[Disk space usage](/ch4#disk-space-usage) - covering (with included columns), [Storing values within the index](/ch4#sec_storage_index_heap) - creating, [Creating an index](/ch13#id340) - full-text search, [Full-Text Search](/ch4#sec_storage_full_text) - geospatial, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - index-range locking, [Index-range locks](/ch8#sec_transactions_2pl_range) - multi-column (concatenated), [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - secondary, [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn) - (see also secondary indexes) - problems with dual writes, [Keeping Systems in Sync](/ch12#sec_stream_sync), [Reasoning about dataflows](/ch13#id443) - sharding and secondary indexes, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes)-[Global Secondary Indexes](/ch7#id167), [Summary](/ch7#summary) - sparse, [The SSTable file format](/ch4#the-sstable-file-format) - SSTables and LSM-trees, [The SSTable file format](/ch4#the-sstable-file-format)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - updating when data changes, [Keeping Systems in Sync](/ch12#sec_stream_sync), [Maintaining materialized views](/ch12#sec_stream_mat_view) - Industrial Revolution, [Remembering the Industrial Revolution](/ch14#id377) - InfiniBand (networks), [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - InfluxDB IOx (storage engine), [Column-Oriented Storage](/ch4#sec_storage_column) - information retrieval (see full-text search) - infrastructure as a service (IaaS), [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud), [Layering of cloud services](/ch1#layering-of-cloud-services) - InnoDB (storage engine) - clustered index on primary key, [Storing values within the index](/ch4#sec_storage_index_heap) - not preventing lost updates, [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - preventing write skew, [Characterizing write skew](/ch8#characterizing-write-skew), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - serializable isolation, [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - instance (cloud computing), [Layering of cloud services](/ch1#layering-of-cloud-services) - integrating different data systems (see data integration) - integrity, [Timeliness and Integrity](/ch13#sec_future_integrity) - coordination-avoiding data systems, [Coordination-avoiding data systems](/ch13#id454) - correctness of dataflow systems, [Correctness of dataflow systems](/ch13#id453) - in consensus formalization, [Single-value consensus](/ch10#single-value-consensus), [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - integrity checks, [Don't just blindly trust what they promise](/ch13#id364) - (see also auditing) - end-to-end, [The end-to-end argument](/ch13#sec_future_e2e_argument), [The end-to-end argument again](/ch13#id456) - use of snapshot isolation, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - maintaining despite software bugs, [Maintaining integrity in the face of software bugs](/ch13#id455) - Interface Definition Language (IDL), [Protocol Buffers](/ch5#sec_encoding_protobuf), [Avro](/ch5#sec_encoding_avro), [Web services](/ch5#sec_web_services) - invariants, [Consistency](/ch8#sec_transactions_acid_consistency) - (see also constraints) - inverted file index (vector index), [Vector Embeddings](/ch4#id92) - inverted index, [Full-Text Search](/ch4#sec_storage_full_text) - irreversibility, minimizing, [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability), [Event Sourcing and CQRS](/ch3#sec_datamodels_events), [Batch Processing](/ch11#ch_batch) - ISDN (Integrated Services Digital Network), [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - isolation (in operating systems) - cgroups (see cgroups) - isolation (in transactions), [Isolation](/ch8#sec_transactions_acid_isolation), [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object), [Glossary](/glossary) - correctness and, [Aiming for Correctness](/ch13#sec_future_correctness) - for single-object writes, [Single-object writes](/ch8#sec_transactions_single_object) - serializability, [Serializability](/ch8#sec_transactions_serializability)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - actual serial execution, [Actual Serial Execution](/ch8#sec_transactions_serial)-[Summary of serial execution](/ch8#summary-of-serial-execution) - serializable snapshot isolation (SSI), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - two-phase locking (2PL), [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl)-[Index-range locks](/ch8#sec_transactions_2pl_range) - violating, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - weak isolation levels, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels)-[Materializing conflicts](/ch8#materializing-conflicts) - preventing lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update)-[Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - read committed, [Read Committed](/ch8#sec_transactions_read_committed)-[Implementing read committed](/ch8#sec_transactions_read_committed_impl) - snapshot isolation, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation)-[Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - IVF (vector index), [Vector Embeddings](/ch4#id92) ### J - Java Database Connectivity (JDBC) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - network drivers, [The Merits of Schemas](/ch5#sec_encoding_schemas) - Java Enterprise Edition (EE), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc), [XA transactions](/ch8#xa-transactions) - Java Message Service (JMS), [Message brokers compared to databases](/ch12#id297) - (see also messaging systems) - comparison to log-based messaging, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Replaying old messages](/ch12#sec_stream_replay) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - message ordering, [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - Java Transaction API (JTA), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc), [XA transactions](/ch8#xa-transactions) - Java Virtual Machine (JVM) - garbage collection, [Process Pauses](/ch9#sec_distributed_clocks_pauses), [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - JIT compilation, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - process reuse in batch processors, [Dataflow Engines](/ch11#sec_batch_dataflow) - Jena (RDF framework), [The RDF data model](/ch3#the-rdf-data-model) - SPARQL query language, [The SPARQL query language](/ch3#the-sparql-query-language) - Jepsen (fault tolerance testing), [Fault injection](/ch9#sec_fault_injection), [Aiming for Correctness](/ch13#sec_future_correctness) - jitter (network delay), [Average, Median, and Percentiles](/ch2#id24), [Network congestion and queueing](/ch9#network-congestion-and-queueing) - JMESPath (query language), [Query languages](/ch11#sec_batch_query_lanauges) - join table, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many), [Property Graphs](/ch3#id56) - joins, [Glossary](/glossary) - expressing as relational operators, [Query languages](/ch11#sec_batch_query_lanauges) - handling GraphQL query, [GraphQL](/ch3#id63) - in application code, [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization), [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - in DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - in relational and document databases, [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization) - secondary indexes and, [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn) - sort-merge joins, [JOIN and GROUP BY](/ch11#sec_batch_join) - stream joins, [Stream Joins](/ch12#sec_stream_joins)-[Time-dependence of joins](/ch12#sec_stream_join_time) - stream-stream join, [Stream-stream join (window join)](/ch12#id440) - stream-table join, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - table-table join, [Table-table join (materialized view maintenance)](/ch12#id326) - time-dependence of, [Time-dependence of joins](/ch12#sec_stream_join_time) - support in document databases, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - JOTM (transaction coordinator), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - journaling (filesystems), [Making B-trees reliable](/ch4#sec_storage_btree_wal) - JSON - aggregation pipeline (query language), [Query languages for documents](/ch3#query-languages-for-documents) - Avro schema representation, [Avro](/ch5#sec_encoding_avro) - binary variants, [Binary encoding](/ch5#binary-encoding) - data locality, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - document data model, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - for application data, issues with, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - GraphQL response, [GraphQL](/ch3#id63) - in relational databases, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - representing a résumé (example), [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - Schema, [JSON Schema](/ch5#json-schema) - JSON-LD, [Triple-Stores and SPARQL](/ch3#id59) - JsonPath (query language), [Query languages](/ch11#sec_batch_query_lanauges) - JuiceFS (distributed filesystem), [Distributed Filesystems](/ch11#sec_batch_dfs), [Object Stores](/ch11#id277) - Jupyter (notebook), [Machine Learning](/ch11#id290) - just-in-time (JIT) compilation, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) ### K - Kafka (messaging), [Message brokers](/ch5#message-brokers), [Using logs for message storage](/ch12#id300) - consumer groups, [Multiple consumers](/ch12#id298) - for data integration, [Unbundled versus integrated systems](/ch13#id448) - for event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - Kafka Connect (database integration), [Implementing change data capture](/ch12#id307), [API support for change streams](/ch12#sec_stream_change_api), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - Kafka Streams (stream processor), [Stream analytics](/ch12#id318), [Maintaining materialized views](/ch12#sec_stream_mat_view) - exactly-once semantics, [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - fault tolerance, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - ksqlDB (stream database), [Maintaining materialized views](/ch12#sec_stream_mat_view) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - log compaction, [Log compaction](/ch12#sec_stream_log_compaction), [Maintaining materialized views](/ch12#sec_stream_mat_view) - message offsets, [Using logs for message storage](/ch12#id300), [Idempotence](/ch12#sec_stream_idempotence) - partitions (sharding), [Sharding](/ch7#ch_sharding) - request routing, [Request Routing](/ch7#sec_sharding_routing) - schema registry, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - tiered storage, [Disk space usage](/ch12#sec_stream_disk_usage) - transactions, [Database-internal Distributed Transactions](/ch8#sec_transactions_internal), [Atomic commit revisited](/ch12#sec_stream_atomic_commit) - unclean leader election, [Subtleties of consensus](/ch10#subtleties-of-consensus) - use of model-checking, [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - kappa architecture, [Unifying batch and stream processing](/ch13#id338) - key-value stores, [Storage and Indexing for OLTP](/ch4#sec_storage_oltp) - comparison to object stores, [Object Stores](/ch11#id277) - in-memory, [Keeping everything in memory](/ch4#sec_storage_inmemory) - LSM storage, [Log-Structured Storage](/ch4#sec_storage_log_structured)-[Disk space usage](/ch4#disk-space-usage) - sharding, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value)-[Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - by hash of key, [Sharding by Hash of Key](/ch7#sec_sharding_hash), [Summary](/ch7#summary) - by key range, [Sharding by Key Range](/ch7#sec_sharding_key_range), [Summary](/ch7#summary) - skew and hot spots, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - Kinesis (messaging), [Message brokers](/ch5#message-brokers), [Using logs for message storage](/ch12#id300) - data warehouse integration, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - Kryo (Java), [Language-Specific Formats](/ch5#id96) - ksqlDB (stream database), [Maintaining materialized views](/ch12#sec_stream_mat_view) - Kubernetes (cluster manager), [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud), [Microservices and Serverless](/ch1#sec_introduction_microservices), [Distributed Job Orchestration](/ch11#id278), [Separation of application code and state](/ch13#id344) - Kubeflow, [Machine Learning](/ch11#id290) - kubelet, [Distributed Job Orchestration](/ch11#id278) - operators, [Distributed Job Orchestration](/ch11#id278) - use of etcd, [Request Routing](/ch7#sec_sharding_routing), [Coordination Services](/ch10#sec_consistency_coordination) - KùzuDB (database), [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - as embedded storage engine, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - Cypher query language, [The Cypher Query Language](/ch3#id57) ### L - labeled property graphs (see property graphs) - lambda architecture, [Unifying batch and stream processing](/ch13#id338) - Lamport timestamps, [Lamport timestamps](/ch10#lamport-timestamps) - Lance (data format), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Column-Oriented Storage](/ch4#sec_storage_column) - (see also column-oriented storage) - large language models (LLMs) - pre-processing training data, [Machine Learning](/ch11#id290) - last write wins (LWW), [Last write wins (discarding concurrent writes)](/ch6#sec_replication_lww), [Detecting Concurrent Writes](/ch6#sec_replication_concurrent), [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - problems with, [Timestamps for ordering events](/ch9#sec_distributed_lww) - prone to lost updates, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - latency, [Latency and Response Time](/ch2#id23) - (see also response time) - across regions, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - instability under two-phase locking, [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - network latency and resource utilization, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - reducing by request hedging, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - response time versus, [Latency and Response Time](/ch2#id23) - tail latency, [Average, Median, and Percentiles](/ch2#id24), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla), [Local Secondary Indexes](/ch7#id166) - law (see legal matters) - layering (of cloud services), [Layering of cloud services](/ch1#layering-of-cloud-services) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader)-[Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - (see also replication) - failover, [Leader failure: Failover](/ch6#leader-failure-failover), [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing) - handling node outages, [Handling Node Outages](/ch6#sec_replication_failover) - implementation of replication logs - change data capture, [Change Data Capture](/ch12#sec_stream_cdc)-[API support for change streams](/ch12#sec_stream_change_api) - (see also changelogs) - statement-based, [Statement-based replication](/ch6#statement-based-replication) - write-ahead log (WAL) shipping, [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - linearizability of operations, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - locking and leader election, [Locking and leader election](/ch10#locking-and-leader-election) - log sequence number, [Setting Up New Followers](/ch6#sec_replication_new_replica), [Consumer offsets](/ch12#sec_stream_log_offsets) - read-scaling architecture, [Problems with Replication Lag](/ch6#sec_replication_lag), [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - relation to consensus, [Consensus](/ch10#sec_consistency_consensus), [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus), [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - setting up new followers, [Setting Up New Followers](/ch6#sec_replication_new_replica) - synchronous versus asynchronous, [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async)-[Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async) - leaderless replication, [Leaderless Replication](/ch6#sec_replication_leaderless)-[Version vectors](/ch6#version-vectors) - (see also replication) - catching up on missed writes, [Catching up on missed writes](/ch6#sec_replication_read_repair) - detecting concurrent writes, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent)-[Version vectors](/ch6#version-vectors) - version vectors, [Version vectors](/ch6#version-vectors) - multi-region, [Multi-region operation](/ch6#multi-region-operation) - quorums, [Quorums for reading and writing](/ch6#sec_replication_quorum_condition)-[Multi-region operation](/ch6#multi-region-operation) - consistency limitations, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations)-[Monitoring staleness](/ch6#monitoring-staleness), [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - leap seconds, [Software faults](/ch2#software-faults), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - in time-of-day clocks, [Time-of-day clocks](/ch9#time-of-day-clocks) - leases, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - implementation with coordination service, [Coordination Services](/ch10#sec_consistency_coordination) - need for fencing, [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing) - relation to consensus, [Single-value consensus](/ch10#single-value-consensus) - ledgers (accounting), [Summary](/ch3#summary) - immutability, [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - legacy systems, maintenance of, [Maintainability](/ch2#sec_introduction_maintainability) - legal matters, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance)-[Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - data deletion, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Disk space usage](/ch4#disk-space-usage) - data residence, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - privacy regulation, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Legislation and Self-Regulation](/ch14#sec_future_legislation) - legitimate interest (GDPR), [Consent and Freedom of Choice](/ch14#id375) - leveled compaction, [Compaction strategies](/ch4#sec_storage_lsm_compaction), [Disk space usage](/ch4#disk-space-usage) - Levenshtein automata, [Full-Text Search](/ch4#sec_storage_full_text) - limping (partial failure), [System Model and Reality](/ch9#sec_distributed_system_model) - Linear (project management software), [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - linear algebra, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - linear scalability, [Describing Load](/ch2#id33) - linearizability, [Solutions for Replication Lag](/ch6#id131), [Linearizability](/ch10#sec_consistency_linearizability)-[Linearizability and network delays](/ch10#linearizability-and-network-delays), [Glossary](/glossary) - and consensus, [Consensus](/ch10#sec_consistency_consensus) - cost of, [The Cost of Linearizability](/ch10#sec_linearizability_cost)-[Linearizability and network delays](/ch10#linearizability-and-network-delays) - CAP theorem, [The CAP theorem](/ch10#the-cap-theorem) - memory on multi-core CPUs, [Linearizability and network delays](/ch10#linearizability-and-network-delays) - definition, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition)-[What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - ID generation, [Linearizable ID Generators](/ch10#sec_consistency_linearizable_id) - in coordination services, [Coordination Services](/ch10#sec_consistency_coordination) - of derived data systems - avoiding coordination, [Coordination-avoiding data systems](/ch13#id454) - of different replication methods, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable)-[Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - using quorums, [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - reads in consensus systems, [Subtleties of consensus](/ch10#subtleties-of-consensus) - relying on, [Relying on Linearizability](/ch10#sec_consistency_linearizability_usage)-[Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - constraints and uniqueness, [Constraints and uniqueness guarantees](/ch10#sec_consistency_uniqueness) - cross-channel timing dependencies, [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - locking and leader election, [Locking and leader election](/ch10#locking-and-leader-election) - versus serializability, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - linked data, [Triple-Stores and SPARQL](/ch3#id59) - LinkedIn - Espresso (database), [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - LIquid (database), [Datalog: Recursive Relational Queries](/ch3#id62) - profile (example), [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - Linux, leap second bug, [Software faults](/ch2#software-faults), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - Litestream (backup tool), [Setting Up New Followers](/ch6#sec_replication_new_replica) - liveness properties, [Safety and liveness](/ch9#sec_distributed_safety_liveness) - LLVM (compiler), [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - LMDB (storage engine), [Compaction strategies](/ch4#sec_storage_lsm_compaction), [B-tree variants](/ch4#b-tree-variants), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - load - coping with, [Principles for Scalability](/ch2#id35) - describing, [Describing Load](/ch2#id33) - load balancing, [Describing Performance](/ch2#sec_introduction_percentiles), [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - in hardware, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - in software, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - using message brokers, [Multiple consumers](/ch12#id298) - load shedding, [Describing Performance](/ch2#sec_introduction_percentiles) - local secondary indexes, [Local Secondary Indexes](/ch7#id166), [Summary](/ch7#summary) - local-first software, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - locality (data access), [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships), [Data locality for reads and writes](/ch3#sec_datamodels_document_locality), [Glossary](/glossary) - in batch processing, [Dataflow Engines](/ch11#sec_batch_dataflow) - in stateful clients, [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients), [Stateful, offline-capable clients](/ch13#id347) - in stream processing, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins), [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance), [Stream processors and services](/ch13#id345), [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - location transparency, [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - in the actor model, [Distributed actor frameworks](/ch5#distributed-actor-frameworks) - lock-in, [Pros and Cons of Cloud Services](/ch1#sec_introduction_cloud_tradeoffs) - locks, [Glossary](/glossary) - deadlock, [Explicit locking](/ch8#explicit-locking), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - distributed locking, [Distributed Locks and Leases](/ch9#sec_distributed_lock_fencing)-[Fencing with multiple replicas](/ch9#fencing-with-multiple-replicas), [Locking and leader election](/ch10#locking-and-leader-election) - fencing tokens, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - implementation with coordination service, [Coordination Services](/ch10#sec_consistency_coordination) - relation to consensus, [Single-value consensus](/ch10#single-value-consensus) - for transaction isolation - in snapshot isolation, [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl) - in two-phase locking (2PL), [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl)-[Index-range locks](/ch8#sec_transactions_2pl_range) - making operations atomic, [Atomic write operations](/ch8#atomic-write-operations) - performance, [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - preventing dirty writes, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - preventing phantoms with index-range locks, [Index-range locks](/ch8#sec_transactions_2pl_range), [Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - read locks (shared mode), [Implementing read committed](/ch8#sec_transactions_read_committed_impl), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - shared mode and exclusive mode, [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - in distributed transactions - deadlock detection, [Problems with XA transactions](/ch8#problems-with-xa-transactions) - in-doubt transactions holding locks, [Holding locks while in doubt](/ch8#holding-locks-while-in-doubt) - materializing conflicts with, [Materializing conflicts](/ch8#materializing-conflicts) - preventing lost updates by explicit locking, [Explicit locking](/ch8#explicit-locking) - log sequence number, [Setting Up New Followers](/ch6#sec_replication_new_replica), [Consumer offsets](/ch12#sec_stream_log_offsets) - logical clocks, [Timestamps for ordering events](/ch9#sec_distributed_lww), [ID Generators and Logical Clocks](/ch10#sec_consistency_logical)-[Enforcing constraints using logical clocks](/ch10#enforcing-constraints-using-logical-clocks), [Ordering events to capture causality](/ch13#sec_future_capture_causality) - for last-write-wins, [Last write wins (discarding concurrent writes)](/ch6#sec_replication_lww) - for read-after-write consistency, [Reading Your Own Writes](/ch6#sec_replication_ryw) - hybrid logical clocks, [Hybrid logical clocks](/ch10#hybrid-logical-clocks) - insufficiency for enforcing constraints, [Enforcing constraints using logical clocks](/ch10#enforcing-constraints-using-logical-clocks) - Lamport timestamps, [Lamport timestamps](/ch10#lamport-timestamps) - logical replication, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - for change data capture, [Implementing change data capture](/ch12#id307) - LogicBlox (database), [Datalog: Recursive Relational Queries](/ch3#id62) - logs (data structure), [Storage and Indexing for OLTP](/ch4#sec_storage_oltp), [Shared logs as consensus](/ch10#sec_consistency_shared_logs), [Glossary](/glossary) - (see also shared logs) - advantages of immutability, [Advantages of immutable events](/ch12#sec_stream_immutability_pros) - and right to erasure, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Disk space usage](/ch4#disk-space-usage) - compaction, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Compaction strategies](/ch4#sec_storage_lsm_compaction), [Log compaction](/ch12#sec_stream_log_compaction), [State, Streams, and Immutability](/ch12#sec_stream_immutability) - for stream operator state, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - implementing uniqueness constraints, [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - log-based messaging, [Log-based Message Brokers](/ch12#sec_stream_log)-[Replaying old messages](/ch12#sec_stream_replay) - comparison to traditional messaging, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Replaying old messages](/ch12#sec_stream_replay) - consumer offsets, [Consumer offsets](/ch12#sec_stream_log_offsets) - disk space usage, [Disk space usage](/ch12#sec_stream_disk_usage) - replaying old messages, [Replaying old messages](/ch12#sec_stream_replay), [Reprocessing data for application evolution](/ch13#sec_future_reprocessing), [Unifying batch and stream processing](/ch13#id338) - slow consumers, [When consumers cannot keep up with producers](/ch12#id459) - using logs for message storage, [Using logs for message storage](/ch12#id300) - log-structured storage, [Storage and Indexing for OLTP](/ch4#sec_storage_oltp)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - log-structured merge tree (see LSM-trees) - relation to consensus, [Shared logs as consensus](/ch10#sec_consistency_shared_logs) - replication, [Single-Leader Replication](/ch6#sec_replication_leader), [Implementation of Replication Logs](/ch6#sec_replication_implementation)-[Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - change data capture, [Change Data Capture](/ch12#sec_stream_cdc)-[API support for change streams](/ch12#sec_stream_change_api) - (see also changelogs) - coordination with snapshot, [Setting Up New Followers](/ch6#sec_replication_new_replica) - logical (row-based) replication, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication) - write-ahead log (WAL) shipping, [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - scalability limits, [The limits of total ordering](/ch13#id335) - Looker (business intelligence software), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Analytics](/ch11#sec_batch_olap) - loose coupling, [Making unbundling work](/ch13#sec_future_unbundling_favor) - lost updates (see updates) - Lotus Notes (sync engine), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - LSM-trees (indexes), [The SSTable file format](/ch4#the-sstable-file-format)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - comparison to B-trees, [Comparing B-Trees and LSM-Trees](/ch4#sec_storage_btree_lsm_comparison)-[Disk space usage](/ch4#disk-space-usage) - Lucene (storage engine), [Full-Text Search](/ch4#sec_storage_full_text) - similarity search, [Full-Text Search](/ch4#sec_storage_full_text) - LWW (see last write wins) ### M - machine learning - batch inference, [Machine Learning](/ch11#id290) - data preparation with DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - deleting training data, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - deploying data products, [Beyond the data lake](/ch1#beyond-the-data-lake) - ethical considerations, [Predictive Analytics](/ch14#id369) - (see also ethics) - feature engineering, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [Machine Learning](/ch11#id290) - in analytics systems, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - iterative processing, [Machine Learning](/ch11#id290) - LLMs (see large language models (LLMs)) - models derived from training data, [Application code as a derivation function](/ch13#sec_future_dataflow_derivation) - relation to batch processing, [Machine Learning](/ch11#id290)-[Machine Learning](/ch11#id290) - using a data lake, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - using GPUs, [Layering of cloud services](/ch1#layering-of-cloud-services), [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - using matrices, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - madsim (deterministic simulation testing), [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - magic scaling sauce, [Principles for Scalability](/ch2#id35) - maintainability, [Maintainability](/ch2#sec_introduction_maintainability)-[Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability), [A Philosophy of Streaming Systems](/ch13#ch_philosophy) - evolvability (see evolvability) - operability, [Operability: Making Life Easy for Operations](/ch2#id37) - simplicity and managing complexity, [Simplicity: Managing Complexity](/ch2#id38) - many-to-many relationships, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many) - modeling as graphs, [Graph-Like Data Models](/ch3#sec_datamodels_graph) - many-to-one relationships, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many) - in star schema, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - MapReduce (batch processing), [Batch Processing](/ch11#ch_batch), [MapReduce](/ch11#sec_batch_mapreduce)-[MapReduce](/ch11#sec_batch_mapreduce) - analysis of user activity events (example), [JOIN and GROUP BY](/ch11#sec_batch_join) - comparison to stream processing, [Processing Streams](/ch12#sec_stream_processing) - disadvantages and limitations of, [MapReduce](/ch11#sec_batch_mapreduce) - fault tolerance, [Handling Faults](/ch11#id281) - higher-level tools, [Query languages](/ch11#sec_batch_query_lanauges) - mapper and reducer functions, [MapReduce](/ch11#sec_batch_mapreduce) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - sort-merge joins, [JOIN and GROUP BY](/ch11#sec_batch_join) - workflows, [Scheduling Workflows](/ch11#sec_batch_workflows) - (see also workflow engines) - marshalling (see encoding) - MartenDB (database), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - master-slave replication (obsolete term), [Single-Leader Replication](/ch6#sec_replication_leader) - materialization, [Glossary](/glossary) - aggregate values, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - conflicts, [Materializing conflicts](/ch8#materializing-conflicts) - materialized views, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - as derived data, [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - in event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - incremental view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - (see also incremental view maintenance (IVM)) - maintaining, using stream processing, [Maintaining materialized views](/ch12#sec_stream_mat_view), [Table-table join (materialized view maintenance)](/ch12#id326) - social network timeline example, [Materializing and Updating Timelines](/ch2#sec_introduction_materializing) - Materialize (database), [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - incremental view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - matrices, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - sparse, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - Maxwell (change data capture), [Implementing change data capture](/ch12#id307) - mean, [Average, Median, and Percentiles](/ch2#id24) - media monitoring, [Search on streams](/ch12#id320) - median, [Average, Median, and Percentiles](/ch2#id24) - meeting room booking (example), [More examples of write skew](/ch8#more-examples-of-write-skew), [Predicate locks](/ch8#predicate-locks), [Enforcing Constraints](/ch13#sec_future_constraints) - Memcached (caching server), [Keeping everything in memory](/ch4#sec_storage_inmemory) - Memgraph (database), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - Cypher query language, [The Cypher Query Language](/ch3#id57) - memory - barrier (CPU instruction), [Linearizability and network delays](/ch10#linearizability-and-network-delays) - corruption, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - in-memory databases, [Keeping everything in memory](/ch4#sec_storage_inmemory) - durability, [Durability](/ch8#durability) - serial transaction execution, [Actual Serial Execution](/ch8#sec_transactions_serial) - in-memory representation of data, [Formats for Encoding Data](/ch5#sec_encoding_formats) - memtable (in LSM-trees), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - random bit-flips in, [Trust, but Verify](/ch13#sec_future_verification) - use by indexes, [Log-Structured Storage](/ch4#sec_storage_log_structured) - memtable (in LSM-trees), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - Mercurial (version control system), [Concurrency control](/ch12#sec_stream_concurrency) - merge (DataFrame operator), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - merging sorted files, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Shuffling Data](/ch11#sec_shuffle) - Merkle trees, [Tools for auditable data systems](/ch13#id366) - Mesos (cluster manager), [Separation of application code and state](/ch13#id344) - message brokers (see messaging systems) - message-passing (see event-driven architecture) - MessagePack (encoding format), [Binary encoding](/ch5#binary-encoding) - messaging systems, [Stream Processing](/ch12#ch_stream)-[Replaying old messages](/ch12#sec_stream_replay) - (see also streams) - backpressure, buffering, or dropping messages, [Messaging Systems](/ch12#sec_stream_messaging) - brokerless messaging, [Direct messaging from producers to consumers](/ch12#id296) - event logs, [Log-based Message Brokers](/ch12#sec_stream_log)-[Replaying old messages](/ch12#sec_stream_replay) - as data model, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - comparison to traditional messaging, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Replaying old messages](/ch12#sec_stream_replay) - consumer offsets, [Consumer offsets](/ch12#sec_stream_log_offsets) - replaying old messages, [Replaying old messages](/ch12#sec_stream_replay), [Reprocessing data for application evolution](/ch13#sec_future_reprocessing), [Unifying batch and stream processing](/ch13#id338) - slow consumers, [When consumers cannot keep up with producers](/ch12#id459) - exactly-once semantics, [Exactly-once message processing](/ch8#sec_transactions_exactly_once), [Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited), [Fault Tolerance](/ch12#sec_stream_fault_tolerance) - message brokers, [Message brokers](/ch12#id433)-[Acknowledgments and redelivery](/ch12#sec_stream_reordering) - acknowledgements and redelivery, [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - comparison to event logs, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Replaying old messages](/ch12#sec_stream_replay) - multiple consumers of same topic, [Multiple consumers](/ch12#id298) - versus RPC, [Event-Driven Architectures](/ch5#sec_encoding_dataflow_msg) - message loss, [Messaging Systems](/ch12#sec_stream_messaging) - reliability, [Messaging Systems](/ch12#sec_stream_messaging) - uniqueness in log-based messaging, [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - metastable failure, [Describing Performance](/ch2#sec_introduction_percentiles) - metered billing - serverless, [Microservices and Serverless](/ch1#sec_introduction_microservices) - storage, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - microbatching, [Microbatching and checkpointing](/ch12#id329) - microservices, [Microservices and Serverless](/ch1#sec_introduction_microservices) - (see also services) - causal dependencies across services, [The limits of total ordering](/ch13#id335) - loose coupling, [Making unbundling work](/ch13#sec_future_unbundling_favor) - relation to batch/stream processors, [Batch Processing](/ch11#ch_batch), [Stream processors and services](/ch13#id345) - Microsoft - Azure Blob Storage (see Azure Blob Storage) - Azure managed disks, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - Azure Service Bus (messaging), [Message brokers](/ch5#message-brokers), [Message brokers compared to databases](/ch12#id297) - Azure SQL DB (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - Azure Storage, [Object Stores](/ch11#id277) - Azure Stream Analytics, [Stream analytics](/ch12#id318) - Azure Synapse Analytics (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - DCOM (Distributed Component Object Model), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - MSDTC (transaction coordinator), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - SQL Server (see SQL Server) - Microsoft Power BI (see Power BI (business intelligence software)) - migrating (rewriting) data, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility), [Different values written at different times](/ch5#different-values-written-at-different-times), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views), [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - MinIO (object storage), [Distributed Filesystems](/ch11#sec_batch_dfs) - mobile apps, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - embedded databases, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - model checking, [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - modulus operator (%), [Hash modulo number of nodes](/ch7#hash-modulo-number-of-nodes) - Mojo (programming language) - memory management, [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - MongoDB (database) - aggregation pipeline, [Query languages for documents](/ch3#query-languages-for-documents) - atomic operations, [Atomic write operations](/ch8#atomic-write-operations) - BSON, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - document data model, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - hash-range sharding, [Sharding by Hash of Key](/ch7#sec_sharding_hash), [Sharding by hash range](/ch7#sharding-by-hash-range) - in the cloud, [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - join support, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - joins (\$lookup operator), [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization) - JSON Schema validation, [JSON Schema](/ch5#json-schema) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - ObjectIds, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - range-based sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - request routing, [Request Routing](/ch7#sec_sharding_routing) - secondary indexes, [Local Secondary Indexes](/ch7#id166) - shard splitting, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - stored procedures, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - monitoring, [Operations in the Cloud Era](/ch1#sec_introduction_operations), [Humans and Reliability](/ch2#id31), [Operability: Making Life Easy for Operations](/ch2#id37) - monotonic clocks, [Monotonic clocks](/ch9#monotonic-clocks) - monotonic reads, [Monotonic Reads](/ch6#sec_replication_monotonic_reads) - Morel (query language), [Query languages](/ch11#sec_batch_query_lanauges) - MSMQ (messaging), [XA transactions](/ch8#xa-transactions) - multi-column indexes, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - multi-leader replication, [Multi-Leader Replication](/ch6#sec_replication_multi_leader)-[Types of conflict](/ch6#sec_replication_write_conflicts) - (see also replication) - collaborative editing, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - conflict detection, [Types of conflict](/ch6#sec_replication_write_conflicts) - conflict resolution, [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts) - for multi-region replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc), [The Cost of Linearizability](/ch10#sec_linearizability_cost) - linearizability, lack of, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - offline-capable clients, [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients) - replication topologies, [Multi-leader replication topologies](/ch6#sec_replication_topologies)-[Problems with different topologies](/ch6#problems-with-different-topologies) - multi-object transactions, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - need for, [The need for multi-object transactions](/ch8#sec_transactions_need) - Multi-Paxos (consensus algorithm), [Consensus in Practice](/ch10#sec_consistency_total_order) - multi-reader single-writer lock, [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - multi-table index cluster tables (Oracle), [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - multi-version concurrency control (MVCC), [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl), [Summary](/ch8#summary) - detecting stale MVCC reads, [Detecting stale MVCC reads](/ch8#detecting-stale-mvcc-reads) - indexes and snapshot isolation, [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - using synchronized clocks, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - multidimensional arrays, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - multitenancy, [Separation of storage and compute](/ch1#sec_introduction_storage_compute), [Network congestion and queueing](/ch9#network-congestion-and-queueing) - by sharding, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - using embedded databases, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - versus Byzantine fault tolerance, [Byzantine Faults](/ch9#sec_distributed_byzantine) - mutual exclusion, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - (see also locks) - MySQL (database) - archiving WAL to object stores, [Setting Up New Followers](/ch6#sec_replication_new_replica) - binlog coordinates, [Setting Up New Followers](/ch6#sec_replication_new_replica) - change data capture, [Implementing change data capture](/ch12#id307), [API support for change streams](/ch12#sec_stream_change_api) - circular replication topology, [Multi-leader replication topologies](/ch6#sec_replication_topologies) - consistent snapshots, [Setting Up New Followers](/ch6#sec_replication_new_replica) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - global transaction identifiers (GTIDs), [Setting Up New Followers](/ch6#sec_replication_new_replica) - in the cloud, [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - InnoDB storage engine (see InnoDB) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - row-based replication, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - sharding (see Vitess (database)) - snapshot isolation support, [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - (see also InnoDB) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication) ### N - N+1 query problem, [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm) - nanomsg (messaging library), [Direct messaging from producers to consumers](/ch12#id296) - Narayana (transaction coordinator), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) - NATS (messaging), [Message brokers](/ch5#message-brokers) - natural language processing, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - Neo4j (database) - Cypher query language, [The Cypher Query Language](/ch3#id57) - graph data model, [Graph-Like Data Models](/ch3#sec_datamodels_graph) - Neon (database), [Setting Up New Followers](/ch6#sec_replication_new_replica) - Nephele (dataflow engine), [Dataflow Engines](/ch11#sec_batch_dataflow) - Neptune (graph database), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - Cypher query language, [The Cypher Query Language](/ch3#id57) - SPARQL query language, [The SPARQL query language](/ch3#the-sparql-query-language) - netcode (game development), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - Network Attached Storage (NAS), [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing), [Distributed Filesystems](/ch11#sec_batch_dfs) - network model (data representation), [Relational Model versus Document Model](/ch3#sec_datamodels_history) - Network Time Protocol (see NTP) - networks - congestion and queueing, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - datacenter network topologies, [Cloud Computing Versus Supercomputing](/ch1#id17) - faults (see faults) - linearizability and network delays, [Linearizability and network delays](/ch10#linearizability-and-network-delays) - network partitions, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - in CAP theorem, [The Cost of Linearizability](/ch10#sec_linearizability_cost) - timeouts and unbounded delays, [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - NewSQL, [Relational Model versus Document Model](/ch3#sec_datamodels_history), [Solutions for Replication Lag](/ch6#id131) - transactions and, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - next-key locking, [Index-range locks](/ch8#sec_transactions_2pl_range) - NFS (network file system), [Distributed Filesystems](/ch11#sec_batch_dfs) - on object storage, [Object Stores](/ch11#id277) - Nimble (data format), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Column-Oriented Storage](/ch4#sec_storage_column) - (see also column-oriented storage) - node (in graphs) (see vertices) - nodes (processes), [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Glossary](/glossary) - handling outages in leader-based replication, [Handling Node Outages](/ch6#sec_replication_failover) - system models for failure, [System Model and Reality](/ch9#sec_distributed_system_model) - noisy neighbors, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - nonblocking atomic commit, [Three-phase commit](/ch8#three-phase-commit) - nondeterministic operations, [Statement-based replication](/ch6#statement-based-replication) - (see also deterministic operations) - in distributed systems, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - in workflow engines, [Durable execution](/ch5#durable-execution) - partial failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - sources of nondeterminism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - nonfunctional requirements, [Defining Nonfunctional Requirements](/ch2#ch_nonfunctional), [Summary](/ch2#summary) - nonrepeatable reads, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - (see also read skew) - normalization (data representation), [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization)-[Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many), [Glossary](/glossary) - foreign key references, [The need for multi-object transactions](/ch8#sec_transactions_need) - in social network case study, [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - in systems of record, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - versus denormalization, [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - NoSQL, [Relational Model versus Document Model](/ch3#sec_datamodels_history), [Solutions for Replication Lag](/ch6#id131), [Unbundling Databases](/ch13#sec_future_unbundling) - transactions and, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview) - Notation3 (N3), [Triple-Stores and SPARQL](/ch3#id59) - NTP (Network Time Protocol), [Unreliable Clocks](/ch9#sec_distributed_clocks) - accuracy, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy), [Timestamps for ordering events](/ch9#sec_distributed_lww) - adjustments to monotonic clocks, [Monotonic clocks](/ch9#monotonic-clocks) - multiple server addresses, [Weak forms of lying](/ch9#weak-forms-of-lying) - numbers, in XML and JSON encodings, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - NumPy (Python library), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes), [Column-Oriented Storage](/ch4#sec_storage_column) - NVMe (Non-Volatile Memory Express) (see solid state drives (SSDs)) ### O - object databases, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - object storage, [Layering of cloud services](/ch1#layering-of-cloud-services), [Object Stores](/ch11#id277)-[Object Stores](/ch11#id277) - Azure Blob Storage (see Azure Blob Storage) - comparison to distributed filesystems, [Object Stores](/ch11#id277) - comparison to key-value stores, [Object Stores](/ch11#id277) - databases backed by, [Setting Up New Followers](/ch6#sec_replication_new_replica) - for backups, [Replication](/ch6#ch_replication) - for cloud data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - for database replication, [Setting Up New Followers](/ch6#sec_replication_new_replica) - Google Cloud Storage (see Google Cloud Storage) - object size, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - S3 (see S3 (object storage)) - storing LSM segment files, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - support for fencing, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - use in data lakes, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - object-relational mapping (ORM) frameworks, [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm) - error handling and aborted transactions, [Handling errors and aborts](/ch8#handling-errors-and-aborts) - unsafe read-modify-write cycle code, [Atomic write operations](/ch8#atomic-write-operations) - object-relational mismatch, [The Object-Relational Mismatch](/ch3#sec_datamodels_document) - observability, [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems), [Humans and Reliability](/ch2#id31), [Operability: Making Life Easy for Operations](/ch2#id37) - observer pattern, [Separation of application code and state](/ch13#id344) - OBT (one big table), [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics), [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - offline systems, [Batch Processing](/ch11#ch_batch) - (see also batch processing) - offline-first applications, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps), [Stateful, offline-capable clients](/ch13#id347) - offsets - consumer offsets in sharded logs, [Consumer offsets](/ch12#sec_stream_log_offsets) - messages in sharded logs, [Using logs for message storage](/ch12#id300) - OLAP (online analytic processing), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Glossary](/glossary) - data cubes, [Materialized Views and Data Cubes](/ch4#sec_storage_materialized_views) - OLTP (online transaction processing), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Glossary](/glossary) - analytics queries versus, [Analytics](/ch11#sec_batch_olap) - data normalization, [Trade-offs of normalization](/ch3#trade-offs-of-normalization) - workload characteristics, [Actual Serial Execution](/ch8#sec_transactions_serial) - on-premises deployment, [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud) - data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - one big table (data warehouse schema), [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics), [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - one-hot encoding, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - one-to-few relationships, [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - one-to-many relationships, [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - JSON representation, [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - online systems, [Batch Processing](/ch11#ch_batch) - (see also services) - versus scientific computing, [Cloud Computing Versus Supercomputing](/ch1#id17) - ontologies, [Triple-Stores and SPARQL](/ch3#id59) - Oozie (workflow scheduler), [Batch Processing](/ch11#ch_batch) - OpenAPI (service definition format), [Microservices and Serverless](/ch1#sec_introduction_microservices), [Web services](/ch5#sec_web_services), [Web services](/ch5#sec_web_services) - use of JSON Schema, [JSON Schema](/ch5#json-schema) - openCypher (see Cypher (query language)) - OpenLink Virtuoso (see Virtuoso (database)) - OpenStack - Swift (object storage), [Object Stores](/ch11#id277) - operability, [Operability: Making Life Easy for Operations](/ch2#id37) - operating systems versus databases, [Unbundling Databases](/ch13#sec_future_unbundling) - operational systems, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - (see also OLTP) - as systems of record, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - ETL into analytical systems, [Data Warehousing](/ch1#sec_introduction_dwh) - operational transformation, [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - operations teams, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - operators (query execution), [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - in stream processing, [Processing Streams](/ch12#sec_stream_processing) - optimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - optimistic locking, [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set) - Oracle (database) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - GoldenGate (change data capture), [Implementing change data capture](/ch12#id307) - hierarchical queries, [Graph Queries in SQL](/ch3#id58), [Graph Queries in SQL](/ch3#id58) - lack of serializability, [Isolation](/ch8#sec_transactions_acid_isolation) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - multi-table index cluster tables, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - not preventing write skew, [Characterizing write skew](/ch8#characterizing-write-skew) - PL/SQL language, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - preventing lost updates, [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - read committed isolation, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - Real Application Clusters (RAC), [Locking and leader election](/ch10#locking-and-leader-election) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation), [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - TimesTen (in-memory database), [Keeping everything in memory](/ch4#sec_storage_inmemory) - WAL-based replication, [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - ORC (data format), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Column-Oriented Storage](/ch4#sec_storage_column) - (see also column-oriented storage) - orchestration (service deployment), [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud), [Microservices and Serverless](/ch1#sec_introduction_microservices) - batch job execution, [Distributed Job Orchestration](/ch11#id278)-[Distributed Job Orchestration](/ch11#id278) - workflow engines, [Batch Processing](/ch11#ch_batch) - ordering - event logs, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - limits of total ordering, [The limits of total ordering](/ch13#id335) - logical timestamps, [Logical Clocks](/ch10#sec_consistency_timestamps) - of auto-incrementing IDs, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - shared logs, [Consensus in Practice](/ch10#sec_consistency_total_order)-[Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - Orkes (workflow engine), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - orphan pages (B-trees), [Making B-trees reliable](/ch4#sec_storage_btree_wal) - outbox pattern, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - outliers (response time), [Average, Median, and Percentiles](/ch2#id24) - outsourcing, [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud) - overload, [Describing Performance](/ch2#sec_introduction_percentiles), [Handling errors and aborts](/ch8#handling-errors-and-aborts) ### P - PACELC principle, [The CAP theorem](/ch10#the-cap-theorem) - package managers, [Separation of application code and state](/ch13#id344) - packet switching, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - packets - corruption of, [Weak forms of lying](/ch9#weak-forms-of-lying) - sending via UDP, [Direct messaging from producers to consumers](/ch12#id296) - PageRank (algorithm), [Graph-Like Data Models](/ch3#sec_datamodels_graph), [Query languages](/ch11#sec_batch_query_lanauges), [Machine Learning](/ch11#id290) - paging (see virtual memory) - pandas (Python library), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes), [Column-Oriented Storage](/ch4#sec_storage_column), [DataFrames](/ch11#id287) - Parquet (data format), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Column-Oriented Storage](/ch4#sec_storage_column), [Archival storage](/ch5#archival-storage), [Query languages](/ch11#sec_batch_query_lanauges) - (see also column-oriented storage) - databases on object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - document data model, [Column-Oriented Storage](/ch4#sec_storage_column) - use in batch processing, [MapReduce](/ch11#sec_batch_mapreduce) - partial failures, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure), [Summary](/ch9#summary) - limping, [System Model and Reality](/ch9#sec_distributed_system_model) - partial synchrony (system model), [System Model and Reality](/ch9#sec_distributed_system_model) - partition key, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons), [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - partitioning (see sharding) - Paxos (consensus algorithm), [Consensus](/ch10#sec_consistency_consensus), [Consensus in Practice](/ch10#sec_consistency_total_order) - ballot number, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - Multi-Paxos, [Consensus in Practice](/ch10#sec_consistency_total_order) - payment card industry (PCI), [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - PCI (payment card industry) compliance, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - percentiles, [Average, Median, and Percentiles](/ch2#id24), [Glossary](/glossary) - calculating efficiently, [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - importance of high percentiles, [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - use in service level agreements (SLAs), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - Percolator (Google), [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - Percona XtraBackup (MySQL tool), [Setting Up New Followers](/ch6#sec_replication_new_replica) - performance - degradation as fault, [System Model and Reality](/ch9#sec_distributed_system_model) - describing, [Describing Performance](/ch2#sec_introduction_percentiles) - of distributed transactions, [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa) - of in-memory databases, [Keeping everything in memory](/ch4#sec_storage_inmemory) - of linearizability, [Linearizability and network delays](/ch10#linearizability-and-network-delays) - of multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - permission isolation, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - perpetual inconsistency, [Timeliness and Integrity](/ch13#sec_future_integrity) - pessimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - pglogical (PostgreSQL extension), [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - pgvector (vector index), [Vector Embeddings](/ch4#id92) - phantoms (transaction isolation), [Phantoms causing write skew](/ch8#sec_transactions_phantom) - materializing conflicts, [Materializing conflicts](/ch8#materializing-conflicts) - preventing, in serializability, [Predicate locks](/ch8#predicate-locks) - physical clocks (see clocks) - pickle (Python), [Language-Specific Formats](/ch5#id96) - Pinot (database), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Column-Oriented Storage](/ch4#sec_storage_column) - handling writes, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - pre-aggregation, [Analytics](/ch11#sec_batch_olap) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived), [Serving Derived Data](/ch11#sec_batch_serving_derived) - pipelined execution - in data warehouse queries, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - pivot table, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - point in time, [Unreliable Clocks](/ch9#sec_distributed_clocks) - point query, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - Polaris (data catalog), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - polling, [Representing Users, Posts, and Follows](/ch2#id20) - polystores, [The meta-database of everything](/ch13#id341) - POSIX (portable operating system interface) - compliant filesystems, [Setting Up New Followers](/ch6#sec_replication_new_replica), [Distributed Filesystems](/ch11#sec_batch_dfs), [Object Stores](/ch11#id277) - Post Office Horizon scandal, [Humans and Reliability](/ch2#id31) - lack of transactions, [Transactions](/ch8#ch_transactions) - PostgreSQL (database) - archiving WAL to object stores, [Setting Up New Followers](/ch6#sec_replication_new_replica) - change data capture, [Implementing change data capture](/ch12#id307), [API support for change streams](/ch12#sec_stream_change_api) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - foreign data wrappers, [The meta-database of everything](/ch13#id341) - full text search support, [Combining Specialized Tools by Deriving Data](/ch13#id442) - in the cloud, [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - JSON Schema validation, [JSON Schema](/ch5#json-schema) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - log sequence number, [Setting Up New Followers](/ch6#sec_replication_new_replica) - logical decoding, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - materialized view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - MVCC implementation, [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl), [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - partitioning vs. sharding, [Sharding](/ch7#ch_sharding) - pgvector (vector index), [Vector Embeddings](/ch4#id92) - PL/pgSQL language, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - PostGIS geospatial indexes, [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - preventing lost updates, [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - preventing write skew, [Characterizing write skew](/ch8#characterizing-write-skew), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi) - read committed isolation, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - representing graphs, [Property Graphs](/ch3#id56) - serializable snapshot isolation (SSI), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi) - sharding (see Citus (database)) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation), [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - WAL-based replication, [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - postings list, [Full-Text Search](/ch4#sec_storage_full_text) - in sharded indexes, [Local Secondary Indexes](/ch7#id166) - postmortems, blameless, [Humans and Reliability](/ch2#id31) - PouchDB (database), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - Power BI (business intelligence software), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Analytics](/ch11#sec_batch_olap) - pre-aggregation, [Analytics](/ch11#sec_batch_olap) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - pre-splitting, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - Precision Time Protocol (PTP), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - predicate locks, [Predicate locks](/ch8#predicate-locks) - predictive analytics, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics), [Predictive Analytics](/ch14#id369)-[Feedback Loops](/ch14#id372) - amplifying bias, [Bias and Discrimination](/ch14#id370) - ethics of (see ethics) - feedback loops, [Feedback Loops](/ch14#id372) - preemption, [Resource Allocation](/ch11#id279) - in distributed schedulers, [Handling Faults](/ch11#id281) - of threads, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - Prefect (workflow scheduler), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows), [Batch Processing](/ch11#ch_batch), [Scheduling Workflows](/ch11#sec_batch_workflows) - cloud data warehouse integration, [Query languages](/ch11#sec_batch_query_lanauges) - Presto (query engine), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - primary keys, [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn), [Glossary](/glossary) - auto-incrementing, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - versus partition key, [Sharding by hash range](/ch7#sharding-by-hash-range) - primary-backup replication (see leader-based replication) - privacy, [Privacy and Tracking](/ch14#id373)-[Legislation and Self-Regulation](/ch14#sec_future_legislation) - consent and freedom of choice, [Consent and Freedom of Choice](/ch14#id375) - data as assets and power, [Data as Assets and Power](/ch14#id376) - deleting data, [Limitations of immutability](/ch12#sec_stream_immutability_limitations) - ethical considerations (see ethics) - legislation and self-regulation, [Legislation and Self-Regulation](/ch14#sec_future_legislation) - meaning of, [Privacy and Use of Data](/ch14#id457) - regulation, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - surveillance, [Surveillance](/ch14#id374) - tracking behavioral data, [Privacy and Tracking](/ch14#id373) - probabilistic algorithms, [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla), [Stream analytics](/ch12#id318) - process pauses, [Process Pauses](/ch9#sec_distributed_clocks_pauses)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - processing time (of events), [Reasoning About Time](/ch12#sec_stream_time) - producers (message streams), [Transmitting Event Streams](/ch12#sec_stream_transmit) - product analytics, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - column-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column) - programming languages - for stored procedures, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - projections (event sourcing), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - Prolog (language), [Datalog: Recursive Relational Queries](/ch3#id62) - (see also Datalog) - property graphs, [Property Graphs](/ch3#id56) - Cypher query language, [The Cypher Query Language](/ch3#id57) - Property Graph Query Language (PGQL), [Graph Queries in SQL](/ch3#id58) - property-based testing, [Humans and Reliability](/ch2#id31), [Formal Methods and Randomized Testing](/ch9#sec_distributed_formal) - Protocol Buffers (data format), [Protocol Buffers](/ch5#sec_encoding_protobuf)-[Field tags and schema evolution](/ch5#field-tags-and-schema-evolution), [Protocol Buffers](/ch5#sec_encoding_protobuf) - field tags and schema evolution, [Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - provenance of data, [Designing for auditability](/ch13#id365) - publish/subscribe model, [Messaging Systems](/ch12#sec_stream_messaging) - publishers (message streams), [Transmitting Event Streams](/ch12#sec_stream_transmit) - Pulsar (streaming platform), [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - PyTorch (machine learning library), [Machine Learning](/ch11#id290) ### Q - Qpid (messaging), [Message brokers compared to databases](/ch12#id297) - quality of service (QoS), [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - Quantcast File System (distributed filesystem), [Object Stores](/ch11#id277) - query engines - compilation and vectorization, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - in cloud data warehouse, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - operators, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - optimizing declarative queries, [Data Models and Query Languages](/ch3#ch_datamodels) - query languages - Cypher, [The Cypher Query Language](/ch3#id57) - Datalog, [Datalog: Recursive Relational Queries](/ch3#id62) - GraphQL, [GraphQL](/ch3#id63) - MongoDB aggregation pipeline, [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization), [Query languages for documents](/ch3#query-languages-for-documents) - recursive SQL queries, [Graph Queries in SQL](/ch3#id58) - SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language) - SQL, [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization) - query optimizers, [Query languages](/ch11#sec_batch_query_lanauges) - query plans, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - queueing delays, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - head-of-line blocking, [Latency and Response Time](/ch2#id23) - latency and response time, [Latency and Response Time](/ch2#id23) - queues (messaging), [Message brokers](/ch5#message-brokers) - QUIC (protocol), [The Limitations of TCP](/ch9#sec_distributed_tcp) - quorums, [Quorums for reading and writing](/ch6#sec_replication_quorum_condition)-[Multi-region operation](/ch6#multi-region-operation), [Glossary](/glossary) - for leaderless replication, [Quorums for reading and writing](/ch6#sec_replication_quorum_condition) - in consensus algorithms, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - limitations of consistency, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations)-[Monitoring staleness](/ch6#monitoring-staleness), [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - making decisions in distributed systems, [The Majority Rules](/ch9#sec_distributed_majority) - monitoring staleness, [Monitoring staleness](/ch6#monitoring-staleness) - multi-region replication, [Multi-region operation](/ch6#multi-region-operation) - relying on durability, [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - quotas, [Operations in the Cloud Era](/ch1#sec_introduction_operations) ### R - R (language), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes), [DataFrames](/ch11#id287) - R-trees (indexes), [Multidimensional and Full-Text Indexes](/ch4#sec_storage_multidimensional) - R2 (object storage), [Layering of cloud services](/ch1#layering-of-cloud-services), [Distributed Filesystems](/ch11#sec_batch_dfs) - RabbitMQ (messaging), [Message brokers](/ch5#message-brokers), [Message brokers compared to databases](/ch12#id297) - quorum queues (replication), [Single-Leader Replication](/ch6#sec_replication_leader) - race conditions, [Isolation](/ch8#sec_transactions_acid_isolation) - (see also concurrency) - avoiding with linearizability, [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - caused by dual writes, [Keeping Systems in Sync](/ch12#sec_stream_sync) - causing loss of money, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - dirty writes, [No dirty writes](/ch8#sec_transactions_dirty_write) - in counter increments, [No dirty writes](/ch8#sec_transactions_dirty_write) - lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update)-[Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - preventing with event logs, [Concurrency control](/ch12#sec_stream_concurrency), [Dataflow: Interplay between state changes and application code](/ch13#id450) - preventing with serializable isolation, [Serializability](/ch8#sec_transactions_serializability) - weak transaction isolation, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels) - write skew, [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts) - Raft (consensus algorithm), [Consensus](/ch10#sec_consistency_consensus), [Consensus in Practice](/ch10#sec_consistency_total_order) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - sensitivity to network problems, [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - term number, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - use in etcd, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - RAID (Redundant Array of Independent Disks), [Separation of storage and compute](/ch1#sec_introduction_storage_compute), [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy), [Distributed Filesystems](/ch11#sec_batch_dfs) - railways, schema migration on, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - RAM (see memory) - RAMCloud (in-memory storage), [Keeping everything in memory](/ch4#sec_storage_inmemory) - random writes (access pattern), [Sequential versus random writes](/ch4#sidebar_sequential) - range queries - in B-trees, [B-Trees](/ch4#sec_storage_b_trees), [Read performance](/ch4#read-performance) - in LSM-trees, [Read performance](/ch4#read-performance) - not efficient in hash maps, [Log-Structured Storage](/ch4#sec_storage_log_structured) - with hash sharding, [Sharding by hash range](/ch7#sharding-by-hash-range) - ranking algorithms, [Machine Learning](/ch11#id290) - Ray (workflow scheduler), [Machine Learning](/ch11#id290) - RDF (Resource Description Framework), [The RDF data model](/ch3#the-rdf-data-model) - querying with SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language) - RDMA (Remote Direct Memory Access), [Layering of cloud services](/ch1#layering-of-cloud-services), [Cloud Computing Versus Supercomputing](/ch1#id17) - React (user interface library), [End-to-end event streams](/ch13#id349) - reactive programming, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - read committed isolation level, [Read Committed](/ch8#sec_transactions_read_committed)-[Implementing read committed](/ch8#sec_transactions_read_committed_impl) - implementing, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - multi-version concurrency control (MVCC), [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl) - no dirty reads, [No dirty reads](/ch8#no-dirty-reads) - no dirty writes, [No dirty writes](/ch8#sec_transactions_dirty_write) - read models (event sourcing), [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - read path (derived data), [Observing Derived State](/ch13#sec_future_observing) - read repair (leaderless replication), [Catching up on missed writes](/ch6#sec_replication_read_repair) - for linearizability, [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - read replicas (see leader-based replication) - read skew (transaction isolation), [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation), [Summary](/ch8#summary) - read uncommitted isolation level, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - read-after-write consistency, [Reading Your Own Writes](/ch6#sec_replication_ryw), [Timeliness and Integrity](/ch13#sec_future_integrity) - cross-device, [Reading Your Own Writes](/ch6#sec_replication_ryw) - in derived data systems, [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions) - read-modify-write cycle, [Preventing Lost Updates](/ch8#sec_transactions_lost_update) - read-scaling architecture, [Problems with Replication Lag](/ch6#sec_replication_lag), [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - versus sharding, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - reads as events, [Reads are events too](/ch13#sec_future_read_events) - real-time - analytics (see product analytics) - collaborative editing, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - publish/subscribe dataflow, [End-to-end event streams](/ch13#id349) - response time guarantees, [Response time guarantees](/ch9#sec_distributed_clocks_realtime) - time-of-day clocks, [Time-of-day clocks](/ch9#time-of-day-clocks) - Realm (database), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - rebalancing shards, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data)-[Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations), [Glossary](/glossary) - (see also sharding) - automatic or manual rebalancing, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - fixed number of shards, [Fixed number of shards](/ch7#fixed-number-of-shards) - fixed number of shards per node, [Sharding by hash range](/ch7#sharding-by-hash-range) - problems with hash mod N, [Hash modulo number of nodes](/ch7#hash-modulo-number-of-nodes) - recency guarantee, [Linearizability](/ch10#sec_consistency_linearizability) - recommendation engines, [Operational Versus Analytical Systems](/ch1#sec_introduction_analytics) - building using DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - iterative processing, [Machine Learning](/ch11#id290) - reconfiguration (consensus), [Subtleties of consensus](/ch10#subtleties-of-consensus) - records, [MapReduce](/ch11#sec_batch_mapreduce) - events in stream processing, [Transmitting Event Streams](/ch12#sec_stream_transmit) - recursive queries - in Cypher, [The Cypher Query Language](/ch3#id57) - in Datalog, [Datalog: Recursive Relational Queries](/ch3#id62) - in SPARQL, [The SPARQL query language](/ch3#the-sparql-query-language) - lack of, in GraphQL, [GraphQL](/ch3#id63) - SQL common table expressions, [Graph Queries in SQL](/ch3#id58) - Red Hat - Apicurio Registry, [JSON Schema](/ch5#json-schema) - red-black tree, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - redelivery (messaging), [Acknowledgments and redelivery](/ch12#sec_stream_reordering) - Redis (database) - atomic operations, [Atomic write operations](/ch8#atomic-write-operations) - CRDT support, [CRDTs and Operational Transformation](/ch6#sec_replication_crdts) - durability, [Keeping everything in memory](/ch4#sec_storage_inmemory) - Lua scripting, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - process-per-core model, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - single-threaded execution, [Actual Serial Execution](/ch8#sec_transactions_serial) - redo log (see write-ahead log) - Redpanda (messaging), [Message brokers](/ch5#message-brokers), [Setting Up New Followers](/ch6#sec_replication_new_replica) - tiered storage, [Disk space usage](/ch12#sec_stream_disk_usage) - Redshift (database), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - redundancy - hardware components, [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy) - of derived data, [Systems of Record and Derived Data](/ch1#sec_introduction_derived) - (see also derived data) - Reed--Solomon codes (error correction), [Distributed Filesystems](/ch11#sec_batch_dfs) - refactoring, [Evolvability: Making Change Easy](/ch2#sec_introduction_evolvability) - (see also evolvability) - regions (geographic distribution), [Reading Your Own Writes](/ch6#sec_replication_ryw) - (see also datacenters) - consensus across, [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - definition, [Reading Your Own Writes](/ch6#sec_replication_ryw) - latency, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - linearizable ID generation, [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - replication across, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc)-[Problems with different topologies](/ch6#problems-with-different-topologies), [The Cost of Linearizability](/ch10#sec_linearizability_cost), [The limits of total ordering](/ch13#id335) - leaderless, [Multi-region operation](/ch6#multi-region-operation) - multi-leader, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - regions (sharding), [Sharding](/ch7#ch_sharding) - register (data structure), [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - regulation (see legal matters) - relational data model, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [Relational Model versus Document Model](/ch3#sec_datamodels_history)-[Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - comparison to document model, [When to Use Which Model](/ch3#sec_datamodels_document_summary)-[Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - graph queries in SQL, [Graph Queries in SQL](/ch3#id58) - in-memory databases with, [Keeping everything in memory](/ch4#sec_storage_inmemory) - many-to-one and many-to-many relationships, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many) - multi-object transactions, need for, [The need for multi-object transactions](/ch8#sec_transactions_need) - object-relational mismatch, [The Object-Relational Mismatch](/ch3#sec_datamodels_document) - representing a reorderable list, [When to Use Which Model](/ch3#sec_datamodels_document_summary) - versus document model - convergence of models, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - data locality, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - relational databases - eventual consistency, [Problems with Replication Lag](/ch6#sec_replication_lag) - history, [Relational Model versus Document Model](/ch3#sec_datamodels_history) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - logical logs, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - philosophy compared to Unix, [Unbundling Databases](/ch13#sec_future_unbundling), [The meta-database of everything](/ch13#id341) - schema changes, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility), [Encoding and Evolution](/ch5#ch_encoding), [Different values written at different times](/ch5#different-values-written-at-different-times) - sharded secondary indexes, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication) - use of B-tree indexes, [B-Trees](/ch4#sec_storage_b_trees) - relationships (see edges) - reliability, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability)-[Humans and Reliability](/ch2#id31), [A Philosophy of Streaming Systems](/ch13#ch_philosophy) - building a reliable system from unreliable components, [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - hardware faults, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - human errors, [Humans and Reliability](/ch2#id31) - importance of, [Humans and Reliability](/ch2#id31) - of messaging systems, [Messaging Systems](/ch12#sec_stream_messaging) - software faults, [Software faults](/ch2#software-faults) - Remote Method Invocation (Java RMI), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - remote procedure calls (RPCs), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc)-[Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - (see also services) - data encoding and evolution, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - issues with, [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - using Avro, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - versus message brokers, [Event-Driven Architectures](/ch5#sec_encoding_dataflow_msg) - renewable energy, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - repeatable reads (transaction isolation), [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - replicas, [Single-Leader Replication](/ch6#sec_replication_leader) - replication, [Replication](/ch6#ch_replication)-[Summary](/ch6#summary), [Glossary](/glossary) - and durability, [Durability](/ch8#durability) - conflict resolution and, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - consistency properties, [Problems with Replication Lag](/ch6#sec_replication_lag)-[Solutions for Replication Lag](/ch6#id131) - consistent prefix reads, [Consistent Prefix Reads](/ch6#sec_replication_consistent_prefix) - monotonic reads, [Monotonic Reads](/ch6#sec_replication_monotonic_reads) - reading your own writes, [Reading Your Own Writes](/ch6#sec_replication_ryw) - in distributed filesystems, [Distributed Filesystems](/ch11#sec_batch_dfs) - leaderless, [Leaderless Replication](/ch6#sec_replication_leaderless)-[Version vectors](/ch6#version-vectors) - detecting concurrent writes, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent)-[Version vectors](/ch6#version-vectors) - limitations of quorum consistency, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations)-[Monitoring staleness](/ch6#monitoring-staleness), [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - monitoring staleness, [Monitoring staleness](/ch6#monitoring-staleness) - multi-leader, [Multi-Leader Replication](/ch6#sec_replication_multi_leader)-[Types of conflict](/ch6#sec_replication_write_conflicts) - across multiple regions, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc), [The Cost of Linearizability](/ch10#sec_linearizability_cost) - conflict resolution, [Dealing with Conflicting Writes](/ch6#sec_replication_write_conflicts)-[Types of conflict](/ch6#sec_replication_write_conflicts) - replication topologies, [Multi-leader replication topologies](/ch6#sec_replication_topologies)-[Problems with different topologies](/ch6#problems-with-different-topologies) - reasons for using, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Replication](/ch6#ch_replication) - sharding and, [Sharding](/ch7#ch_sharding) - single-leader, [Single-Leader Replication](/ch6#sec_replication_leader)-[Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - failover, [Leader failure: Failover](/ch6#leader-failure-failover) - implementation of replication logs, [Implementation of Replication Logs](/ch6#sec_replication_implementation)-[Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - relation to consensus, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus), [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - setting up new followers, [Setting Up New Followers](/ch6#sec_replication_new_replica) - synchronous versus asynchronous, [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async)-[Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async) - state machine replication, [Statement-based replication](/ch6#statement-based-replication), [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs), [Using shared logs](/ch10#sec_consistency_smr), [Databases and Streams](/ch12#sec_stream_databases) - event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - using consensus, [Pros and cons of consensus](/ch10#pros-and-cons-of-consensus) - using erasure coding, [Distributed Filesystems](/ch11#sec_batch_dfs) - using object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - versus backups, [Replication](/ch6#ch_replication) - with heterogeneous data systems, [Keeping Systems in Sync](/ch12#sec_stream_sync) - replication logs (see logs) - representations of data (see data models) - reprocessing data, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing), [Unifying batch and stream processing](/ch13#id338) - (see also evolvability) - from log-based messaging, [Replaying old messages](/ch12#sec_stream_replay) - request hedging, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - request identifiers, [Uniquely identifying requests](/ch13#id355), [Multi-shard request processing](/ch13#id360) - request routing, [Request Routing](/ch7#sec_sharding_routing)-[Request Routing](/ch7#sec_sharding_routing) - approaches to, [Request Routing](/ch7#sec_sharding_routing) - residence laws for data, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed), [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - resilient systems, [Reliability and Fault Tolerance](/ch2#sec_introduction_reliability) - (see also fault tolerance) - resource isolation, [Cloud Computing Versus Supercomputing](/ch1#id17), [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - resource limits, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - response time - as performance metric, [Describing Performance](/ch2#sec_introduction_percentiles), [Batch Processing](/ch11#ch_batch) - guarantees on, [Response time guarantees](/ch9#sec_distributed_clocks_realtime) - impact on users, [Average, Median, and Percentiles](/ch2#id24) - in replicated systems, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - latency versus, [Latency and Response Time](/ch2#id23) - mean and percentiles, [Average, Median, and Percentiles](/ch2#id24) - user experience, [Average, Median, and Percentiles](/ch2#id24) - responsibility and accountability, [Responsibility and Accountability](/ch14#id371) - REST (Representational State Transfer), [Web services](/ch5#sec_web_services) - (see also services) - Restate (workflow engine), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - RethinkDB (database) - join support, [Convergence of document and relational databases](/ch3#convergence-of-document-and-relational-databases) - key-range sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - retry storm, [Describing Performance](/ch2#sec_introduction_percentiles), [Software faults](/ch2#software-faults) - reverse ETL, [Beyond the data lake](/ch1#beyond-the-data-lake) - Riak (database) - CRDT support, [CRDTs and Operational Transformation](/ch6#sec_replication_crdts), [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - dotted version vectors, [Version vectors](/ch6#version-vectors) - gossip protocol, [Request Routing](/ch7#sec_sharding_routing) - hash sharding, [Fixed number of shards](/ch7#fixed-number-of-shards) - leaderless replication, [Leaderless Replication](/ch6#sec_replication_leaderless) - linearizability, lack of, [Linearizability and quorums](/ch10#sec_consistency_quorum_linearizable) - multi-region support, [Multi-region operation](/ch6#multi-region-operation) - rebalancing, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - secondary indexes, [Local Secondary Indexes](/ch7#id166) - sloppy quorums, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - vnodes (sharding), [Sharding](/ch7#ch_sharding) - ring buffers, [Disk space usage](/ch12#sec_stream_disk_usage) - RisingWave (database) - incremental view maintenance, [Maintaining materialized views](/ch12#sec_stream_mat_view) - rockets, [Byzantine Faults](/ch9#sec_distributed_byzantine) - RocksDB (storage engine), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - as embedded storage engine, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - leveled compaction, [Compaction strategies](/ch4#sec_storage_lsm_compaction) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - rollbacks (transactions), [Transactions](/ch8#ch_transactions) - rolling upgrades, [Tolerating hardware faults through redundancy](/ch2#tolerating-hardware-faults-through-redundancy), [Encoding and Evolution](/ch5#ch_encoding), [Faults and Partial Failures](/ch9#sec_distributed_partial_failure) - in a multitenant system, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - routing (see request routing) - row-based replication, [Logical (row-based) log replication](/ch6#logical-row-based-log-replication) - row-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column) - rowhammer (memory corruption), [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - RPCs (see remote procedure calls) - rules (Datalog), [Datalog: Recursive Relational Queries](/ch3#id62) - Rust (programming language) - memory management, [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) ### S - S3 (object storage), [Layering of cloud services](/ch1#layering-of-cloud-services), [Setting Up New Followers](/ch6#sec_replication_new_replica), [Batch Processing](/ch11#ch_batch), [Distributed Filesystems](/ch11#sec_batch_dfs), [Object Stores](/ch11#id277) - checking data integrity, [Don't just blindly trust what they promise](/ch13#id364) - conditional writes, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - object size, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - S3 Express One Zone, [Object Stores](/ch11#id277), [Object Stores](/ch11#id277) - use in MapReduce, [MapReduce](/ch11#sec_batch_mapreduce) - workflow example, [Scheduling Workflows](/ch11#sec_batch_workflows) - SaaS (see software as a service (SaaS)) - safety and liveness properties, [Safety and liveness](/ch9#sec_distributed_safety_liveness) - in consensus algorithms, [Single-value consensus](/ch10#single-value-consensus) - in transactions, [Transactions](/ch8#ch_transactions) - sagas (see compensating transactions) - Samza (stream processor), [Stream analytics](/ch12#id318) - SAP HANA (database), [Data Storage for Analytics](/ch4#sec_storage_analytics) - scalability, [Scalability](/ch2#sec_introduction_scalability)-[Principles for Scalability](/ch2#id35), [A Philosophy of Streaming Systems](/ch13#ch_philosophy) - auto-scaling, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - by sharding, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - describing load, [Describing Load](/ch2#id33) - describing performance, [Describing Performance](/ch2#sec_introduction_percentiles) - linear, [Describing Load](/ch2#id33) - principles for, [Principles for Scalability](/ch2#id35) - replication and, [Problems with Replication Lag](/ch6#sec_replication_lag) - scaling up versus scaling out, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing) - scaling out, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing) - (see also shared-nothing architecture) - by sharding, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - scaling up, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing) - SCD (slowly changing dimension), [Time-dependence of joins](/ch12#sec_stream_join_time) - scheduling - algorithms, [Resource Allocation](/ch11#id279) - batch jobs, [Distributed Job Orchestration](/ch11#id278)-[Scheduling Workflows](/ch11#sec_batch_workflows) - gang scheduling, [Resource Allocation](/ch11#id279) - schema-on-read, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - comparison to evolvable schema, [The Merits of Schemas](/ch5#sec_encoding_schemas) - schema-on-write, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - schemaless databases (see schema-on-read) - schemas, [Glossary](/glossary) - Avro, [Avro](/ch5#sec_encoding_avro)-[Dynamically generated schemas](/ch5#dynamically-generated-schemas) - reader determining writer's schema, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - schema evolution, [The writer's schema and the reader's schema](/ch5#the-writers-schema-and-the-readers-schema) - dynamically generated, [Dynamically generated schemas](/ch5#dynamically-generated-schemas) - evolution of, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - affecting application code, [Encoding and Evolution](/ch5#ch_encoding) - compatibility checking, [But what is the writer's schema?](/ch5#but-what-is-the-writers-schema) - in databases, [Dataflow Through Databases](/ch5#sec_encoding_dataflow_db)-[Archival storage](/ch5#archival-storage) - in service calls, [Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - flexibility in document model, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - for analytics, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics)-[Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - for JSON and XML, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json), [JSON Schema](/ch5#json-schema) - generation and migration using ORMs, [Object-relational mapping (ORM)](/ch3#object-relational-mapping-orm) - merits of, [The Merits of Schemas](/ch5#sec_encoding_schemas) - migration, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - Protocol Buffers, [Protocol Buffers](/ch5#sec_encoding_protobuf)-[Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - schema evolution, [Field tags and schema evolution](/ch5#field-tags-and-schema-evolution) - schema migration on railways, [Reprocessing data for application evolution](/ch13#sec_future_reprocessing) - traditional approach to design, fallacy in, [Deriving several views from the same event log](/ch12#sec_stream_deriving_views) - scientific computing, [Cloud Computing Versus Supercomputing](/ch1#id17) - scikit-learn (Python library), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - ScyllaDB (database) - cluster metadata, [Request Routing](/ch7#sec_sharding_routing) - consistency level ANY, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - hash-range sharding, [Sharding by Hash of Key](/ch7#sec_sharding_hash), [Sharding by hash range](/ch7#sharding-by-hash-range) - last-write-wins conflict resolution, [Detecting Concurrent Writes](/ch6#sec_replication_concurrent) - leaderless replication, [Leaderless Replication](/ch6#sec_replication_leaderless) - lightweight transactions, [Single-object writes](/ch8#sec_transactions_single_object) - linearizability, lack of, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - log-structured storage, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - multi-region support, [Multi-region operation](/ch6#multi-region-operation) - use of clocks, [Limitations of Quorum Consistency](/ch6#sec_replication_quorum_limitations), [Timestamps for ordering events](/ch9#sec_distributed_lww) - vnodes (sharding), [Sharding](/ch7#ch_sharding) - search engines (see full-text search) - searching on streams, [Search on streams](/ch12#id320) - secondaries (see leader-based replication) - secondary indexes, [Multi-Column and Secondary Indexes](/ch4#sec_storage_index_multicolumn), [Glossary](/glossary) - for many-to-many relationships, [Many-to-One and Many-to-Many Relationships](/ch3#sec_datamodels_many_to_many) - problems with dual writes, [Keeping Systems in Sync](/ch12#sec_stream_sync), [Reasoning about dataflows](/ch13#id443) - sharding, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes)-[Global Secondary Indexes](/ch7#id167), [Summary](/ch7#summary) - global, [Global Secondary Indexes](/ch7#id167) - index maintenance, [Maintaining derived state](/ch13#id446) - local, [Local Secondary Indexes](/ch7#id166) - updating, transaction isolation and, [The need for multi-object transactions](/ch8#sec_transactions_need) - secondary sort (MapReduce), [JOIN and GROUP BY](/ch11#sec_batch_join) - sed (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis) - self-hosting, [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud) - data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - self-joins, [Summary](/ch12#id332) - self-validating systems, [Don't just blindly trust what they promise](/ch13#id364) - semantic search, [Vector Embeddings](/ch4#id92) - semantic similarity, [Vector Embeddings](/ch4#id92) - semantic web, [Triple-Stores and SPARQL](/ch3#id59) - semi-synchronous replication, [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async) - sequential writes (access pattern), [Sequential versus random writes](/ch4#sidebar_sequential) - serializability, [Isolation](/ch8#sec_transactions_acid_isolation), [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels), [Serializability](/ch8#sec_transactions_serializability)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation), [Glossary](/glossary) - linearizability versus, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - pessimistic versus optimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - serial execution, [Actual Serial Execution](/ch8#sec_transactions_serial)-[Summary of serial execution](/ch8#summary-of-serial-execution) - sharding, [Sharding](/ch8#sharding) - using stored procedures, [Encapsulating transactions in stored procedures](/ch8#encapsulating-transactions-in-stored-procedures), [Using shared logs](/ch10#sec_consistency_smr) - serializable snapshot isolation (SSI), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - detecting stale MVCC reads, [Detecting stale MVCC reads](/ch8#detecting-stale-mvcc-reads) - detecting writes that affect prior reads, [Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - distributed execution, [Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - performance of SSI, [Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - preventing write skew, [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise)-[Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - strict serializability, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - timeliness vs. integrity, [Timeliness and Integrity](/ch13#sec_future_integrity) - two-phase locking (2PL), [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl)-[Index-range locks](/ch8#sec_transactions_2pl_range) - index-range locks, [Index-range locks](/ch8#sec_transactions_2pl_range) - performance, [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - Serializable (Java), [Language-Specific Formats](/ch5#id96) - serialization, [Formats for Encoding Data](/ch5#sec_encoding_formats) - (see also encoding) - serverless, [Microservices and Serverless](/ch1#sec_introduction_microservices) - service discovery, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery), [Request Routing](/ch7#sec_sharding_routing), [Service discovery](/ch10#service-discovery) - registration, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - using DNS, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery), [Request Routing](/ch7#sec_sharding_routing), [Service discovery](/ch10#service-discovery) - service level agreements (SLAs), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla), [Describing Load](/ch2#id33) - service mesh, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery) - Service Organization Control (SOC), [Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - service time, [Latency and Response Time](/ch2#id23) - service-oriented architecture (SOA), [Microservices and Serverless](/ch1#sec_introduction_microservices) - (see also services) - services, [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc)-[Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - microservices, [Microservices and Serverless](/ch1#sec_introduction_microservices) - causal dependencies across services, [The limits of total ordering](/ch13#id335) - loose coupling, [Making unbundling work](/ch13#sec_future_unbundling_favor) - relation to batch/stream processors, [Batch Processing](/ch11#ch_batch), [Stream processors and services](/ch13#id345) - remote procedure calls (RPCs), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc)-[Data encoding and evolution for RPC](/ch5#data-encoding-and-evolution-for-rpc) - issues with, [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - similarity to databases, [Dataflow Through Services: REST and RPC](/ch5#sec_encoding_dataflow_rpc) - web services, [Web services](/ch5#sec_web_services) - session windows (stream processing), [Types of windows](/ch12#id324) - (see also windows) - sharding, [Sharding](/ch7#ch_sharding)-[Summary](/ch7#summary), [Glossary](/glossary) - and consensus, [Using shared logs](/ch10#sec_consistency_smr) - and replication, [Sharding](/ch7#ch_sharding) - distributed transactions across shards, [Distributed Transactions](/ch8#sec_transactions_distributed) - hot shards, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - in batch processing, [Batch Processing](/ch11#ch_batch) - key-range splitting, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - multi-shard operations, [Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - enforcing constraints, [Multi-shard request processing](/ch13#id360) - secondary index maintenance, [Maintaining derived state](/ch13#id446) - of key-value data, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value)-[Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - by key range, [Sharding by Key Range](/ch7#sec_sharding_key_range) - skew and hot spots, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - origin of the term, [Sharding](/ch7#ch_sharding) - partition key, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons), [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - rebalancing - of key-range sharded data, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data) - rebalancing shards, [Rebalancing key-range sharded data](/ch7#rebalancing-key-range-sharded-data)-[Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - automatic or manual rebalancing, [Operations: Automatic or Manual Rebalancing](/ch7#sec_sharding_operations) - problems with hash mod N, [Hash modulo number of nodes](/ch7#hash-modulo-number-of-nodes) - using fixed number of shards, [Fixed number of shards](/ch7#fixed-number-of-shards) - using N shards per node, [Sharding by hash range](/ch7#sharding-by-hash-range) - request routing, [Request Routing](/ch7#sec_sharding_routing)-[Request Routing](/ch7#sec_sharding_routing) - secondary indexes, [Sharding and Secondary Indexes](/ch7#sec_sharding_secondary_indexes)-[Global Secondary Indexes](/ch7#id167) - global, [Global Secondary Indexes](/ch7#id167) - local, [Local Secondary Indexes](/ch7#id166) - serial execution of transactions and, [Sharding](/ch8#sharding) - sorting sharded data, [Shuffling Data](/ch11#sec_shuffle) - shared logs, [Consensus in Practice](/ch10#sec_consistency_total_order)-[Pros and cons of consensus](/ch10#pros-and-cons-of-consensus), [The limits of total ordering](/ch13#id335), [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - algorithms, [Consensus in Practice](/ch10#sec_consistency_total_order) - for event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - for messaging, [Log-based Message Brokers](/ch12#sec_stream_log)-[Replaying old messages](/ch12#sec_stream_replay) - relation to consensus, [Shared logs as consensus](/ch10#sec_consistency_shared_logs) - using, [Using shared logs](/ch10#sec_consistency_smr) - shared mode (locks), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - shared-disk architecture, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing), [Distributed Filesystems](/ch11#sec_batch_dfs) - shared-memory architecture, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing) - shared-nothing architecture, [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing), [Glossary](/glossary) - distributed filesystems, [Distributed Filesystems](/ch11#sec_batch_dfs) - (see also distributed filesystems) - use of network, [Unreliable Networks](/ch9#sec_distributed_networks) - sharks - biting undersea cables, [Network Faults in Practice](/ch9#sec_distributed_network_faults) - counting (example), [Query languages for documents](/ch3#query-languages-for-documents) - shredding (deletion) (see crypto-shredding) - shredding (in columnar encoding), [Column-Oriented Storage](/ch4#sec_storage_column) - shredding (in relational model), [When to Use Which Model](/ch3#sec_datamodels_document_summary) - shuffle (batch processing), [Shuffling Data](/ch11#sec_shuffle)-[Shuffling Data](/ch11#sec_shuffle) - siblings (concurrent values), [Manual conflict resolution](/ch6#manual-conflict-resolution), [Capturing the happens-before relationship](/ch6#capturing-the-happens-before-relationship), [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - (see also conflicts) - silo, [Data Warehousing](/ch1#sec_introduction_dwh) - similarity search - edit distance, [Full-Text Search](/ch4#sec_storage_full_text) - genome data, [Summary](/ch3#summary) - simplicity, [Simplicity: Managing Complexity](/ch2#id38) - Singer, [Data Warehousing](/ch1#sec_introduction_dwh) - single-instruction-multi-data (SIMD) instructions, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - single-leader replication (see leader-based replication) - single-threaded execution, [Atomic write operations](/ch8#atomic-write-operations), [Actual Serial Execution](/ch8#sec_transactions_serial) - in stream processing, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Concurrency control](/ch12#sec_stream_concurrency), [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - SingleStore (database) - in-memory storage, [Keeping everything in memory](/ch4#sec_storage_inmemory) - site reliability engineer, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - size-tiered compaction, [Compaction strategies](/ch4#sec_storage_lsm_compaction), [Disk space usage](/ch4#disk-space-usage) - skew, [Glossary](/glossary) - clock skew, [Relying on Synchronized Clocks](/ch9#sec_distributed_clocks_relying)-[Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval), [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - in transaction isolation - read skew, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation), [Summary](/ch8#summary) - write skew, [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts), [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise)-[Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - (see also write skew) - meanings of, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - unbalanced workload, [Sharding of Key-Value Data](/ch7#sec_sharding_key_value) - compensating for, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - due to celebrities, [Skewed Workloads and Relieving Hot Spots](/ch7#sec_sharding_skew) - for time-series data, [Sharding by Key Range](/ch7#sec_sharding_key_range) - skip list, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - SLA (see service level agreements) - Slack (group chat) - GraphQL example, [GraphQL](/ch3#id63) - SlateDB (database), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Setting Up New Followers](/ch6#sec_replication_new_replica) - sliding windows (stream processing), [Types of windows](/ch12#id324) - (see also windows) - sloppy quorums, [Single-Leader Versus Leaderless Replication Performance](/ch6#sec_replication_leaderless_perf) - slowly changing dimension (data warehouses), [Time-dependence of joins](/ch12#sec_stream_join_time) - smearing (leap seconds adjustments), [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - snapshots (databases) - as backups, [Replication](/ch6#ch_replication) - computing derived data, [Creating an index](/ch13#id340) - in change data capture, [Initial snapshot](/ch12#sec_stream_cdc_snapshot) - serializable snapshot isolation (SSI), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - setting up a new replica, [Setting Up New Followers](/ch6#sec_replication_new_replica) - snapshot isolation and repeatable read, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation)-[Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - implementing with MVCC, [Multi-version concurrency control (MVCC)](/ch8#sec_transactions_snapshot_impl) - indexes and MVCC, [Indexes and snapshot isolation](/ch8#indexes-and-snapshot-isolation) - visibility rules, [Visibility rules for observing a consistent snapshot](/ch8#sec_transactions_mvcc_visibility) - synchronized clocks for global snapshots, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - Snowflake (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native), [Layering of cloud services](/ch1#layering-of-cloud-services), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Batch Processing](/ch11#ch_batch) - column-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column) - handling writes, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - sharding and clustering, [Sharding by hash range](/ch7#sharding-by-hash-range) - Snowpark, [Query languages](/ch11#sec_batch_query_lanauges) - Snowflake (ID generator), [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - snowflake schemas, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - SOAP (web services), [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - SOC2 (see Service Organization Control (SOC)) - social graph, [Graph-Like Data Models](/ch3#sec_datamodels_graph) - society - responsibility towards, [Data Systems, Law, and Society](/ch1#sec_introduction_compliance), [Legislation and Self-Regulation](/ch14#sec_future_legislation) - sociotechnical systems, [Humans and Reliability](/ch2#id31) - software as a service (SaaS), [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs), [Cloud Versus Self-Hosting](/ch1#sec_introduction_cloud) - ETL from, [Data Warehousing](/ch1#sec_introduction_dwh) - multitenancy, [Sharding for Multitenancy](/ch7#sec_sharding_multitenancy) - software bugs, [Software faults](/ch2#software-faults) - maintaining integrity, [Maintaining integrity in the face of software bugs](/ch13#id455) - solar storm, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - solid state drives (SSDs) - access patterns, [Sequential versus random writes](/ch4#sidebar_sequential) - compared to object storage, [Setting Up New Followers](/ch6#sec_replication_new_replica) - detecting corruption, [The end-to-end argument](/ch13#sec_future_e2e_argument), [Don't just blindly trust what they promise](/ch13#id364) - failure rate, [Hardware and Software Faults](/ch2#sec_introduction_hardware_faults) - faults in, [Durability](/ch8#durability) - firmware bugs, [Software faults](/ch2#software-faults) - read throughput, [Read performance](/ch4#read-performance) - sequential vs. random writes, [Sequential versus random writes](/ch4#sidebar_sequential) - Solr (search server) - local secondary indexes, [Local Secondary Indexes](/ch7#id166) - request routing, [Request Routing](/ch7#sec_sharding_routing) - use of Lucene, [Full-Text Search](/ch4#sec_storage_full_text) - sort (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Sorting Versus In-memory Aggregation](/ch11#id275), [Distributed Job Orchestration](/ch11#id278) - sort-merge joins (MapReduce), [JOIN and GROUP BY](/ch11#sec_batch_join) - Sorted String Tables (see SSTables) - sorting - sort order in column storage, [Sort Order in Column Storage](/ch4#sort-order-in-column-storage) - source of truth (see systems of record) - Spanner (database) - consistency model, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - data locality, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - in the cloud, [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native) - snapshot isolation using clocks, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - transactions, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - TrueTime API, [Clock readings with a confidence interval](/ch9#clock-readings-with-a-confidence-interval) - Spark (processing framework), [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native), [Batch Processing](/ch11#ch_batch), [Dataflow Engines](/ch11#sec_batch_dataflow) - cost efficiency, [Query languages](/ch11#sec_batch_query_lanauges) - DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes), [DataFrames](/ch11#id287) - fault tolerance, [Handling Faults](/ch11#id281) - for data warehouses, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - high availability using ZooKeeper, [Coordination Services](/ch10#sec_consistency_coordination) - MLlib, [Machine Learning](/ch11#id290) - query optimizer, [Query languages](/ch11#sec_batch_query_lanauges) - shuffling data, [Shuffling Data](/ch11#sec_shuffle) - Spark Streaming, [Stream analytics](/ch12#id318) - microbatching, [Microbatching and checkpointing](/ch12#id329) - streaming SQL support, [Complex event processing](/ch12#id317) - use for ETL, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - SPARQL (query language), [The SPARQL query language](/ch3#the-sparql-query-language) - sparse index, [The SSTable file format](/ch4#the-sstable-file-format) - sparse matrices, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - split brain, [Leader failure: Failover](/ch6#leader-failure-failover), [Request Routing](/ch7#sec_sharding_routing), [Glossary](/glossary) - enforcing constraints, [Uniqueness constraints require consensus](/ch13#id452) - in consensus algorithms, [Consensus](/ch10#sec_consistency_consensus), [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - preventing, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - using fencing tokens to avoid, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens)-[Fencing with multiple replicas](/ch9#fencing-with-multiple-replicas) - spot instances, [Handling Faults](/ch11#id281) - spreadsheets, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - dataflow programming, [Designing Applications Around Dataflow](/ch13#sec_future_dataflow) - pivot table, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - SQL (Structured Query Language), [Simplicity: Managing Complexity](/ch2#id38), [Relational Model versus Document Model](/ch3#sec_datamodels_history), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - for analytics, [Data Warehousing](/ch1#sec_introduction_dwh), [Column-Oriented Storage](/ch4#sec_storage_column) - graph queries in, [Graph Queries in SQL](/ch3#id58) - isolation levels standard, issues with, [Snapshot isolation, repeatable read, and naming confusion](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - joins, [Normalization, Denormalization, and Joins](/ch3#sec_datamodels_normalization) - résumé (example), [The document data model for one-to-many relationships](/ch3#the-document-data-model-for-one-to-many-relationships) - social network home timelines (example), [Representing Users, Posts, and Follows](/ch2#id20) - SQL injection vulnerability, [Byzantine Faults](/ch9#sec_distributed_byzantine) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication) - stored procedures, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - support in batch processing frameworks, [Batch Processing](/ch11#ch_batch) - views, [Datalog: Recursive Relational Queries](/ch3#id62) - SQL Server (database) - archiving WAL to object stores, [Setting Up New Followers](/ch6#sec_replication_new_replica) - change data capture, [Implementing change data capture](/ch12#id307) - data warehousing support, [Data Storage for Analytics](/ch4#sec_storage_analytics) - distributed transaction support, [XA transactions](/ch8#xa-transactions) - leader-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - preventing lost updates, [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - preventing write skew, [Characterizing write skew](/ch8#characterizing-write-skew), [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - read committed isolation, [Implementing read committed](/ch8#sec_transactions_read_committed_impl) - serializable isolation, [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - T-SQL language, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - SQLite (database), [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems), [Compaction strategies](/ch4#sec_storage_lsm_compaction) - archiving WAL to object stores, [Setting Up New Followers](/ch6#sec_replication_new_replica) - SRE (site reliability engineer), [Operations in the Cloud Era](/ch1#sec_introduction_operations) - SSDs (see solid state drives) - SSTables (storage format), [The SSTable file format](/ch4#the-sstable-file-format)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - constructing and maintaining, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - making LSM-Tree from, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - staged rollout (see rolling upgrades) - staleness (old data), [Reading Your Own Writes](/ch6#sec_replication_ryw) - cross-channel timing dependencies, [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - in leaderless databases, [Writing to the Database When a Node Is Down](/ch6#id287) - in multi-version concurrency control, [Detecting stale MVCC reads](/ch8#detecting-stale-mvcc-reads) - monitoring for, [Monitoring staleness](/ch6#monitoring-staleness) - of client state, [Pushing state changes to clients](/ch13#id348) - versus linearizability, [Linearizability](/ch10#sec_consistency_linearizability) - versus timeliness, [Timeliness and Integrity](/ch13#sec_future_integrity) - standbys (see leader-based replication) - star replication topologies, [Multi-leader replication topologies](/ch6#sec_replication_topologies) - star schemas, [Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics)-[Stars and Snowflakes: Schemas for Analytics](/ch3#sec_datamodels_analytics) - Star Wars analogy (event time versus processing time), [Event time versus processing time](/ch12#id322) - starvation (scheduling), [Resource Allocation](/ch11#id279) - state - derived from log of immutable events, [State, Streams, and Immutability](/ch12#sec_stream_immutability) - interplay between state changes and application code, [Dataflow: Interplay between state changes and application code](/ch13#id450) - maintaining derived state, [Maintaining derived state](/ch13#id446) - maintenance by stream processor in stream-stream joins, [Stream-stream join (window join)](/ch12#id440) - observing derived state, [Observing Derived State](/ch13#sec_future_observing)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - rebuilding after stream processor failure, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - separation of application code and, [Separation of application code and state](/ch13#id344) - state machine replication, [Statement-based replication](/ch6#statement-based-replication), [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs), [Using shared logs](/ch10#sec_consistency_smr), [Databases and Streams](/ch12#sec_stream_databases) - event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - stateless systems, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - statically typed languages - analogy to schema-on-write, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - statistical and numerical algorithms, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - StatsD (metrics aggregator), [Direct messaging from producers to consumers](/ch12#id296) - stock market feeds, [Direct messaging from producers to consumers](/ch12#id296) - STONITH (Shoot The Other Node In The Head), [Leader failure: Failover](/ch6#leader-failure-failover) - problems with, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - stop-the-world (see garbage collection) - storage - composing data storage technologies, [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - Storage Area Network (SAN), [Shared-Memory, Shared-Disk, and Shared-Nothing Architecture](/ch2#sec_introduction_shared_nothing), [Distributed Filesystems](/ch11#sec_batch_dfs) - storage engines, [Storage and Retrieval](/ch4#ch_storage)-[Summary](/ch4#summary) - column-oriented, [Column-Oriented Storage](/ch4#sec_storage_column)-[Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - column compression, [Column Compression](/ch4#sec_storage_column_compression)-[Column Compression](/ch4#sec_storage_column_compression) - defined, [Column-Oriented Storage](/ch4#sec_storage_column) - Parquet, [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses), [Column-Oriented Storage](/ch4#sec_storage_column), [Archival storage](/ch5#archival-storage) - sort order in, [Sort Order in Column Storage](/ch4#sort-order-in-column-storage)-[Sort Order in Column Storage](/ch4#sort-order-in-column-storage) - versus wide-column model, [Column Compression](/ch4#sec_storage_column_compression) - writing to, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - in-memory storage, [Keeping everything in memory](/ch4#sec_storage_inmemory) - durability, [Durability](/ch8#durability) - row-oriented, [Storage and Indexing for OLTP](/ch4#sec_storage_oltp)-[Keeping everything in memory](/ch4#sec_storage_inmemory) - B-trees, [B-Trees](/ch4#sec_storage_b_trees)-[B-tree variants](/ch4#b-tree-variants) - comparing B-trees and LSM-trees, [Comparing B-Trees and LSM-Trees](/ch4#sec_storage_btree_lsm_comparison)-[Disk space usage](/ch4#disk-space-usage) - defined, [Column-Oriented Storage](/ch4#sec_storage_column) - log-structured, [Log-Structured Storage](/ch4#sec_storage_log_structured)-[Compaction strategies](/ch4#sec_storage_lsm_compaction) - stored procedures, [Encapsulating transactions in stored procedures](/ch8#encapsulating-transactions-in-stored-procedures)-[Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs), [Glossary](/glossary) - and shared logs, [Using shared logs](/ch10#sec_consistency_smr) - pros and cons of, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - similarity to stream processors, [Application code as a derivation function](/ch13#sec_future_dataflow_derivation) - Storm (stream processor), [Stream analytics](/ch12#id318) - distributed RPC, [Event-Driven Architectures and RPC](/ch12#sec_stream_actors_drpc), [Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - Trident state handling, [Idempotence](/ch12#sec_stream_idempotence) - straggler events, [Handling straggler events](/ch12#id323) - Stream Control Transmission Protocol (SCTP), [The Limitations of TCP](/ch9#sec_distributed_tcp) - stream processing, [Processing Streams](/ch12#sec_stream_processing)-[Summary](/ch12#id332), [Glossary](/glossary) - accessing external services within job, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins), [Microbatching and checkpointing](/ch12#id329), [Idempotence](/ch12#sec_stream_idempotence), [Exactly-once execution of an operation](/ch13#id353) - combining with batch processing, [Unifying batch and stream processing](/ch13#id338) - comparison to batch processing, [Processing Streams](/ch12#sec_stream_processing) - complex event processing (CEP), [Complex event processing](/ch12#id317) - fault tolerance, [Fault Tolerance](/ch12#sec_stream_fault_tolerance)-[Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - atomic commit, [Atomic commit revisited](/ch12#sec_stream_atomic_commit) - idempotence, [Idempotence](/ch12#sec_stream_idempotence) - microbatching and checkpointing, [Microbatching and checkpointing](/ch12#id329) - rebuilding state after a failure, [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - for data integration, [Batch and Stream Processing](/ch13#sec_future_batch_streaming)-[Unifying batch and stream processing](/ch13#id338) - for event sourcing, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - maintaining derived state, [Maintaining derived state](/ch13#id446) - maintenance of materialized views, [Maintaining materialized views](/ch12#sec_stream_mat_view) - messaging systems (see messaging systems) - reasoning about time, [Reasoning About Time](/ch12#sec_stream_time)-[Types of windows](/ch12#id324) - event time versus processing time, [Event time versus processing time](/ch12#id322), [Microbatching and checkpointing](/ch12#id329), [Unifying batch and stream processing](/ch13#id338) - knowing when window is ready, [Handling straggler events](/ch12#id323) - types of windows, [Types of windows](/ch12#id324) - relation to databases (see streams) - relation to services, [Stream processors and services](/ch13#id345) - relationship to batch processing, [Batch Processing](/ch11#ch_batch) - search on streams, [Search on streams](/ch12#id320) - single-threaded execution, [Logs compared to traditional messaging](/ch12#sec_stream_logs_vs_messaging), [Concurrency control](/ch12#sec_stream_concurrency) - stream analytics, [Stream analytics](/ch12#id318) - stream joins, [Stream Joins](/ch12#sec_stream_joins)-[Time-dependence of joins](/ch12#sec_stream_join_time) - stream-stream join, [Stream-stream join (window join)](/ch12#id440) - stream-table join, [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - table-table join, [Table-table join (materialized view maintenance)](/ch12#id326) - time-dependence of, [Time-dependence of joins](/ch12#sec_stream_join_time) - streams, [Stream Processing](/ch12#ch_stream)-[Replaying old messages](/ch12#sec_stream_replay) - end-to-end, pushing events to clients, [End-to-end event streams](/ch13#id349) - messaging systems (see messaging systems) - processing (see stream processing) - relation to databases, [Databases and Streams](/ch12#sec_stream_databases)-[Limitations of immutability](/ch12#sec_stream_immutability_limitations) - (see also changelogs) - API support for change streams, [API support for change streams](/ch12#sec_stream_change_api) - change data capture, [Change Data Capture](/ch12#sec_stream_cdc)-[API support for change streams](/ch12#sec_stream_change_api) - derivative of state by time, [State, Streams, and Immutability](/ch12#sec_stream_immutability) - event sourcing, [Change data capture versus event sourcing](/ch12#sec_stream_event_sourcing) - keeping systems in sync, [Keeping Systems in Sync](/ch12#sec_stream_sync)-[Keeping Systems in Sync](/ch12#sec_stream_sync) - philosophy of immutable events, [State, Streams, and Immutability](/ch12#sec_stream_immutability)-[Limitations of immutability](/ch12#sec_stream_immutability_limitations) - topics, [Transmitting Event Streams](/ch12#sec_stream_transmit) - strict serializability, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - timeliness vs. integrity, [Timeliness and Integrity](/ch13#sec_future_integrity) - striping (in columnar encoding), [Column-Oriented Storage](/ch4#sec_storage_column) - strong consistency (see linearizability) - strong eventual consistency, [Automatic conflict resolution](/ch6#automatic-conflict-resolution) - strong one-copy serializability, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - subjects, predicates, and objects (in triple-stores), [Triple-Stores and SPARQL](/ch3#id59) - subscribers (message streams), [Transmitting Event Streams](/ch12#sec_stream_transmit) - (see also consumers) - supercomputers, [Cloud Computing Versus Supercomputing](/ch1#id17) - Superset (data visualization software), [Analytics](/ch11#sec_batch_olap) - surveillance, [Surveillance](/ch14#id374) - (see also privacy) - sushi principle, [From data warehouse to data lake](/ch1#from-data-warehouse-to-data-lake) - sustainability, [Distributed Versus Single-Node Systems](/ch1#sec_introduction_distributed) - Swagger (service definition format), [Web services](/ch5#sec_web_services) - swapping to disk (see virtual memory) - Swift (programming language) - memory management, [Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - sync engines, [Sync Engines and Local-First Software](/ch6#sec_replication_offline_clients)-[Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - examples of, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - for local-first software, [Real-time collaboration, offline-first, and local-first apps](/ch6#real-time-collaboration-offline-first-and-local-first-apps) - synchronous networks, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks), [Glossary](/glossary) - comparison to asynchronous networks, [Synchronous Versus Asynchronous Networks](/ch9#sec_distributed_sync_networks) - system model, [System Model and Reality](/ch9#sec_distributed_system_model) - synchronous replication, [Synchronous Versus Asynchronous Replication](/ch6#sec_replication_sync_async), [Glossary](/glossary) - with multiple leaders, [Multi-Leader Replication](/ch6#sec_replication_multi_leader) - system administrator, [Operations in the Cloud Era](/ch1#sec_introduction_operations) - system models, [Knowledge, Truth, and Lies](/ch9#sec_distributed_truth), [System Model and Reality](/ch9#sec_distributed_system_model)-[Deterministic simulation testing](/ch9#deterministic-simulation-testing) - assumptions in, [Trust, but Verify](/ch13#sec_future_verification) - correctness of algorithms, [Defining the correctness of an algorithm](/ch9#defining-the-correctness-of-an-algorithm) - mapping to the real world, [Mapping system models to the real world](/ch9#mapping-system-models-to-the-real-world) - safety and liveness, [Safety and liveness](/ch9#sec_distributed_safety_liveness) - systems of record, [Systems of Record and Derived Data](/ch1#sec_introduction_derived), [Glossary](/glossary) - change data capture, [Implementing change data capture](/ch12#id307), [Reasoning about dataflows](/ch13#id443) - event logs, [Event Sourcing and CQRS](/ch3#sec_datamodels_events) - treating event log as, [State, Streams, and Immutability](/ch12#sec_stream_immutability) - systems thinking, [Feedback Loops](/ch14#id372) ### T - t-digest (algorithm), [Use of Response Time Metrics](/ch2#sec_introduction_slo_sla) - table-table joins, [Table-table join (materialized view maintenance)](/ch12#id326) - Tableau (data visualization software), [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp), [Analytics](/ch11#sec_batch_olap) - tail (Unix tool), [Using logs for message storage](/ch12#id300) - tail latency (see latency) - tail vertex (property graphs), [Property Graphs](/ch3#id56) - task (workflows) (see workflow engines) - TCP (Transmission Control Protocol), [The Limitations of TCP](/ch9#sec_distributed_tcp) - comparison to circuit switching, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - comparison to UDP, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - connection failures, [Detecting Faults](/ch9#id307) - flow control, [Network congestion and queueing](/ch9#network-congestion-and-queueing), [Messaging Systems](/ch12#sec_stream_messaging) - packet checksums, [Weak forms of lying](/ch9#weak-forms-of-lying), [The end-to-end argument](/ch13#sec_future_e2e_argument), [Trust, but Verify](/ch13#sec_future_verification) - reliability and duplicate suppression, [Duplicate suppression](/ch13#id354) - retransmission timeouts, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - use for transaction sessions, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object) - Temporal (workflow engine), [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - Tensorflow (machine learning library), [Machine Learning](/ch11#id290) - Teradata (database), [Cloud-Native System Architecture](/ch1#sec_introduction_cloud_native), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - term-partitioned indexes (see global secondary indexes) - termination (consensus), [Single-value consensus](/ch10#single-value-consensus), [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - testing, [Humans and Reliability](/ch2#id31) - thrashing (out of memory), [Process Pauses](/ch9#sec_distributed_clocks_pauses) - threads (concurrency) - actor model, [Distributed actor frameworks](/ch5#distributed-actor-frameworks), [Event-Driven Architectures and RPC](/ch12#sec_stream_actors_drpc) - (see also event-driven architecture) - atomic operations, [Atomicity](/ch8#sec_transactions_acid_atomicity) - background threads, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables) - execution pauses, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable), [Process Pauses](/ch9#sec_distributed_clocks_pauses)-[Process Pauses](/ch9#sec_distributed_clocks_pauses) - memory barriers, [Linearizability and network delays](/ch10#linearizability-and-network-delays) - preemption, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - single (see single-threaded execution) - three-phase commit, [Three-phase commit](/ch8#three-phase-commit) - three-way relationships, [Property Graphs](/ch3#id56) - Thrift (data format), [Protocol Buffers](/ch5#sec_encoding_protobuf) - throughput, [Describing Performance](/ch2#sec_introduction_percentiles), [Describing Load](/ch2#id33), [Batch Processing](/ch11#ch_batch) - TIBCO, [Message brokers](/ch5#message-brokers) - Enterprise Message Service, [Message brokers compared to databases](/ch12#id297) - StreamBase (stream analytics), [Complex event processing](/ch12#id317) - TiDB (database) - consensus-based replication, [Single-Leader Replication](/ch6#sec_replication_leader) - regions (sharding), [Sharding](/ch7#ch_sharding) - request routing, [Request Routing](/ch7#sec_sharding_routing) - serving derived data, [Serving Derived Data](/ch11#sec_batch_serving_derived) - sharded secondary indexes, [Global Secondary Indexes](/ch7#id167) - snapshot isolation support, [Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - timestamp oracle, [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - transactions, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - use of model-checking, [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - tiered storage, [Setting Up New Followers](/ch6#sec_replication_new_replica), [Disk space usage](/ch12#sec_stream_disk_usage) - TigerBeetle (database), [Summary](/ch3#summary) - deterministic simulation testing, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - TigerGraph (database) - GSQL language, [Graph Queries in SQL](/ch3#id58) - Tigris (object storage), [Distributed Filesystems](/ch11#sec_batch_dfs) - TileDB (database), [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - time - concurrency and, [The "happens-before" relation and concurrency](/ch6#sec_replication_happens_before) - cross-channel timing dependencies, [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - in distributed systems, [Unreliable Clocks](/ch9#sec_distributed_clocks)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - (see also clocks) - clock synchronization and accuracy, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - relying on synchronized clocks, [Relying on Synchronized Clocks](/ch9#sec_distributed_clocks_relying)-[Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - process pauses, [Process Pauses](/ch9#sec_distributed_clocks_pauses)-[Limiting the impact of garbage collection](/ch9#sec_distributed_gc_impact) - reasoning about, in stream processors, [Reasoning About Time](/ch12#sec_stream_time)-[Types of windows](/ch12#id324) - event time versus processing time, [Event time versus processing time](/ch12#id322), [Microbatching and checkpointing](/ch12#id329), [Unifying batch and stream processing](/ch13#id338) - knowing when window is ready, [Handling straggler events](/ch12#id323) - timestamp of events, [Whose clock are you using, anyway?](/ch12#id438) - types of windows, [Types of windows](/ch12#id324) - system models for distributed systems, [System Model and Reality](/ch9#sec_distributed_system_model) - time-dependence in stream joins, [Time-dependence of joins](/ch12#sec_stream_join_time) - time series data - as DataFrames, [DataFrames, Matrices, and Arrays](/ch3#sec_datamodels_dataframes) - column-oriented storage, [Column-Oriented Storage](/ch4#sec_storage_column) - time-of-day clocks, [Time-of-day clocks](/ch9#time-of-day-clocks) - hybrid logical clocks, [Hybrid logical clocks](/ch10#hybrid-logical-clocks) - timeliness, [Timeliness and Integrity](/ch13#sec_future_integrity) - coordination-avoiding data systems, [Coordination-avoiding data systems](/ch13#id454) - correctness of dataflow systems, [Correctness of dataflow systems](/ch13#id453) - timeouts, [Unreliable Networks](/ch9#sec_distributed_networks), [Glossary](/glossary) - dynamic configuration of, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - for failover, [Leader failure: Failover](/ch6#leader-failure-failover) - length of, [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - TimescaleDB (database), [Column-Oriented Storage](/ch4#sec_storage_column) - timestamps, [Logical Clocks](/ch10#sec_consistency_timestamps) - assigning to events in stream processing, [Whose clock are you using, anyway?](/ch12#id438) - for read-after-write consistency, [Reading Your Own Writes](/ch6#sec_replication_ryw) - for transaction ordering, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) - insufficiency for enforcing constraints, [Enforcing constraints using logical clocks](/ch10#enforcing-constraints-using-logical-clocks) - key range sharding by, [Sharding by Key Range](/ch7#sec_sharding_key_range) - Lamport, [Lamport timestamps](/ch10#lamport-timestamps) - logical, [Ordering events to capture causality](/ch13#sec_future_capture_causality) - ordering events, [Timestamps for ordering events](/ch9#sec_distributed_lww) - timestamp oracle, [Implementing a linearizable ID generator](/ch10#implementing-a-linearizable-id-generator) - TLA+ (specification language), [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - token bucket (limiting retries), [Describing Performance](/ch2#sec_introduction_percentiles) - tombstones, [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Disk space usage](/ch4#disk-space-usage), [Log compaction](/ch12#sec_stream_log_compaction) - topics (messaging), [Message brokers](/ch5#message-brokers), [Transmitting Event Streams](/ch12#sec_stream_transmit) - torn pages (B-trees), [Making B-trees reliable](/ch4#sec_storage_btree_wal) - total order, [Glossary](/glossary) - broadcast (see shared logs) - limits of, [The limits of total ordering](/ch13#id335) - on logical timestamps, [Logical Clocks](/ch10#sec_consistency_timestamps) - tracing, [Problems with Distributed Systems](/ch1#sec_introduction_dist_sys_problems) - tracking behavioral data, [Privacy and Tracking](/ch14#id373) - (see also privacy) - trade-offs, [Trade-offs in Data Systems Architecture](/ch1#ch_tradeoffs)-[Data Systems, Law, and Society](/ch1#sec_introduction_compliance) - transaction coordinator (see coordinator) - transaction manager (see coordinator) - transaction processing, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp)-[Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - comparison to analytics, [Characterizing Transaction Processing and Analytics](/ch1#sec_introduction_oltp) - comparison to data warehousing, [Data Storage for Analytics](/ch4#sec_storage_analytics) - transactions, [Transactions](/ch8#ch_transactions)-[Summary](/ch8#summary), [Glossary](/glossary) - ACID properties of, [The Meaning of ACID](/ch8#sec_transactions_acid) - atomicity, [Atomicity](/ch8#sec_transactions_acid_atomicity) - consistency, [Consistency](/ch8#sec_transactions_acid_consistency) - durability, [Making B-trees reliable](/ch4#sec_storage_btree_wal), [Durability](/ch8#durability) - isolation, [Isolation](/ch8#sec_transactions_acid_isolation) - and derived data integrity, [Timeliness and Integrity](/ch13#sec_future_integrity) - and replication, [Solutions for Replication Lag](/ch6#id131) - compensating (see compensating transactions) - concept of, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview) - distributed transactions, [Distributed Transactions](/ch8#sec_transactions_distributed)-[Exactly-once message processing revisited](/ch8#exactly-once-message-processing-revisited) - avoiding, [Derived data versus distributed transactions](/ch13#sec_future_derived_vs_transactions), [Making unbundling work](/ch13#sec_future_unbundling_favor), [Enforcing Constraints](/ch13#sec_future_constraints)-[Coordination-avoiding data systems](/ch13#id454) - failure amplification, [Maintaining derived state](/ch13#id446) - for sharded systems, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - in doubt/uncertain status, [Coordinator failure](/ch8#coordinator-failure), [Holding locks while in doubt](/ch8#holding-locks-while-in-doubt) - two-phase commit, [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc)-[Three-phase commit](/ch8#three-phase-commit) - use of, [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa)-[Exactly-once message processing](/ch8#sec_transactions_exactly_once) - XA transactions, [XA transactions](/ch8#xa-transactions)-[Problems with XA transactions](/ch8#problems-with-xa-transactions) - OLTP versus analytics queries, [Analytics](/ch11#sec_batch_olap) - purpose of, [Transactions](/ch8#ch_transactions) - serializability, [Serializability](/ch8#sec_transactions_serializability)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - actual serial execution, [Actual Serial Execution](/ch8#sec_transactions_serial)-[Summary of serial execution](/ch8#summary-of-serial-execution) - pessimistic versus optimistic concurrency control, [Pessimistic versus optimistic concurrency control](/ch8#pessimistic-versus-optimistic-concurrency-control) - serializable snapshot isolation (SSI), [Serializable Snapshot Isolation (SSI)](/ch8#sec_transactions_ssi)-[Performance of serializable snapshot isolation](/ch8#performance-of-serializable-snapshot-isolation) - two-phase locking (2PL), [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl)-[Index-range locks](/ch8#sec_transactions_2pl_range) - single-object and multi-object, [Single-Object and Multi-Object Operations](/ch8#sec_transactions_multi_object)-[Handling errors and aborts](/ch8#handling-errors-and-aborts) - handling errors and aborts, [Handling errors and aborts](/ch8#handling-errors-and-aborts) - need for multi-object transactions, [The need for multi-object transactions](/ch8#sec_transactions_need) - single-object writes, [Single-object writes](/ch8#sec_transactions_single_object) - snapshot isolation (see snapshots) - strict serializability, [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition) - weak isolation levels, [Weak Isolation Levels](/ch8#sec_transactions_isolation_levels)-[Materializing conflicts](/ch8#materializing-conflicts) - preventing lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update)-[Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - read committed, [Read Committed](/ch8#sec_transactions_read_committed)-[Snapshot Isolation and Repeatable Read](/ch8#sec_transactions_snapshot_isolation) - traversal (graphs), [Property Graphs](/ch3#id56) - trie (data structure), [Constructing and merging SSTables](/ch4#constructing-and-merging-sstables), [Full-Text Search](/ch4#sec_storage_full_text) - as SSTable index, [The SSTable file format](/ch4#the-sstable-file-format) - triggers (databases), [Transmitting Event Streams](/ch12#sec_stream_transmit) - Trino (data warehouse), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - federated databases, [The meta-database of everything](/ch13#id341) - query optimizer, [Query languages](/ch11#sec_batch_query_lanauges) - use for ETL, [Extract--Transform--Load (ETL)](/ch11#sec_batch_etl_usage) - workflow example, [Scheduling Workflows](/ch11#sec_batch_workflows) - triple-stores, [Triple-Stores and SPARQL](/ch3#id59)-[The SPARQL query language](/ch3#the-sparql-query-language) - SPARQL query language, [The SPARQL query language](/ch3#the-sparql-query-language) - tumbling windows (stream processing), [Types of windows](/ch12#id324) - (see also windows) - in microbatching, [Microbatching and checkpointing](/ch12#id329) - Turbopuffer (vector search), [Setting Up New Followers](/ch6#sec_replication_new_replica) - Turtle (RDF data format), [Triple-Stores and SPARQL](/ch3#id59) - Twitter (see X (social network)) - two-phase commit (2PC), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc)-[Coordinator failure](/ch8#coordinator-failure), [Glossary](/glossary) - confusion with two-phase locking, [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl) - coordinator failure, [Coordinator failure](/ch8#coordinator-failure) - coordinator recovery, [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - how it works, [A system of promises](/ch8#a-system-of-promises) - performance cost, [Distributed Transactions Across Different Systems](/ch8#sec_transactions_xa) - problems with XA transactions, [Problems with XA transactions](/ch8#problems-with-xa-transactions) - transactions holding locks, [Holding locks while in doubt](/ch8#holding-locks-while-in-doubt) - two-phase locking (2PL), [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl)-[Index-range locks](/ch8#sec_transactions_2pl_range), [What Makes a System Linearizable?](/ch10#sec_consistency_lin_definition), [Glossary](/glossary) - confusion with two-phase commit, [Two-Phase Locking (2PL)](/ch8#sec_transactions_2pl) - growing and shrinking phases, [Implementation of two-phase locking](/ch8#implementation-of-two-phase-locking) - index-range locks, [Index-range locks](/ch8#sec_transactions_2pl_range) - performance of, [Performance of two-phase locking](/ch8#performance-of-two-phase-locking) - type checking, dynamic versus static, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) ### U - UDP (User Datagram Protocol) - comparison to TCP, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - multicast, [Direct messaging from producers to consumers](/ch12#id296) - Ultima Online (game), [Sharding](/ch7#ch_sharding) - unbounded datasets, [Stream Processing](/ch12#ch_stream), [Glossary](/glossary) - (see also streams) - unbounded delays, [Glossary](/glossary) - in networks, [Timeouts and Unbounded Delays](/ch9#sec_distributed_queueing) - process pauses, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - unbundling databases, [Unbundling Databases](/ch13#sec_future_unbundling)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - composing data storage technologies, [Composing Data Storage Technologies](/ch13#id447)-[Unbundled versus integrated systems](/ch13#id448) - federation versus unbundling, [The meta-database of everything](/ch13#id341) - designing applications around dataflow, [Designing Applications Around Dataflow](/ch13#sec_future_dataflow)-[Stream processors and services](/ch13#id345) - observing derived state, [Observing Derived State](/ch13#sec_future_observing)-[Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - materialized views and caching, [Materialized views and caching](/ch13#id451) - multi-shard data processing, [Multi-shard data processing](/ch13#sec_future_unbundled_multi_shard) - pushing state changes to clients, [Pushing state changes to clients](/ch13#id348) - uncertain (transaction status) (see in doubt) - union type (in Avro), [Schema evolution rules](/ch5#schema-evolution-rules) - uniq (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Simple Log Analysis](/ch11#sec_batch_log_analysis), [Distributed Job Orchestration](/ch11#id278) - uniqueness constraints - asynchronously checked, [Loosely interpreted constraints](/ch13#id362) - requiring consensus, [Uniqueness constraints require consensus](/ch13#id452) - requiring linearizability, [Constraints and uniqueness guarantees](/ch10#sec_consistency_uniqueness) - uniqueness in log-based messaging, [Uniqueness in log-based messaging](/ch13#sec_future_uniqueness_log) - Unity (data catalog), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - universally unique identifiers (see UUIDs) - Unix philosophy - comparison to relational databases, [Unbundling Databases](/ch13#sec_future_unbundling), [The meta-database of everything](/ch13#id341) - comparison to stream processing, [Processing Streams](/ch12#sec_stream_processing) - Unix pipes, [Simple Log Analysis](/ch11#sec_batch_log_analysis) - compared to distributed batch processing, [Scheduling Workflows](/ch11#sec_batch_workflows) - UPDATE statement (SQL), [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - updates - preventing lost updates, [Preventing Lost Updates](/ch8#sec_transactions_lost_update)-[Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - atomic write operations, [Atomic write operations](/ch8#atomic-write-operations) - automatically detecting lost updates, [Automatically detecting lost updates](/ch8#automatically-detecting-lost-updates) - compare-and-set (CAS), [Conditional writes (compare-and-set)](/ch8#sec_transactions_compare_and_set) - conflict resolution and replication, [Conflict resolution and replication](/ch8#conflict-resolution-and-replication) - using explicit locking, [Explicit locking](/ch8#explicit-locking) - preventing write skew, [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts) - utilization - batch process scheduling, [Resource Allocation](/ch11#id279) - increasing through preemption, [Handling Faults](/ch11#id281) - trade-off with latency, [Can we not simply make network delays predictable?](/ch9#can-we-not-simply-make-network-delays-predictable) - uTP protocol (BitTorrent), [The Limitations of TCP](/ch9#sec_distributed_tcp) - UUIDs, [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) ### V - validity (consensus), [Single-value consensus](/ch10#single-value-consensus), [Atomic commitment as consensus](/ch10#atomic-commitment-as-consensus) - vBuckets (sharding), [Sharding](/ch7#ch_sharding) - vector clocks, [Version vectors](/ch6#version-vectors) - (see also version vectors) - and Lamport/hybrid logical clocks, [Lamport/hybrid logical clocks versus vector clocks](/ch10#lamporthybrid-logical-clocks-vs-vector-clocks) - and version vectors, [Version vectors](/ch6#version-vectors) - vector embedding, [Vector Embeddings](/ch4#id92) - vectorized processing, [Query Execution: Compilation and Vectorization](/ch4#sec_storage_vectorized) - vendor lock-in, [Pros and Cons of Cloud Services](/ch1#sec_introduction_cloud_tradeoffs) - Venice (database), [Serving Derived Data](/ch11#sec_batch_serving_derived) - verification, [Trust, but Verify](/ch13#sec_future_verification)-[Tools for auditable data systems](/ch13#id366) - avoiding blind trust, [Don't just blindly trust what they promise](/ch13#id364) - designing for auditability, [Designing for auditability](/ch13#id365) - end-to-end integrity checks, [The end-to-end argument again](/ch13#id456) - tools for auditable data systems, [Tools for auditable data systems](/ch13#id366) - version control systems - merge conflicts, [Manual conflict resolution](/ch6#manual-conflict-resolution) - reliance on immutable data, [Concurrency control](/ch12#sec_stream_concurrency) - version vectors, [Problems with different topologies](/ch6#problems-with-different-topologies), [Version vectors](/ch6#version-vectors) - dotted, [Version vectors](/ch6#version-vectors) - versus vector clocks, [Version vectors](/ch6#version-vectors) - Vertica (database), [Cloud Data Warehouses](/ch4#sec_cloud_data_warehouses) - handling writes, [Writing to Column-Oriented Storage](/ch4#writing-to-column-oriented-storage) - vertical scaling (see scaling up) - vertices (in graphs), [Graph-Like Data Models](/ch3#sec_datamodels_graph) - property graph model, [Property Graphs](/ch3#id56) - video games, [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - video transcoding (example), [Cross-channel timing dependencies](/ch10#cross-channel-timing-dependencies) - views (SQL queries), [Datalog: Recursive Relational Queries](/ch3#id62) - materialized views (see materialization) - Viewstamped Replication (consensus algorithm), [Consensus](/ch10#sec_consistency_consensus), [Consensus in Practice](/ch10#sec_consistency_total_order) - use of model-checking, [Model checking and specification languages](/ch9#model-checking-and-specification-languages) - view number, [From single-leader replication to consensus](/ch10#from-single-leader-replication-to-consensus) - virtual block device, [Separation of storage and compute](/ch1#sec_introduction_storage_compute) - virtual file system, [Distributed Filesystems](/ch11#sec_batch_dfs) - comparison to distributed filesystems, [Distributed Filesystems](/ch11#sec_batch_dfs) - virtual machines, [Layering of cloud services](/ch1#layering-of-cloud-services) - context switches, [Process Pauses](/ch9#sec_distributed_clocks_pauses) - network performance, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - noisy neighbors, [Network congestion and queueing](/ch9#network-congestion-and-queueing) - virtualized clocks in, [Clock Synchronization and Accuracy](/ch9#sec_distributed_clock_accuracy) - virtual memory - process pauses due to page faults, [Latency and Response Time](/ch2#id23), [Process Pauses](/ch9#sec_distributed_clocks_pauses) - Virtuoso (database), [The SPARQL query language](/ch3#the-sparql-query-language) - VisiCalc (spreadsheets), [Designing Applications Around Dataflow](/ch13#sec_future_dataflow) - Vitess (database) - key-range sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - vnodes (sharding), [Sharding](/ch7#ch_sharding) - vocabularies, [Triple-Stores and SPARQL](/ch3#id59) - Voice over IP (VoIP), [Network congestion and queueing](/ch9#network-congestion-and-queueing) - VoltDB (database) - cross-shard serializability, [Sharding](/ch8#sharding) - deterministic stored procedures, [Pros and cons of stored procedures](/ch8#sec_transactions_stored_proc_tradeoffs) - in-memory storage, [Keeping everything in memory](/ch4#sec_storage_inmemory) - process-per-core model, [Pros and Cons of Sharding](/ch7#sec_sharding_reasons) - secondary indexes, [Local Secondary Indexes](/ch7#id166) - serial execution of transactions, [Actual Serial Execution](/ch8#sec_transactions_serial) - statement-based replication, [Statement-based replication](/ch6#statement-based-replication), [Rebuilding state after a failure](/ch12#sec_stream_state_fault_tolerance) - transactions in stream processing, [Atomic commit revisited](/ch12#sec_stream_atomic_commit) ### W - WAL (write-ahead log), [Making B-trees reliable](/ch4#sec_storage_btree_wal) - WAL-G (backup tool), [Setting Up New Followers](/ch6#sec_replication_new_replica) - WarpStream (messaging), [Disk space usage](/ch12#sec_stream_disk_usage) - web services (see services) - webhooks, [Direct messaging from producers to consumers](/ch12#id296) - webMethods (messaging), [Message brokers](/ch5#message-brokers) - WebSocket (protocol), [Pushing state changes to clients](/ch13#id348) - wide-column data model, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - versus column-oriented storage, [Column Compression](/ch4#sec_storage_column_compression) - windows (stream processing), [Stream analytics](/ch12#id318), [Reasoning About Time](/ch12#sec_stream_time)-[Types of windows](/ch12#id324) - infinite windows for changelogs, [Maintaining materialized views](/ch12#sec_stream_mat_view), [Stream-table join (stream enrichment)](/ch12#sec_stream_table_joins) - knowing when all events have arrived, [Handling straggler events](/ch12#id323) - stream joins within a window, [Stream-stream join (window join)](/ch12#id440) - types of windows, [Types of windows](/ch12#id324) - WITH RECURSIVE syntax (SQL), [Graph Queries in SQL](/ch3#id58) - Word2Vec (language model), [Vector Embeddings](/ch4#id92) - workflow engines, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - Airflow (see Airflow (workflow scheduler)) - batch processing, [Scheduling Workflows](/ch11#sec_batch_workflows) - Camunda (see Camunda (workflow engine)) - Dagster (see Dagster (workflow scheduler)) - durable execution, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - ETL (see ETL (extract-transform-load)) - executor, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows) - orchestrators, [Durable Execution and Workflows](/ch5#sec_encoding_dataflow_workflows), [Batch Processing](/ch11#ch_batch) - Orkes (see Orkes (workflow engine)) - Prefect (see Prefect (workflow scheduler)) - reliance on determinism, [Deterministic simulation testing](/ch9#deterministic-simulation-testing) - Restate (see Restate (workflow engine)) - Temporal (see Temporal (workflow engine)) - working set, [Sorting Versus In-memory Aggregation](/ch11#id275) - write amplification, [Write amplification](/ch4#write-amplification) - write path (derived data), [Observing Derived State](/ch13#sec_future_observing) - write skew (transaction isolation), [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Materializing conflicts](/ch8#materializing-conflicts) - characterizing, [Write Skew and Phantoms](/ch8#sec_transactions_write_skew)-[Phantoms causing write skew](/ch8#sec_transactions_phantom), [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise) - examples of, [Write Skew and Phantoms](/ch8#sec_transactions_write_skew), [More examples of write skew](/ch8#more-examples-of-write-skew) - materializing conflicts, [Materializing conflicts](/ch8#materializing-conflicts) - occurrence in practice, [Maintaining integrity in the face of software bugs](/ch13#id455) - phantoms, [Phantoms causing write skew](/ch8#sec_transactions_phantom) - preventing - in snapshot isolation, [Decisions based on an outdated premise](/ch8#decisions-based-on-an-outdated-premise)-[Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - in two-phase locking, [Predicate locks](/ch8#predicate-locks)-[Index-range locks](/ch8#sec_transactions_2pl_range) - options for, [Characterizing write skew](/ch8#characterizing-write-skew) - write-ahead log (WAL), [Making B-trees reliable](/ch4#sec_storage_btree_wal), [Write-ahead log (WAL) shipping](/ch6#write-ahead-log-wal-shipping) - in durable execution, [Durable execution](/ch5#durable-execution) - writes (database) - atomic write operations, [Atomic write operations](/ch8#atomic-write-operations) - detecting writes affecting prior reads, [Detecting writes that affect prior reads](/ch8#sec_detecting_writes_affect_reads) - preventing dirty writes with read committed, [No dirty writes](/ch8#sec_transactions_dirty_write) - WS-\* framework, [The problems with remote procedure calls (RPCs)](/ch5#sec_problems_with_rpc) - WS-AtomicTransaction (2PC), [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc) ### X - X (social network) - constructing home timelines (example), [Case Study: Social Network Home Timelines](/ch2#sec_introduction_twitter), [Deriving several views from the same event log](/ch12#sec_stream_deriving_views), [Table-table join (materialized view maintenance)](/ch12#id326), [Materialized views and caching](/ch13#id451) - cost of joins, [Denormalization in the social networking case study](/ch3#denormalization-in-the-social-networking-case-study) - describing load, [Describing Load](/ch2#id33) - fault tolerance, [Fault Tolerance](/ch2#id27) - performance metrics, [Describing Performance](/ch2#sec_introduction_percentiles) - DistributedLog (event log), [Using logs for message storage](/ch12#id300) - Snowflake (ID generator), [ID Generators and Logical Clocks](/ch10#sec_consistency_logical) - XA transactions, [Two-Phase Commit (2PC)](/ch8#sec_transactions_2pc), [XA transactions](/ch8#xa-transactions)-[Problems with XA transactions](/ch8#problems-with-xa-transactions) - heuristic decisions, [Recovering from coordinator failure](/ch8#recovering-from-coordinator-failure) - problems with, [Problems with XA transactions](/ch8#problems-with-xa-transactions) - xargs (Unix tool), [Simple Log Analysis](/ch11#sec_batch_log_analysis) - XFS (file system), [Distributed Filesystems](/ch11#sec_batch_dfs) - XGBoost (machine learning library), [Machine Learning](/ch11#id290) - XML - binary variants, [Binary encoding](/ch5#binary-encoding) - data locality, [Data locality for reads and writes](/ch3#sec_datamodels_document_locality) - encoding RDF data, [The RDF data model](/ch3#the-rdf-data-model) - for application data, issues with, [JSON, XML, and Binary Variants](/ch5#sec_encoding_json) - in relational databases, [Schema flexibility in the document model](/ch3#sec_datamodels_schema_flexibility) - XML databases, [Relational Model versus Document Model](/ch3#sec_datamodels_history), [Query languages for documents](/ch3#query-languages-for-documents) - Xorq (query engine), [The meta-database of everything](/ch13#id341) - XPath, [Query languages for documents](/ch3#query-languages-for-documents) - XQuery, [Query languages for documents](/ch3#query-languages-for-documents) ### Y - Yahoo - response time study, [Average, Median, and Percentiles](/ch2#id24) - YARN (job scheduler), [Distributed Job Orchestration](/ch11#id278), [Separation of application code and state](/ch13#id344) - ApplicationMaster, [Distributed Job Orchestration](/ch11#id278) - Yjs (CRDT library), [Pros and cons of sync engines](/ch6#pros-and-cons-of-sync-engines) - YugabyteDB (database) - hash-range sharding, [Sharding by hash range](/ch7#sharding-by-hash-range) - key-range sharding, [Sharding by Key Range](/ch7#sec_sharding_key_range) - multi-leader replication, [Geographically Distributed Operation](/ch6#sec_replication_multi_dc) - request routing, [Request Routing](/ch7#sec_sharding_routing) - sharded secondary indexes, [Global Secondary Indexes](/ch7#id167) - tablets (sharding), [Sharding](/ch7#ch_sharding) - transactions, [What Exactly Is a Transaction?](/ch8#sec_transactions_overview), [Database-internal Distributed Transactions](/ch8#sec_transactions_internal) - use of clock synchronization, [Synchronized clocks for global snapshots](/ch9#sec_distributed_spanner) ### Z - Zab (consensus algorithm), [Consensus](/ch10#sec_consistency_consensus), [Consensus in Practice](/ch10#sec_consistency_total_order) - use in ZooKeeper, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - zero-copy, [Formats for Encoding Data](/ch5#sec_encoding_formats) - zero-disk architecture (ZDA), [Setting Up New Followers](/ch6#sec_replication_new_replica) - ZeroMQ (messaging library), [Direct messaging from producers to consumers](/ch12#id296) - zombies (split brain), [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens) - zones (cloud computing) (see availability zones) - ZooKeeper (coordination service), [Coordination Services](/ch10#sec_consistency_coordination)-[Service discovery](/ch10#service-discovery) - generating fencing tokens, [Fencing off zombies and delayed requests](/ch9#sec_distributed_fencing_tokens), [Using shared logs](/ch10#sec_consistency_smr), [Coordination Services](/ch10#sec_consistency_coordination) - linearizable operations, [Implementing Linearizable Systems](/ch10#sec_consistency_implementing_linearizable) - locks and leader election, [Locking and leader election](/ch10#locking-and-leader-election) - observers, [Service discovery](/ch10#service-discovery) - use for service discovery, [Load balancers, service discovery, and service meshes](/ch5#sec_encoding_service_discovery), [Service discovery](/ch10#service-discovery) - use for shard assignment, [Request Routing](/ch7#sec_sharding_routing) - use of Zab algorithm, [Consensus](/ch10#sec_consistency_consensus)