Title: 544
Tags: lec29

Q: When multiple nodes in a system can be affected by the same fault, we say they are in the same _________
A: failure domain
these nodes are vulnerable to "correlated failure"

Q: What are some examples of "faults", and what does fault tolerance mean?
A: Few examples of faults: disk failure, network down, node dies, rack loses power, data center is destroyed.
Fault tolerance: the system can keep "working" even when certain faults occur.
Fault tolerance can take many forms, because different systems can handle different kinds of faults.  For example, HDFS can handle a DataNode failure, but it is not geographically distributed, so it couldn't handle loss of a whole data center.  Cassandra can be geographically distributed, so it could potentially.  Fault tolerance also varies based on what "working" mean -- for example, some systems might insure durability (your data is not lost), but not availability (so the system might be offline for a while).

Q: Given a token T for a row that is replicated N times, how do you "walk the ring" to find the N nodes necessary?
A: Start on T, and keep increasing it, recording every vnode you match or pass in a replica set.  Stop when the replica set contains N nodes.
Special cases:
1. if you get to the biggest possible token number, wrap around to the smallest
2. if you find a vnode that is in the same failure domain as another vnode already found, potentially skip it.  This depends on the policy, which is pluggable.  The simplest policy will just skip vnodes on the same physical node, but more complex policies might also skip nodes that are in the same rack or datacenter.

Q: What does "ack" stand for, and what does it mean in the context of writing data?
A: "ack" is short for "acknowledgement", and receiving an acknowledgement back from a system means our data is "committed" (in other words, "safe" even if certain bad things happen).
Different systems guarantee different levels of safety for committed data.  For example, for one system it might mean the data is in RAM on two workers.  On a system with a stronger guarantee, in contrast, it might mean the data is on disk on three workers.

Q: If you receive ACK, what does it mean?  If you don't receive an ACK, what does it mean?
A: ACK: the data is definitely committed
No ACK: the data MAY or MAY NOT be committed (perhaps the write works fine, but an issue arose with sending the message back to report that)

Q: Cassandra has RF, W, and R options.  How are these used when reading and writing data?
A: Writing: a coordinator will attempt to write RF copies of the data.  It will report success (ack) if at least W nodes successfully write it.
Reading: a coordinator will attempt to read the data from R nodes.  1 of these will be a full data read, and (R-1) workers will just send back a checksum of the data.

Q: In terms of RF, W, and R: what should you do if you want a stronger durability guarantee when you get an ack?
A: Increase W (RF must be >= W, so RF might need to increase too).

Q: In terms of RF, W, and R: what should you do if you want better write availability?
A: Decrease W -- then our write can be considered successful even if fewer nodes are currently working.

Q: In terms of RF, W, and R: what should you do if you want better read availability?
A: Decrease R: then we can read the data even if fewer nodes are currently working.

Q: In terms of RF, W, and R: what should you do if you want to guarantee that reads will see the last written data?
A: Make sure W+R > RF.
This guarantees the write and read quorums will overlap.  For example, if you have 5 nodes, and two subsets, of size 2 and 4, those two subsets are mathematically guaranteed to have some overlap.

Q: What does eventually consistent mean?
A: It means different versions of the data on different workers can temporarily "diverge" -- this means that neither version 1 or version 2 of a row is strictly "newest".
Perhaps version 1 has the latest data in column A, but version 2 has the latest data in column B.  Eventually, though, the rows should sync up.

Q: In eventually consistent systems, sometimes you do a read, and you get conflicting versions of the data back.  Neither version can be considered strictly better.  What are some approaches to handling this?  What does Cassandra do?
A: Approach 1: Pass all the different versions back to the client doing the reads -- let the client sort it out.
Approach 2: Automatically merge all the versions to get a new version.
Cassandra automatically merges.  It keeps timestamps on every cell, and uses a LWW policy (last writer wins), meaning we compare the versions cell by cell, picking the data with the freshest timestamp in each case.