Title: 544
Tags: lec32

Q: When is a Kafka message considered "committed"?
A: When it is written to ALL in-sync replicas.

Q: In Kafka, both consumers and follower replicas fetch data from the leader replica.  But how do the fetches differ?
A: Consumers fetch only "committed" messages.
Followers fetch both committed and uncomitted messages.
This is because we only want consumers to see committed messages.  But the way messages become committed is by being received by more replicas, so the follower replicas need to fetch uncommitted data.

Q: If min.insync.replicas is X and the number of currently in-sync replicas is Y, why doesn't Kafka consider a message committed as soon as it is written to at least X replicas?  (assume X<Y in this question)
A: Hypothetical scenario showing why we can't do this.
Let's say X=2 and Y=3.
1. write occurs on two replicas (one "in-sync" replica doesn't have the message)
2. leader returns an "ack" indicating message is committed at offset 100
3. leader dies
4. Kafka selects a new leader from those that are in-sync -- let's say it selects the one that didn't receive the message
5. the new leader is the authority on what messages are at each offset, and might write a different messages at offset 100.  This will mean a previously committed message will have changed
That's why a Kafka leader WILL NOT return an ack in step 2.  It will wait until the message is on the third replica too.

Q: In order to be considered "in sync", does a follower replica in Kafka need to have ALL the messages the leader has seen?
A: No, it is allowed to lag a bit.  This is configurable -- for example, you might define "in sync" to mean the follower has retrieved a batch reading the end of the topic partition in the last N seconds.

Q: How do "elections" in distributed systems usually differ from a regular democratic election humans might participate in?
A: In a human election, we usually have a couple goals:
1. represent the will of the people
2. have a well defined process so it is clear who the leader is (it is problematic if there is disagreement about who the leader is)
In distributed systems, we usually just care about (2).  So in Kafka, for example, "election" of a replica leader is really just a special "controller" broker making a decision without input from other brokers.

Q: What is an example of a "write anomoly" in Kafka?
A: A message was acknowledged as successfully written, but it was not actually readable later.

Q: What is an example of a "read anomoly" in Kafka?
A: A message was read, but when you try to read it again later, it has dissapeared.

Q: Producers can be created like this: KafkaProducer(..., acks=????)
What are the options, and what do they each do?
A: 0: don't wait for the leader to send an ack
1: ack means leader has written message to its own log
"all": message has been written to all in-sync replicas

Q: How might we end up with "at-least-once" semantics?
A: This happens when a sender keeps retrying until it gets an acknowledgement of success.  If the ack is received on the Nth time, then for the first (N-1) times, the message may or may not have gone through.  This means the message will have been sent between 1 and N times.

Q: How might you end up with "at-most-once" semantics?
A: You try to send a message, and you never retry sending, even if it isn't acknowledged.  If not acknowledged, there's still a chance the message made it through, and the failure happened before the ack could be sent.  So the messages will have been received either 0 or 1 times.

Q: What does it mean for an operation to be idempotent?
A: the effect of performing the operation multiple times is equivalent to the effect of performing the operation a single time

Q: How might we make an operation idempotent, even if it not naturally so?
A: Associate the operation with a unique ID, and whatever system is receiving the operation requests is responsible for remembering previously seen IDs and supressing/ignoring duplicates.

Q: What is the problem with "auto commit" for Kafka consumers?
A: Auto commit happens on some timer, and it's possible the consumer created some output (based on messages in a batch), then crashed before commiting the offset after those messages.  When a new consumer takes over, it will start from slightly old offsets, and re-process messages that have already influenced the output.  This could result in "overcounting" or other incorrect output.

Q: If we have Kafka consumers (a) processing messages from a Kafka topic and (b) writing output based on that processing, how can we guarantee each message is considered exactly once in the output, even if the consumer crashes and restarts?
A: Whenever the consumer writes output, we want to simultaneously (that is, "atomically") write offsets to indicate what messages were used to create that output.  For example, if the consumer is writing to a MySQL database, it might do two INSERTs as part of the same transaction (one INSERT for the actual output, and another INSERT to record the corresponding offsets somewhere).