Title: 544
Tags: lec19

Q: What does GFS stand for?
A: Google File System

Q: What is the Hadoop system similar to GFS?
A: HDFS (Hadoop File System)

Q: What is a Hadoop system similar to Google's BigTable?
A: HBase

Q: What is a distributed analytics system that many people have started using instead of MapReduce?
A: Spark

Q: When Google MapReduce and BigTable were first created, where did they store their data?
A: GFS

Q: What is the technical word for having multiple copies of our data on different systems?
A: replication

Q: What is the technical word for breaking our data (for example, a file) into smaller pieces that are more manageable?
A: partitioning

Q: HDFS has a boss/worker architecture.  What are the specific names given to the boss and workers in an HDFS cluster?
A: boss: NameNode
worker: DataNode

Q: When a client wants to read an HDFS file, how does it interact with the NameNode and DataNodes?
A: First it tells the NameNode what it wants to read, and the NameNode responds with information about where the client can find the data.
Then, the client fetches the data directly from the DataNodes.
Data does not flow through the NameNode because otherwise having a single NameNode would be a major bottleneck.

Q: What are pipelined writes?
A: If a client wants to write 3 replicas of a block to DataNodes DN1, DN2, and DN3, it does not send the data directly to each of them.
Instead, it sends the data to DN1, which forwards to DN2, which forwards to DN3.
This prevents any one computer from having too much work (most importantly, the client isn't doing all the work, because the client may be a weaker machine than the servers).

Q: If you write 1 MB to a triply replicated file, how much disk I/O does that generate?  What if you read 1 MB from a triply replicated file?
A: Write: 3 MB (need a copy on each machine, for safety)
Read: 1 MB (the three machines have identical data, so there is no point in requesting it from more than one)

Q: How do you format an HDFS NameNode?
A: hdfs namenode -format

Q: How do you start running an HDFS NameNode with hostname "main" and port 9000?
A: hdfs namenode --fs hdfs://main:9000

Q: How can you get an HDFS cluster report for hdfs://main:9000/?
A: hdfs dfsadmin -fs hdfs://main:9000/ -report

Q: How can you start a DataNode and connect it to a NameNode with hostname "main" on port 9000?
A: hdfs datanode -fs hdfs://main:9000

Q: If you want to use commands like mkdir, cp, ls, cat, and du on HDFS, how do you do it?
A: hdfs dfs -COMMAND ARGS
For example, to run "ls" on the "data" directory:
hdfs dfs -ls hdfs://main:9000/data/