In November I read The Hadoop Distributed File System. I wanted to understand how a file system built from ordinary servers could store very large datasets reliably and feed computation at high aggregate bandwidth.
HDFS is deliberately not a general POSIX file system. It is designed for large files, streaming access and write-once or append-oriented workloads.
Paper: The Hadoop Distributed File System (PDF)
Separate metadata from block data
HDFS has two main server roles:
- The NameNode owns the namespace and block-placement metadata.
- DataNodes store block replicas and serve bytes to clients.
A file is divided into large blocks. The NameNode records which blocks form a file and which DataNodes currently hold each block. The actual file content does not flow through the NameNode.
This separation keeps the metadata service focused on small control operations while data transfer scales across the disks and network interfaces of many DataNodes.
Why blocks are large
HDFS workloads commonly scan large datasets. Large blocks reduce metadata volume and let clients perform long sequential transfers rather than many small seeks.
The trade-off is poor efficiency for millions of tiny files. Every file and block consumes NameNode memory even when the file contains little data. HDFS is strongest when the dataset contains fewer, larger objects.
The read path
To read a file:
- The client asks the NameNode for block locations.
- The NameNode returns replicas ordered by network proximity.
- The client reads directly from a nearby DataNode.
- At a block boundary, the client chooses a replica for the next block.
If a DataNode fails or a checksum is invalid, the client can retry another replica and report the bad copy.
Keeping the client in control avoids proxying data through a central service. It also lets the client choose a local or same-rack replica when possible.
The write pipeline
For a new block, the NameNode selects a set of DataNodes according to the replication and rack-placement policy. The client streams packets through a pipeline:
client → DN1 → DN2 → DN3 ↑ ↑ ↑ └──── acknowledgements ────┘Each DataNode stores the packet and forwards it to the next. Acknowledgements travel back through the pipeline. The client advances after the required acknowledgements arrive.
This is more network-efficient than sending three independent copies from the client. It also creates ordered error handling: if one DataNode fails, the pipeline is rebuilt around the remaining nodes and replication can be restored.
Rack-aware replication
Replica placement must survive more than disk failure. Machines in one rack share power and a top-of-rack switch.
HDFS balances competing goals:
- Keep some replicas close enough to reduce write traffic.
- Put replicas on another rack to survive rack failure.
- Spread reads and recovery work across the cluster.
The exact policy evolves, but the general lesson is stable: replica count is not the same as failure independence.
NameNode state
The NameNode keeps the namespace and block map in memory for fast access. Persistent namespace state is represented by:
- FsImage: a checkpoint of the file-system namespace.
- Edit log: operations since that checkpoint.
At startup the NameNode loads the image and replays edits. Checkpointing merges them so the log does not grow without bound.
DataNode locations are not treated as permanent namespace state. DataNodes send block reports, allowing the NameNode to reconstruct the current placement map. This follows a useful principle: persist the authoritative intent; rediscover volatile placement from the storage nodes.
Heartbeats, block reports and repair
DataNodes send frequent heartbeats to show liveness and periodic block reports to describe stored replicas. If heartbeats stop, the NameNode marks the DataNode unavailable and schedules under-replicated blocks elsewhere.
Repair consumes real cluster bandwidth. HDFS limits replication work per node so a large failure does not overwhelm foreground reads and writes.
The NameNode also handles over-replication, invalid replicas and rebalancing as nodes join or storage usage becomes uneven.
Checksums and corruption
Replication protects against missing copies, but it does not by itself detect corrupted bytes. HDFS stores checksums for chunks of block data. Clients verify them while reading.
When a checksum fails, another replica can satisfy the read and the bad replica can be removed and recreated. This combination—detect, route around, repair—is more robust than assuming disks fail only by becoming completely unavailable.
The single-NameNode trade-off
The 2010 design centralizes namespace authority in one NameNode. This simplifies consistency and placement decisions, but limits the number of files and creates an availability concern.
The paper discusses the evolution of checkpoint and backup nodes, but later HDFS high-availability and federation work goes beyond the original architecture.
This historical boundary matters. The original paper explains the core data path well, but present-day HDFS includes mechanisms developed after it.
Why computation runs near storage
HDFS and MapReduce were designed together. DataNodes store blocks on local disks, and the scheduler tries to run map tasks on those same machines. For large scans, it is cheaper to send a small executable to the data than to send all data across the cluster network.
This coupling is less universal in modern cloud systems, where compute and object storage are often separated. It remains a useful reminder that architecture depends on the relative cost of network, disk, elasticity and operational isolation.
Failure scenarios I worked through
A DataNode dies during a read: the client chooses another replica.
A DataNode dies during a write: the pipeline is rebuilt, acknowledged packets remain tracked and the NameNode schedules a replacement replica.
A disk silently corrupts data: checksum verification detects it; a healthy replica serves the read.
A rack fails: cross-rack replicas keep blocks available.
The NameNode restarts: it reconstructs namespace state from image and edits, then receives fresh block reports.
Each response depends on separating metadata truth, observed block placement and verified data content.
What I keep from the paper
HDFS is strong because its assumptions are explicit:
files are largeaccess is mostly streamingwrites are simplefailures are normalcompute often follows dataThose assumptions justify large blocks, central metadata, direct client-to-DataNode transfer and pipeline replication.
When designing a storage layer, I would ask whether the workload truly matches those assumptions. If it needs low-latency random updates, many tiny files or full POSIX semantics, copying the HDFS architecture would import the wrong trade-offs. If it needs durable, high-throughput scans over very large datasets, the design remains remarkably clear.