org.apache.hadoop.hdfs.server.datanode
Class DataNode

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.hdfs.server.datanode.DataNode
All Implemented Interfaces:
Runnable, org.apache.hadoop.conf.Configurable, ClientDatanodeProtocol, FSConstants, InterDatanodeProtocol, org.apache.hadoop.ipc.VersionedProtocol

@InterfaceAudience.Private
public class DataNode
extends org.apache.hadoop.conf.Configured
implements InterDatanodeProtocol, ClientDatanodeProtocol, FSConstants, Runnable

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.


Nested Class Summary
 
Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction
 
Field Summary
 org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner
           
 org.apache.hadoop.util.Daemon blockScannerThread
           
 FSDatasetInterface data
           
static String DN_CLIENTTRACE_FORMAT
           
 DatanodeRegistration dnRegistration
           
static String EMPTY_DEL_HINT
           
 org.apache.hadoop.ipc.Server ipcServer
           
static org.apache.commons.logging.Log LOG
           
 DatanodeProtocol namenode
           
static int PKT_HEADER_LEN
          Header size for a packet
 
Fields inherited from interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol
versionID
 
Fields inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
BLOCK_INVALIDATE_CHUNK, BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_BYTES_PER_CHECKSUM, DEFAULT_DATA_SOCKET_SIZE, DEFAULT_FILE_BUFFER_SIZE, DEFAULT_REPLICATION_FACTOR, DEFAULT_WRITE_PACKET_SIZE, HDFS_URI_SCHEME, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE
 
Method Summary
protected  void checkDiskError()
          Check if there is a disk failure and if so, handle the error
protected  void checkDiskError(Exception e)
          Check if there is no space in disk
static DataNode createDataNode(String[] args, org.apache.hadoop.conf.Configuration conf)
          Instantiate & Start a single datanode daemon and wait for it to finish.
static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeID datanodeid, org.apache.hadoop.conf.Configuration conf)
           
static InetSocketAddress createSocketAddr(String target)
          Deprecated. 
static DataNode getDataNode()
          Return the DataNode object
 DatanodeRegistration getDatanodeRegistration()
          Return DatanodeRegistration
 FSDatasetInterface getFSDataset()
          This method is used for testing.
 String getNamenode()
          Return the namenode's identifier
 InetSocketAddress getNameNodeAddr()
           
 long getProtocolVersion(String protocol, long clientVersion)
          
 long getReplicaVisibleLength(Block block)
          Return the visible length of a replica.
 InetSocketAddress getSelfAddr()
           
 ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
          Initialize a replica recovery.
static DataNode instantiateDataNode(String[] args, org.apache.hadoop.conf.Configuration conf)
          Instantiate a single datanode object.
static void main(String[] args)
           
protected  Socket newSocket()
          Creates either NIO or regular depending on socketWriteTimeout.
protected  void notifyNamenodeReceivedBlock(Block block, String delHint)
           
 void offerService()
          Main loop for the DataNode.
 org.apache.hadoop.util.Daemon recoverBlocks(Collection<BlockRecoveryCommand.RecoveringBlock> blocks)
           
 void run()
          No matter what kind of exception we get, keep retrying to offerService().
static void runDatanodeDaemon(DataNode dn)
          Start a single datanode daemon and wait for it to finish.
 void scheduleBlockReport(long delay)
          This methods arranges for the data node to send the block report at the next heartbeat.
static void setNewStorageID(DatanodeRegistration dnReg)
           
 void shutdown()
          Shut down this instance of the datanode.
 String toString()
           
 Block updateReplicaUnderRecovery(Block oldBlock, long recoveryId, long newLength)
          Update replica with the new generation stamp and length.
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

DN_CLIENTTRACE_FORMAT

public static final String DN_CLIENTTRACE_FORMAT
See Also:
Constant Field Values

namenode

public DatanodeProtocol namenode

data

public FSDatasetInterface data

dnRegistration

public DatanodeRegistration dnRegistration

EMPTY_DEL_HINT

public static final String EMPTY_DEL_HINT
See Also:
Constant Field Values

blockScanner

public org.apache.hadoop.hdfs.server.datanode.DataBlockScanner blockScanner

blockScannerThread

public org.apache.hadoop.util.Daemon blockScannerThread

ipcServer

public org.apache.hadoop.ipc.Server ipcServer

PKT_HEADER_LEN

public static final int PKT_HEADER_LEN
Header size for a packet

See Also:
Constant Field Values
Method Detail

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)
                                          throws IOException
Deprecated. 

Use NetUtils.createSocketAddr(String) instead.

Throws:
IOException

newSocket

protected Socket newSocket()
                    throws IOException
Creates either NIO or regular depending on socketWriteTimeout.

Throws:
IOException

getDataNode

public static DataNode getDataNode()
Return the DataNode object


createInterDataNodeProtocolProxy

public static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeID datanodeid,
                                                                     org.apache.hadoop.conf.Configuration conf)
                                                              throws IOException
Throws:
IOException

getNameNodeAddr

public InetSocketAddress getNameNodeAddr()

getSelfAddr

public InetSocketAddress getSelfAddr()

getDatanodeRegistration

public DatanodeRegistration getDatanodeRegistration()
Return DatanodeRegistration


getNamenode

public String getNamenode()
Return the namenode's identifier


setNewStorageID

public static void setNewStorageID(DatanodeRegistration dnReg)

shutdown

public void shutdown()
Shut down this instance of the datanode. Returns only after shutdown is complete. This method can only be called by the offerService thread. Otherwise, deadlock might occur.


checkDiskError

protected void checkDiskError(Exception e)
                       throws IOException
Check if there is no space in disk

Parameters:
e - that caused this checkDiskError call
Throws:
IOException

checkDiskError

protected void checkDiskError()
Check if there is a disk failure and if so, handle the error


offerService

public void offerService()
                  throws Exception
Main loop for the DataNode. Runs until shutdown, forever calling remote NameNode functions.

Throws:
Exception

notifyNamenodeReceivedBlock

protected void notifyNamenodeReceivedBlock(Block block,
                                           String delHint)

run

public void run()
No matter what kind of exception we get, keep retrying to offerService(). That's the loop that connects to the NameNode and provides basic DataNode functionality. Only stop when "shouldRun" is turned off (which can only happen at shutdown).

Specified by:
run in interface Runnable

runDatanodeDaemon

public static void runDatanodeDaemon(DataNode dn)
                              throws IOException
Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:
IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                                           org.apache.hadoop.conf.Configuration conf)
                                    throws IOException
Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon(DataNode) subsequently.

Throws:
IOException

createDataNode

public static DataNode createDataNode(String[] args,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException
Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

scheduleBlockReport

public void scheduleBlockReport(long delay)
This methods arranges for the data node to send the block report at the next heartbeat.


getFSDataset

public FSDatasetInterface getFSDataset()
This method is used for testing. Examples are adding and deleting blocks directly. The most common usage will be when the data node's storage is similated.

Returns:
the fsdataset that stores the blocks

main

public static void main(String[] args)

recoverBlocks

public org.apache.hadoop.util.Daemon recoverBlocks(Collection<BlockRecoveryCommand.RecoveringBlock> blocks)

initReplicaRecovery

public ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                        throws IOException
Description copied from interface: InterDatanodeProtocol
Initialize a replica recovery.

Specified by:
initReplicaRecovery in interface InterDatanodeProtocol
Returns:
actual state of the replica on this data-node or null if data-node does not have the replica.
Throws:
IOException

updateReplicaUnderRecovery

public Block updateReplicaUnderRecovery(Block oldBlock,
                                        long recoveryId,
                                        long newLength)
                                 throws IOException
Update replica with the new generation stamp and length.

Specified by:
updateReplicaUnderRecovery in interface InterDatanodeProtocol
Throws:
IOException

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Specified by:
getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol
Throws:
IOException

getReplicaVisibleLength

public long getReplicaVisibleLength(Block block)
                             throws IOException
Return the visible length of a replica.

Specified by:
getReplicaVisibleLength in interface ClientDatanodeProtocol
Throws:
IOException


Copyright © 2009 The Apache Software Foundation