org.apache.hadoop.hdfs.server.datanode
Class FSDataset

java.lang.Object
  extended by org.apache.hadoop.hdfs.server.datanode.FSDataset
All Implemented Interfaces:
FSConstants, FSDatasetInterface, FSDatasetMBean

@InterfaceAudience.Private
public class FSDataset
extends Object
implements FSConstants, FSDatasetInterface

FSDataset manages a set of data blocks. Each block has a unique name and an extent on disk.


Nested Class Summary
 
Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
FSConstants.DatanodeReportType, FSConstants.SafeModeAction, FSConstants.UpgradeAction
 
Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.datanode.FSDatasetInterface
FSDatasetInterface.BlockInputStreams, FSDatasetInterface.BlockWriteStreams, FSDatasetInterface.MetaDataInputStream
 
Field Summary
static String METADATA_EXTENSION
           
static short METADATA_VERSION
           
 
Fields inherited from interface org.apache.hadoop.hdfs.protocol.FSConstants
BLOCK_INVALIDATE_CHUNK, BLOCKREPORT_INITIAL_DELAY, BLOCKREPORT_INTERVAL, BUFFER_SIZE, DEFAULT_BLOCK_SIZE, DEFAULT_BYTES_PER_CHECKSUM, DEFAULT_DATA_SOCKET_SIZE, DEFAULT_FILE_BUFFER_SIZE, DEFAULT_REPLICATION_FACTOR, DEFAULT_WRITE_PACKET_SIZE, HDFS_URI_SCHEME, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_RECOVER_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, QUOTA_DONT_SET, QUOTA_RESET, SIZE_OF_INTEGER, SMALL_BUFFER_SIZE
 
Constructor Summary
FSDataset(DataStorage storage, org.apache.hadoop.conf.Configuration conf)
          An FSDataset has a directory where it loads its data files.
 
Method Summary
 void adjustCrcChannelPosition(Block b, FSDatasetInterface.BlockWriteStreams streams, int checksumSize)
          Sets the offset in the meta file so that the last checksum will be overwritten.
 org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface append(Block b, long newGS, long expectedBlockLen)
          Append to a finalized replica and returns the meta info of the replica
 void checkAndUpdate(long blockId, File diskFile, File diskMetaFile, org.apache.hadoop.hdfs.server.datanode.FSDataset.FSVolume vol)
          Reconcile the difference between blocks on the disk and blocks in volumeMap Check the given block for inconsistencies.
 void checkDataDir()
          check if a data directory is healthy if some volumes failed - make sure to remove all the blocks that belong to these volumes
 org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface createRbw(Block b)
          Creates a RBW replica and returns the meta info of the replica
 org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface createTemporary(Block b)
          Creates a temporary replica and returns the meta information of the replica
 void finalizeBlock(Block b)
          Complete the block write!
 File findBlockFile(long blockId)
          Return the block file for the given ID
 File getBlockFile(Block b)
          Get File name for a given block.
 InputStream getBlockInputStream(Block b)
          Returns an input stream to read the contents of the specified block
 InputStream getBlockInputStream(Block b, long seekOffset)
          Returns an input stream at specified offset of the specified block
 BlockListAsLongs getBlockReport()
          Generates a block report from the in-memory block map.
 long getCapacity()
          Return total capacity, used and unused
 long getDfsUsed()
          Return the total space used by dfs datanode
 File getFile(Block b)
          Turn the block identifier into a filename; ignore generation stamp!!!
 long getLength(Block b)
          Find the block's on-disk length
 FSDatasetInterface.MetaDataInputStream getMetaDataInputStream(Block b)
          Returns metaData of block b as an input stream (and its length)
 long getMetaDataLength(Block b)
          Returns the length of the metadata file of the specified block
protected  File getMetaFile(Block b)
           
 long getRemaining()
          Return how many bytes can still be stored in the FSDataset
 ReplicaInfo getReplica(long blockId)
          Deprecated. use fetchReplicaInfo(long) instead.
 long getReplicaVisibleLength(Block block)
          Get visible length of the specified replica.
 String getStorageInfo()
          Returns the storage id of the underlying storage
 Block getStoredBlock(long blkid)
          
 FSDatasetInterface.BlockInputStreams getTmpInputStreams(Block b, long blkOffset, long ckoff)
          Returns handles to the block file and its metadata file
 boolean hasEnoughResource()
          Return true - if there are still valid volumes on the DataNode.
 ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
          Initialize a replica recovery.
 void invalidate(Block[] invalidBlks)
          We're informed that a block is no longer valid.
 boolean isValidBlock(Block b)
          Check whether the given block is a valid one.
 boolean metaFileExists(Block b)
          Does the meta file exist for this block?
 org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface recoverAppend(Block b, long newGS, long expectedBlockLen)
          Recover a failed append to a finalized replica and returns the meta info of the replica
 void recoverClose(Block b, long newGS, long expectedBlockLen)
          Recover a failed pipeline close It bumps the replica's generation stamp and finalize it if RBW replica
 org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface recoverRbw(Block b, long newGS, long minBytesRcvd, long maxBytesRcvd)
          Recovers a RBW replica and returns the meta info of the replica
 void shutdown()
          Shutdown the FSDataset
 String toString()
          Stringifies the name of the storage
 void unfinalizeBlock(Block b)
          Remove the temporary block file (if any)
 boolean unlinkBlock(Block block, int numLinks)
          Make a copy of the block if this block is linked to an existing snapshot.
 ReplicaInfo updateReplicaUnderRecovery(Block oldBlock, long recoveryId, long newlength)
          Update replica's generation stamp and length and finalize it.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

METADATA_EXTENSION

public static final String METADATA_EXTENSION
See Also:
Constant Field Values

METADATA_VERSION

public static final short METADATA_VERSION
See Also:
Constant Field Values
Constructor Detail

FSDataset

public FSDataset(DataStorage storage,
                 org.apache.hadoop.conf.Configuration conf)
          throws IOException
An FSDataset has a directory where it loads its data files.

Throws:
IOException
Method Detail

getMetaFile

protected File getMetaFile(Block b)
                    throws IOException
Throws:
IOException

findBlockFile

public File findBlockFile(long blockId)
Return the block file for the given ID


getStoredBlock

public Block getStoredBlock(long blkid)
                     throws IOException

Specified by:
getStoredBlock in interface FSDatasetInterface
Returns:
the generation stamp stored with the block.
Throws:
IOException

metaFileExists

public boolean metaFileExists(Block b)
                       throws IOException
Description copied from interface: FSDatasetInterface
Does the meta file exist for this block?

Specified by:
metaFileExists in interface FSDatasetInterface
Parameters:
b - - the block
Returns:
true of the metafile for specified block exits
Throws:
IOException

getMetaDataLength

public long getMetaDataLength(Block b)
                       throws IOException
Description copied from interface: FSDatasetInterface
Returns the length of the metadata file of the specified block

Specified by:
getMetaDataLength in interface FSDatasetInterface
Parameters:
b - - the block for which the metadata length is desired
Returns:
the length of the metadata file for the specified block.
Throws:
IOException

getMetaDataInputStream

public FSDatasetInterface.MetaDataInputStream getMetaDataInputStream(Block b)
                                                              throws IOException
Description copied from interface: FSDatasetInterface
Returns metaData of block b as an input stream (and its length)

Specified by:
getMetaDataInputStream in interface FSDatasetInterface
Parameters:
b - - the block
Returns:
the metadata input stream;
Throws:
IOException

getDfsUsed

public long getDfsUsed()
                throws IOException
Return the total space used by dfs datanode

Specified by:
getDfsUsed in interface FSDatasetMBean
Returns:
the total space used by dfs datanode
Throws:
IOException

hasEnoughResource

public boolean hasEnoughResource()
Return true - if there are still valid volumes on the DataNode.

Specified by:
hasEnoughResource in interface FSDatasetInterface
Returns:
true if more then minimum valid volumes left in the FSDataSet

getCapacity

public long getCapacity()
                 throws IOException
Return total capacity, used and unused

Specified by:
getCapacity in interface FSDatasetMBean
Returns:
total capacity of storage (used and unused)
Throws:
IOException

getRemaining

public long getRemaining()
                  throws IOException
Return how many bytes can still be stored in the FSDataset

Specified by:
getRemaining in interface FSDatasetMBean
Returns:
The amount of free storage space
Throws:
IOException

getLength

public long getLength(Block b)
               throws IOException
Find the block's on-disk length

Specified by:
getLength in interface FSDatasetInterface
Returns:
the specified block's on-disk length (excluding metadta)
Throws:
IOException

getBlockFile

public File getBlockFile(Block b)
                  throws IOException
Get File name for a given block.

Throws:
IOException

getBlockInputStream

public InputStream getBlockInputStream(Block b)
                                throws IOException
Description copied from interface: FSDatasetInterface
Returns an input stream to read the contents of the specified block

Specified by:
getBlockInputStream in interface FSDatasetInterface
Returns:
an input stream to read the contents of the specified block
Throws:
IOException

getBlockInputStream

public InputStream getBlockInputStream(Block b,
                                       long seekOffset)
                                throws IOException
Description copied from interface: FSDatasetInterface
Returns an input stream at specified offset of the specified block

Specified by:
getBlockInputStream in interface FSDatasetInterface
Returns:
an input stream to read the contents of the specified block, starting at the offset
Throws:
IOException

getTmpInputStreams

public FSDatasetInterface.BlockInputStreams getTmpInputStreams(Block b,
                                                               long blkOffset,
                                                               long ckoff)
                                                        throws IOException
Returns handles to the block file and its metadata file

Specified by:
getTmpInputStreams in interface FSDatasetInterface
Returns:
an input stream to read the contents of the specified block, starting at the offset
Throws:
IOException

unlinkBlock

public boolean unlinkBlock(Block block,
                           int numLinks)
                    throws IOException
Make a copy of the block if this block is linked to an existing snapshot. This ensures that modifying this block does not modify data in any existing snapshots.

Parameters:
block - Block
numLinks - Unlink if the number of links exceed this value
Returns:
- true if the specified block was unlinked or the block is not in any snapshot.
Throws:
IOException

append

public org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface append(Block b,
                                                                                long newGS,
                                                                                long expectedBlockLen)
                                                                         throws IOException
Description copied from interface: FSDatasetInterface
Append to a finalized replica and returns the meta info of the replica

Specified by:
append in interface FSDatasetInterface
Parameters:
b - block
newGS - the new generation stamp for the replica
expectedBlockLen - the number of bytes the replica is expected to have
Returns:
the meata info of the replica which is being written to
Throws:
IOException

recoverAppend

public org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface recoverAppend(Block b,
                                                                                       long newGS,
                                                                                       long expectedBlockLen)
                                                                                throws IOException
Description copied from interface: FSDatasetInterface
Recover a failed append to a finalized replica and returns the meta info of the replica

Specified by:
recoverAppend in interface FSDatasetInterface
Parameters:
b - block
newGS - the new generation stamp for the replica
expectedBlockLen - the number of bytes the replica is expected to have
Returns:
the meta info of the replica which is being written to
Throws:
IOException

recoverClose

public void recoverClose(Block b,
                         long newGS,
                         long expectedBlockLen)
                  throws IOException
Description copied from interface: FSDatasetInterface
Recover a failed pipeline close It bumps the replica's generation stamp and finalize it if RBW replica

Specified by:
recoverClose in interface FSDatasetInterface
Parameters:
b - block
newGS - the new generation stamp for the replica
expectedBlockLen - the number of bytes the replica is expected to have
Throws:
IOException

createRbw

public org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface createRbw(Block b)
                                                                            throws IOException
Description copied from interface: FSDatasetInterface
Creates a RBW replica and returns the meta info of the replica

Specified by:
createRbw in interface FSDatasetInterface
Parameters:
b - block
Returns:
the meta info of the replica which is being written to
Throws:
IOException - if an error occurs

recoverRbw

public org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface recoverRbw(Block b,
                                                                                    long newGS,
                                                                                    long minBytesRcvd,
                                                                                    long maxBytesRcvd)
                                                                             throws IOException
Description copied from interface: FSDatasetInterface
Recovers a RBW replica and returns the meta info of the replica

Specified by:
recoverRbw in interface FSDatasetInterface
Parameters:
b - block
newGS - the new generation stamp for the replica
minBytesRcvd - the minimum number of bytes that the replica could have
maxBytesRcvd - the maximum number of bytes that the replica could have
Returns:
the meta info of the replica which is being written to
Throws:
IOException - if an error occurs

createTemporary

public org.apache.hadoop.hdfs.server.datanode.ReplicaInPipelineInterface createTemporary(Block b)
                                                                                  throws IOException
Description copied from interface: FSDatasetInterface
Creates a temporary replica and returns the meta information of the replica

Specified by:
createTemporary in interface FSDatasetInterface
Parameters:
b - block
Returns:
the meta info of the replica which is being written to
Throws:
IOException - if an error occurs

adjustCrcChannelPosition

public void adjustCrcChannelPosition(Block b,
                                     FSDatasetInterface.BlockWriteStreams streams,
                                     int checksumSize)
                              throws IOException
Sets the offset in the meta file so that the last checksum will be overwritten.

Specified by:
adjustCrcChannelPosition in interface FSDatasetInterface
Parameters:
b - block
streams - The stream for the data file and checksum file
checksumSize - number of bytes each checksum has
Throws:
IOException

finalizeBlock

public void finalizeBlock(Block b)
                   throws IOException
Complete the block write!

Specified by:
finalizeBlock in interface FSDatasetInterface
Throws:
IOException

unfinalizeBlock

public void unfinalizeBlock(Block b)
                     throws IOException
Remove the temporary block file (if any)

Specified by:
unfinalizeBlock in interface FSDatasetInterface
Throws:
IOException

getBlockReport

public BlockListAsLongs getBlockReport()
Generates a block report from the in-memory block map.

Specified by:
getBlockReport in interface FSDatasetInterface
Returns:
- the block report - the full list of blocks stored

isValidBlock

public boolean isValidBlock(Block b)
Check whether the given block is a valid one. valid means finalized

Specified by:
isValidBlock in interface FSDatasetInterface
Returns:
- true if the specified block is valid

invalidate

public void invalidate(Block[] invalidBlks)
                throws IOException
We're informed that a block is no longer valid. We could lazily garbage-collect the block, but why bother? just get rid of it.

Specified by:
invalidate in interface FSDatasetInterface
Parameters:
invalidBlks - - the blocks to be invalidated
Throws:
IOException

getFile

public File getFile(Block b)
Turn the block identifier into a filename; ignore generation stamp!!!


checkDataDir

public void checkDataDir()
                  throws org.apache.hadoop.util.DiskChecker.DiskErrorException
check if a data directory is healthy if some volumes failed - make sure to remove all the blocks that belong to these volumes

Specified by:
checkDataDir in interface FSDatasetInterface
Throws:
org.apache.hadoop.util.DiskChecker.DiskErrorException

toString

public String toString()
Description copied from interface: FSDatasetInterface
Stringifies the name of the storage

Specified by:
toString in interface FSDatasetInterface
Overrides:
toString in class Object

shutdown

public void shutdown()
Description copied from interface: FSDatasetInterface
Shutdown the FSDataset

Specified by:
shutdown in interface FSDatasetInterface

getStorageInfo

public String getStorageInfo()
Description copied from interface: FSDatasetMBean
Returns the storage id of the underlying storage

Specified by:
getStorageInfo in interface FSDatasetMBean

checkAndUpdate

public void checkAndUpdate(long blockId,
                           File diskFile,
                           File diskMetaFile,
                           org.apache.hadoop.hdfs.server.datanode.FSDataset.FSVolume vol)
Reconcile the difference between blocks on the disk and blocks in volumeMap Check the given block for inconsistencies. Look at the current state of the block and reconcile the differences as follows:

Parameters:
blockId - Block that differs
diskFile - Block file on the disk
diskMetaFile - Metadata file from on the disk
vol - Volume of the block file

getReplica

@Deprecated
public ReplicaInfo getReplica(long blockId)
Deprecated. use fetchReplicaInfo(long) instead.

Description copied from interface: FSDatasetInterface
Get reference to the replica meta info in the replicasMap. To be called from methods that are synchronized on FSDataset

Specified by:
getReplica in interface FSDatasetInterface
Returns:
replica from the replicas map

initReplicaRecovery

public ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                        throws IOException
Description copied from interface: FSDatasetInterface
Initialize a replica recovery.

Specified by:
initReplicaRecovery in interface FSDatasetInterface
Returns:
actual state of the replica on this data-node or null if data-node does not have the replica.
Throws:
IOException

updateReplicaUnderRecovery

public ReplicaInfo updateReplicaUnderRecovery(Block oldBlock,
                                              long recoveryId,
                                              long newlength)
                                       throws IOException
Description copied from interface: FSDatasetInterface
Update replica's generation stamp and length and finalize it.

Specified by:
updateReplicaUnderRecovery in interface FSDatasetInterface
Throws:
IOException

getReplicaVisibleLength

public long getReplicaVisibleLength(Block block)
                             throws IOException
Description copied from interface: FSDatasetInterface
Get visible length of the specified replica.

Specified by:
getReplicaVisibleLength in interface FSDatasetInterface
Throws:
IOException


Copyright © 2009 The Apache Software Foundation