org.apache.hadoop.fs
Class AbstractFileSystem

java.lang.Object
  extended by org.apache.hadoop.fs.AbstractFileSystem

@InterfaceAudience.Public
@InterfaceStability.Evolving
public abstract class AbstractFileSystem
extends Object

This class provides an interface for implementors of a Hadoop file system (analogous to the VFS of Unix). Applications do not access this class; instead they access files across all file systems using FileContext. Pathnames passed to AbstractFileSystem can be fully qualified URI that matches the "this" file system (ie same scheme and authority) or a Slash-relative name that is assumed to be relative to the root of the "this" file system .


Field Summary
protected  FileSystem.Statistics statistics
          The statistics for this file system.
 
Constructor Summary
protected AbstractFileSystem(URI uri, String supportedScheme, boolean authorityNeeded, int defaultPort)
          Constructor to be called by subclasses.
 
Method Summary
protected  void checkPath(Path path)
          Check that a Path belongs to this FileSystem.
protected  void checkScheme(URI uri, String supportedScheme)
           
protected static void clearStatistics()
           
protected  FSDataOutputStream create(Path f, EnumSet<CreateFlag> createFlag, Options.CreateOpts... opts)
          The specification of this method matches that of FileContext.create(Path, EnumSet, Options.CreateOpts...) except that the Path f must be fully qualified and the permission is absolute (i.e.
protected abstract  FSDataOutputStream createInternal(Path f, EnumSet<CreateFlag> flag, FsPermission absolutePermission, int bufferSize, short replication, long blockSize, Progressable progress, int bytesPerChecksum, boolean createParent)
          The specification of this method matches that of create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.
protected  void createSymlink(Path target, Path link, boolean createParent)
          The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);
protected abstract  boolean delete(Path f, boolean recursive)
          The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.
protected abstract  BlockLocation[] getFileBlockLocations(Path f, long start, long len)
          The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.
protected abstract  FileChecksum getFileChecksum(Path f)
          The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.
protected  FileStatus getFileLinkStatus(Path f)
          The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component.
protected abstract  FileStatus getFileStatus(Path f)
          The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.
protected abstract  FsStatus getFsStatus()
          The specification of this method matches that of FileContext.getFsStatus(Path).
protected  FsStatus getFsStatus(Path f)
          The specification of this method matches that of FileContext.getFsStatus(Path) except that Path f must be for this file system.
protected  Path getHomeDirectory()
          Return the current user's home directory in this file system.
protected  Path getInitialWorkingDirectory()
          Some file systems like LocalFileSystem have an initial workingDir that is used as the starting workingDir.
protected  Path getLinkTarget(Path f)
          The specification of this method matches that of FileContext.getLinkTarget(Path);
protected abstract  FsServerDefaults getServerDefaults()
          Return a set of server default configuration values.
protected  FileSystem.Statistics getStatistics()
           
protected static FileSystem.Statistics getStatistics(String scheme, Class<? extends AbstractFileSystem> cls)
          Get the statistics for a particular file system.
protected  URI getUri()
          Returns a URI whose scheme and authority identify this FileSystem.
protected abstract  int getUriDefaultPort()
          The default port of this file system.
protected  String getUriPath(Path p)
          Get the path-part of a pathname.
protected abstract  FileStatus[] listStatus(Path f)
          The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.
protected  Iterator<FileStatus> listStatusIterator(Path f)
          The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.
protected abstract  void mkdir(Path dir, FsPermission permission, boolean createParent)
          The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e.
protected  FSDataInputStream open(Path f)
          The specification of this method matches that of FileContext.open(Path) except that Path f must be for this file system.
protected abstract  FSDataInputStream open(Path f, int bufferSize)
          The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.
protected static void printStatistics()
           
protected  void rename(Path src, Path dst, Options.Rename... options)
          The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.
protected abstract  void renameInternal(Path src, Path dst)
          The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed.
protected  void renameInternal(Path src, Path dst, boolean overwrite)
          The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.
protected abstract  void setOwner(Path f, String username, String groupname)
          The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.
protected abstract  void setPermission(Path f, FsPermission permission)
          The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.
protected abstract  boolean setReplication(Path f, short replication)
          The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.
protected abstract  void setTimes(Path f, long mtime, long atime)
          The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.
protected abstract  void setVerifyChecksum(boolean verifyChecksum)
          The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.
protected  boolean supportsSymlinks()
          Returns true if the file system supports symlinks, false otherwise.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

statistics

protected FileSystem.Statistics statistics
The statistics for this file system.

Constructor Detail

AbstractFileSystem

protected AbstractFileSystem(URI uri,
                             String supportedScheme,
                             boolean authorityNeeded,
                             int defaultPort)
                      throws URISyntaxException
Constructor to be called by subclasses.

Parameters:
uri - for this file system.
supportedScheme - the scheme supported by the implementor
authorityNeeded - if true then theURI must have authority, if false then the URI must have null authority.
Throws:
URISyntaxException - uri has syntax error
Method Detail

getStatistics

protected FileSystem.Statistics getStatistics()

getStatistics

protected static FileSystem.Statistics getStatistics(String scheme,
                                                     Class<? extends AbstractFileSystem> cls)
Get the statistics for a particular file system.

Parameters:
cls - the class to lookup
Returns:
a statistics object

clearStatistics

protected static void clearStatistics()

printStatistics

protected static void printStatistics()

checkScheme

protected void checkScheme(URI uri,
                           String supportedScheme)

getUriDefaultPort

protected abstract int getUriDefaultPort()
The default port of this file system.

Returns:
default port of this file system's Uri scheme A uri with a port of -1 => default port;

getUri

protected URI getUri()
Returns a URI whose scheme and authority identify this FileSystem.

Returns:
the uri of this file system.

checkPath

protected void checkPath(Path path)
Check that a Path belongs to this FileSystem. If the path is fully qualified URI, then its scheme and authority matches that of this file system. Otherwise the path must be slash-relative name.

Throws:
InvalidPathException - if the path is invalid

getUriPath

protected String getUriPath(Path p)
Get the path-part of a pathname. Checks that URI matches this file system and that the path-part is a valid name.

Parameters:
p - path
Returns:
path-part of the Path p

getInitialWorkingDirectory

protected Path getInitialWorkingDirectory()
Some file systems like LocalFileSystem have an initial workingDir that is used as the starting workingDir. For other file systems like HDFS there is no built in notion of an initial workingDir.

Returns:
the initial workingDir if the file system has such a notion otherwise return a null.

getHomeDirectory

protected Path getHomeDirectory()
Return the current user's home directory in this file system. The default implementation returns "/user/$USER/".

Returns:
current user's home directory.

getServerDefaults

protected abstract FsServerDefaults getServerDefaults()
                                               throws IOException
Return a set of server default configuration values.

Returns:
server default configuration values
Throws:
IOException - an I/O error occurred

create

protected final FSDataOutputStream create(Path f,
                                          EnumSet<CreateFlag> createFlag,
                                          Options.CreateOpts... opts)
                                   throws org.apache.hadoop.security.AccessControlException,
                                          FileAlreadyExistsException,
                                          FileNotFoundException,
                                          ParentNotDirectoryException,
                                          UnsupportedFileSystemException,
                                          org.apache.hadoop.fs.UnresolvedLinkException,
                                          IOException
The specification of this method matches that of FileContext.create(Path, EnumSet, Options.CreateOpts...) except that the Path f must be fully qualified and the permission is absolute (i.e. umask has been applied).

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
UnsupportedFileSystemException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

createInternal

protected abstract FSDataOutputStream createInternal(Path f,
                                                     EnumSet<CreateFlag> flag,
                                                     FsPermission absolutePermission,
                                                     int bufferSize,
                                                     short replication,
                                                     long blockSize,
                                                     Progressable progress,
                                                     int bytesPerChecksum,
                                                     boolean createParent)
                                              throws org.apache.hadoop.security.AccessControlException,
                                                     FileAlreadyExistsException,
                                                     FileNotFoundException,
                                                     ParentNotDirectoryException,
                                                     UnsupportedFileSystemException,
                                                     org.apache.hadoop.fs.UnresolvedLinkException,
                                                     IOException
The specification of this method matches that of create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
UnsupportedFileSystemException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

mkdir

protected abstract void mkdir(Path dir,
                              FsPermission permission,
                              boolean createParent)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileAlreadyExistsException,
                              FileNotFoundException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException
The specification of this method matches that of FileContext.mkdir(Path, FsPermission, boolean) except that the Path f must be fully qualified and the permission is absolute (i.e. umask has been applied).

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

delete

protected abstract boolean delete(Path f,
                                  boolean recursive)
                           throws org.apache.hadoop.security.AccessControlException,
                                  FileNotFoundException,
                                  org.apache.hadoop.fs.UnresolvedLinkException,
                                  IOException
The specification of this method matches that of FileContext.delete(Path, boolean) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

open

protected FSDataInputStream open(Path f)
                          throws org.apache.hadoop.security.AccessControlException,
                                 FileNotFoundException,
                                 org.apache.hadoop.fs.UnresolvedLinkException,
                                 IOException
The specification of this method matches that of FileContext.open(Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

open

protected abstract FSDataInputStream open(Path f,
                                          int bufferSize)
                                   throws org.apache.hadoop.security.AccessControlException,
                                          FileNotFoundException,
                                          org.apache.hadoop.fs.UnresolvedLinkException,
                                          IOException
The specification of this method matches that of FileContext.open(Path, int) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setReplication

protected abstract boolean setReplication(Path f,
                                          short replication)
                                   throws org.apache.hadoop.security.AccessControlException,
                                          FileNotFoundException,
                                          org.apache.hadoop.fs.UnresolvedLinkException,
                                          IOException
The specification of this method matches that of FileContext.setReplication(Path, short) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

rename

protected final void rename(Path src,
                            Path dst,
                            Options.Rename... options)
                     throws org.apache.hadoop.security.AccessControlException,
                            FileAlreadyExistsException,
                            FileNotFoundException,
                            ParentNotDirectoryException,
                            org.apache.hadoop.fs.UnresolvedLinkException,
                            IOException
The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

renameInternal

protected abstract void renameInternal(Path src,
                                       Path dst)
                                throws org.apache.hadoop.security.AccessControlException,
                                       FileAlreadyExistsException,
                                       FileNotFoundException,
                                       ParentNotDirectoryException,
                                       org.apache.hadoop.fs.UnresolvedLinkException,
                                       IOException
The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system and NO OVERWRITE is performed. File systems that do not have a built in overwrite need implement only this method and can take advantage of the default impl of the other renameInternal(Path, Path, boolean)

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

renameInternal

protected void renameInternal(Path src,
                              Path dst,
                              boolean overwrite)
                       throws org.apache.hadoop.security.AccessControlException,
                              FileAlreadyExistsException,
                              FileNotFoundException,
                              ParentNotDirectoryException,
                              org.apache.hadoop.fs.UnresolvedLinkException,
                              IOException
The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileAlreadyExistsException
FileNotFoundException
ParentNotDirectoryException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

supportsSymlinks

protected boolean supportsSymlinks()
Returns true if the file system supports symlinks, false otherwise.


createSymlink

protected void createSymlink(Path target,
                             Path link,
                             boolean createParent)
                      throws IOException,
                             org.apache.hadoop.fs.UnresolvedLinkException
The specification of this method matches that of FileContext.createSymlink(Path, Path, boolean);

Throws:
IOException
org.apache.hadoop.fs.UnresolvedLinkException

getLinkTarget

protected Path getLinkTarget(Path f)
                      throws IOException
The specification of this method matches that of FileContext.getLinkTarget(Path);

Throws:
IOException

setPermission

protected abstract void setPermission(Path f,
                                      FsPermission permission)
                               throws org.apache.hadoop.security.AccessControlException,
                                      FileNotFoundException,
                                      org.apache.hadoop.fs.UnresolvedLinkException,
                                      IOException
The specification of this method matches that of FileContext.setPermission(Path, FsPermission) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setOwner

protected abstract void setOwner(Path f,
                                 String username,
                                 String groupname)
                          throws org.apache.hadoop.security.AccessControlException,
                                 FileNotFoundException,
                                 org.apache.hadoop.fs.UnresolvedLinkException,
                                 IOException
The specification of this method matches that of FileContext.setOwner(Path, String, String) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setTimes

protected abstract void setTimes(Path f,
                                 long mtime,
                                 long atime)
                          throws org.apache.hadoop.security.AccessControlException,
                                 FileNotFoundException,
                                 org.apache.hadoop.fs.UnresolvedLinkException,
                                 IOException
The specification of this method matches that of FileContext.setTimes(Path, long, long) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileChecksum

protected abstract FileChecksum getFileChecksum(Path f)
                                         throws org.apache.hadoop.security.AccessControlException,
                                                FileNotFoundException,
                                                org.apache.hadoop.fs.UnresolvedLinkException,
                                                IOException
The specification of this method matches that of FileContext.getFileChecksum(Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileStatus

protected abstract FileStatus getFileStatus(Path f)
                                     throws org.apache.hadoop.security.AccessControlException,
                                            FileNotFoundException,
                                            org.apache.hadoop.fs.UnresolvedLinkException,
                                            IOException
The specification of this method matches that of FileContext.getFileStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFileLinkStatus

protected FileStatus getFileLinkStatus(Path f)
                                throws org.apache.hadoop.security.AccessControlException,
                                       FileNotFoundException,
                                       UnsupportedFileSystemException,
                                       IOException
The specification of this method matches that of FileContext.getFileLinkStatus(Path) except that an UnresolvedLinkException may be thrown if a symlink is encountered in the path leading up to the final path component. If the file system does not support symlinks then the behavior is equivalent to getFileStatus(Path).

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
UnsupportedFileSystemException
IOException

getFileBlockLocations

protected abstract BlockLocation[] getFileBlockLocations(Path f,
                                                         long start,
                                                         long len)
                                                  throws org.apache.hadoop.security.AccessControlException,
                                                         FileNotFoundException,
                                                         org.apache.hadoop.fs.UnresolvedLinkException,
                                                         IOException
The specification of this method matches that of FileContext.getFileBlockLocations(Path, long, long) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFsStatus

protected FsStatus getFsStatus(Path f)
                        throws org.apache.hadoop.security.AccessControlException,
                               FileNotFoundException,
                               org.apache.hadoop.fs.UnresolvedLinkException,
                               IOException
The specification of this method matches that of FileContext.getFsStatus(Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

getFsStatus

protected abstract FsStatus getFsStatus()
                                 throws org.apache.hadoop.security.AccessControlException,
                                        FileNotFoundException,
                                        IOException
The specification of this method matches that of FileContext.getFsStatus(Path).

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
IOException

listStatusIterator

protected Iterator<FileStatus> listStatusIterator(Path f)
                                           throws org.apache.hadoop.security.AccessControlException,
                                                  FileNotFoundException,
                                                  org.apache.hadoop.fs.UnresolvedLinkException,
                                                  IOException
The specification of this method matches that of FileContext.listStatus(Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

listStatus

protected abstract FileStatus[] listStatus(Path f)
                                    throws org.apache.hadoop.security.AccessControlException,
                                           FileNotFoundException,
                                           org.apache.hadoop.fs.UnresolvedLinkException,
                                           IOException
The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
FileNotFoundException
org.apache.hadoop.fs.UnresolvedLinkException
IOException

setVerifyChecksum

protected abstract void setVerifyChecksum(boolean verifyChecksum)
                                   throws org.apache.hadoop.security.AccessControlException,
                                          IOException
The specification of this method matches that of FileContext.setVerifyChecksum(boolean, Path) except that Path f must be for this file system.

Throws:
org.apache.hadoop.security.AccessControlException
IOException


Copyright © 2009 The Apache Software Foundation