org.apache.hadoop.mapred.gridmix
Class Gridmix

java.lang.Object
  extended by org.apache.hadoop.conf.Configured
      extended by org.apache.hadoop.mapred.gridmix.Gridmix
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class Gridmix
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Driver class for the Gridmix3 benchmark. Gridmix accepts a timestamped stream (trace) of job/task descriptions. For each job in the trace, the client will submit a corresponding, synthetic job to the target cluster at the rate in the original trace. The intent is to provide a benchmark that can be configured and extended to closely match the measured resource profile of actual, production loads.


Field Summary
static String GRIDMIX_OUT_DIR
          Output (scratch) directory for submitted jobs.
static String GRIDMIX_QUE_DEP
          The depth of the queue of job descriptions.
static String GRIDMIX_SUB_MUL
          Multiplier to accelerate or decelerate job submission.
static String GRIDMIX_SUB_THR
          Number of submitting threads at the client and upper bound for in-memory split data.
static org.apache.commons.logging.Log LOG
           
 
Constructor Summary
Gridmix()
           
 
Method Summary
protected  InputStream createInputStream(String in)
           
protected  org.apache.hadoop.mapred.gridmix.JobFactory createJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter, String traceIn, org.apache.hadoop.fs.Path scratchDir, org.apache.hadoop.conf.Configuration conf, CountDownLatch startFlag)
           
protected  org.apache.hadoop.mapred.gridmix.JobMonitor createJobMonitor(Statistics stats)
           
protected  org.apache.hadoop.mapred.gridmix.JobSubmitter createJobSubmitter(org.apache.hadoop.mapred.gridmix.JobMonitor monitor, int threads, int queueDepth, org.apache.hadoop.mapred.gridmix.FilePool pool)
           
static void main(String[] argv)
           
protected  void printUsage(PrintStream out)
           
 int run(String[] argv)
           
protected  void writeInputData(long genbytes, org.apache.hadoop.fs.Path ioPath)
          Write random bytes at the path provided.
 
Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

GRIDMIX_OUT_DIR

public static final String GRIDMIX_OUT_DIR
Output (scratch) directory for submitted jobs. Relative paths are resolved against the path provided as input and absolute paths remain independent of it. The default is "gridmix".

See Also:
Constant Field Values

GRIDMIX_SUB_THR

public static final String GRIDMIX_SUB_THR
Number of submitting threads at the client and upper bound for in-memory split data. Submitting threads precompute InputSplits for submitted jobs. This limits the number of splits held in memory waiting for submission and also permits parallel computation of split data.

See Also:
Constant Field Values

GRIDMIX_QUE_DEP

public static final String GRIDMIX_QUE_DEP
The depth of the queue of job descriptions. Before splits are computed, a queue of pending descriptions is stored in memoory. This parameter limits the depth of that queue.

See Also:
Constant Field Values

GRIDMIX_SUB_MUL

public static final String GRIDMIX_SUB_MUL
Multiplier to accelerate or decelerate job submission. As a crude means of sizing a job trace to a cluster, the time separating two jobs is multiplied by this factor.

See Also:
Constant Field Values
Constructor Detail

Gridmix

public Gridmix()
Method Detail

writeInputData

protected void writeInputData(long genbytes,
                              org.apache.hadoop.fs.Path ioPath)
                       throws IOException,
                              InterruptedException
Write random bytes at the path provided.

Throws:
IOException
InterruptedException
See Also:
GenerateData

createInputStream

protected InputStream createInputStream(String in)
                                 throws IOException
Throws:
IOException

createJobMonitor

protected org.apache.hadoop.mapred.gridmix.JobMonitor createJobMonitor(Statistics stats)
                                                                throws IOException
Throws:
IOException

createJobSubmitter

protected org.apache.hadoop.mapred.gridmix.JobSubmitter createJobSubmitter(org.apache.hadoop.mapred.gridmix.JobMonitor monitor,
                                                                           int threads,
                                                                           int queueDepth,
                                                                           org.apache.hadoop.mapred.gridmix.FilePool pool)
                                                                    throws IOException
Throws:
IOException

createJobFactory

protected org.apache.hadoop.mapred.gridmix.JobFactory createJobFactory(org.apache.hadoop.mapred.gridmix.JobSubmitter submitter,
                                                                       String traceIn,
                                                                       org.apache.hadoop.fs.Path scratchDir,
                                                                       org.apache.hadoop.conf.Configuration conf,
                                                                       CountDownLatch startFlag)
                                                                throws IOException
Throws:
IOException

run

public int run(String[] argv)
        throws IOException,
               InterruptedException
Specified by:
run in interface org.apache.hadoop.util.Tool
Throws:
IOException
InterruptedException

main

public static void main(String[] argv)
                 throws Exception
Throws:
Exception

printUsage

protected void printUsage(PrintStream out)


Copyright © 2009 The Apache Software Foundation