adams.flow.transformer
Class WekaExperimentEvaluation

java.lang.Object
  adams.core.ConsoleObject
      adams.core.option.AbstractOptionHandler
          adams.flow.core.AbstractActor
              adams.flow.transformer.AbstractTransformer
                  adams.flow.transformer.WekaExperimentEvaluation

All Implemented Interfaces:

AdditionalInformationHandler, CleanUpHandler, Debuggable, DebugOutputHandler, Destroyable, OptionHandler, QuickInfoSupporter, ShallowCopySupporter<AbstractActor>, SizeOfHandler, Stoppable, VariableChangeListener, ErrorHandler, InputConsumer, OutputProducer, Serializable, Comparable

public class WekaExperimentEvaluation
extends AbstractTransformer

Generates evaluation output of an experiment that was run previously.

Input/output:
- accepts:
weka.experiment.Experiment
- generates:
java.lang.String

Valid options are:

-D <int> (property: debugLevel)
    The greater the number the more additional info the scheme may output to 
    the console (0 = off).
    default: 0
    minimum: 0
 

-name <java.lang.String> (property: name)
    The name of the actor.
    default: WekaExperimentEvaluation
 

-annotation <adams.core.base.BaseText> (property: annotations)
    The annotations to attach to this actor.
    default: 
 

-skip (property: skip)
    If set to true, transformation is skipped and the input token is just forwarded 
    as it is.
 

-stop-flow-on-error (property: stopFlowOnError)
    If set to true, the flow gets stopped in case this actor encounters an error;
     useful for critical actors.
 

-tester <weka.experiment.Tester> (property: tester)
    The testing algorithm to use for performing the evaluations.
    default: weka.experiment.PairedCorrectedTTester -R 0 -S 0.05
 

-comparison <Elapsed_Time_training|Elapsed_Time_testing|UserCPU_Time_training|UserCPU_Time_testing|Serialized_Model_Size|Serialized_Train_Set_Size|Serialized_Test_Set_Size|Number_of_training_instances|Number_of_testing_instances|Number_correct (nominal)|Number_incorrect (nominal)|Number_unclassified (nominal)|Percent_correct (nominal)|Percent_incorrect (nominal)|Percent_unclassified (nominal)|Kappa_statistic (nominal)|Mean_absolute_error|Root_mean_squared_error|Relative_absolute_error|Root_relative_squared_error|Correlation_coefficient (numeric)|SF_prior_entropy|SF_scheme_entropy|SF_entropy_gain|SF_mean_prior_entropy|SF_mean_scheme_entropy|SF_mean_entropy_gain|KB_information (nominal)|KB_mean_information (nominal)|KB_relative_information (nominal)|True_positive_rate (nominal)|Num_true_positives (nominal)|False_positive_rate (nominal)|Num_false_positives (nominal)|True_negative_rate (nominal)|Num_true_negatives (nominal)|False_negative_rate (nominal)|Num_false_negatives (nominal)|IR_precision (nominal)|IR_recall (nominal)|F_measure (nominal)|Area_under_ROC (nominal)|Area_under_PRC (nominal)|Weighted_avg_true_positive_rate|Weighted_avg_false_positive_rate|Weighted_avg_true_negative_rate|Weighted_avg_false_negative_rate|Weighted_avg_IR_precision|Weighted_avg_IR_recall|Weighted_avg_F_measure|Weighted_avg_area_under_ROC|Weighted_avg_area_under_PRC> (property: comparisonField)
    The field to base the comparison of algorithms on.
    default: PERCENT_CORRECT
 

-significance <double> (property: significance)
    The significance level (0-1).
    default: 0.05
    minimum: 1.0E-4
    maximum: 0.9999
 

-test <int> (property: testBase)
    The index of the test base (normally the first classifier, ie '0').
    default: 0
    minimum: 0
 

-row <adams.core.base.BaseString> [-row ...] (property: row)
    The list of fields that define a row (normally the dataset).
    default: Key_Dataset
 

-col <adams.core.base.BaseString> [-col ...] (property: column)
    The list of fields that define a column (normally the schemes).
    default: Key_Scheme, Key_Scheme_options, Key_Scheme_version_ID
 

-swap (property: swapRowsAndColumns)
    If set to true, rows and columns will be swapped.
 

-format <weka.experiment.ResultMatrix> (property: outputFormat)
    The output format for generating the output.
    default: weka.experiment.ResultMatrixPlainText -mean-prec 2 -stddev-prec 2 -col-name-width 0 -row-name-width 25 -mean-width 0 -stddev-width 0 -sig-width 0 -count-width 5 -remove-filter -print-col-names -print-row-names -enum-col-names
 

-header (property: outputHeader)
    If set to true, then a header describing the experiment evaluation will 
    get output as well.
 

Version:

$Revision: 5714 $

Author:

fracpete (fracpete at waikato dot ac dot nz)

See Also:

Serialized Form

Field Summary

protected BaseString[]	m_Column the column (= classifiers).
protected ExperimentStatistic	m_ComparisonField the comparison field.
protected weka.experiment.ResultMatrix	m_OutputFormat the output format.
protected boolean	m_OutputHeader whether to output the header.
protected BaseString[]	m_Row the row (= datasets).
protected double	m_Significance the significance.
protected boolean	m_SwapRowsAndColumns whether to swap rows and columns.
protected int	m_TestBase the test base.
protected weka.experiment.Tester	m_Tester the tester class to use.

Fields inherited from class adams.flow.transformer.AbstractTransformer

BACKUP_INPUT, BACKUP_OUTPUT, m_InputToken, m_OutputToken

Fields inherited from class adams.flow.core.AbstractActor

FILE_EXTENSION, FILE_EXTENSION_GZ, m_Annotations, m_BackupState, m_DetectedObjectVariables, m_DetectedVariables, m_ErrorHandler, m_Executed, m_FullName, m_Headless, m_Name, m_Parent, m_Root, m_Self, m_Skip, m_StopFlowOnError, m_StopMessage, m_Stopped, m_StorageHandler, m_VariablesUpdated

Fields inherited from class adams.core.option.AbstractOptionHandler

m_DebugLevel, m_OptionManager

Constructor Summary

WekaExperimentEvaluation()

Method Summary

Class[]	accepts() Returns the class that the consumer accepts.
String	columnTipText() Returns the tip text for this property.
String	comparisonFieldTipText() Returns the tip text for this property.
void	defineOptions() Adds options to the internal list of options.
protected Vector<String>	determineColumnNames(BaseString[] list, String defaultList, weka.core.Instances inst) Returns a vector with column names of the dataset, listed in "list".
protected String	doExecute() Executes the flow item.
protected void	evaluateExperiment(weka.experiment.Experiment exp) Evaluates the experiment.
Class[]	generates() Returns the class of objects that it generates.
BaseString[]	getColumn() Returns the list of fields that identify a column.
ExperimentStatistic	getComparisonField() Returns the comparison field.
protected weka.core.Instances	getData(weka.experiment.Experiment exp) Loads the experimental results.
weka.experiment.ResultMatrix	getOutputFormat() Returns the output format in use for generating the output.
boolean	getOutputHeader() Returns whether to output the header of the result matrix as well.
String	getQuickInfo() Returns a quick info about the actor, which will be displayed in the GUI.
BaseString[]	getRow() Returns the list of fields that identify a row.
double	getSignificance() Returns the current significance level (0-1).
boolean	getSwapRowsAndColumns() Returns whether to swap rows and columns.
int	getTestBase() Returns the index of the test base.
weka.experiment.Tester	getTester() Returns the Tester in use.
protected weka.experiment.Tester	getTester(weka.experiment.Experiment exp, weka.core.Instances data) Sets up the testing algorithm and returns it.
String	globalInfo() Returns a string describing the object.
String	outputFormatTipText() Returns the tip text for this property.
String	outputHeaderTipText() Returns the tip text for this property.
String	rowTipText() Returns the tip text for this property.
void	setColumn(BaseString[] value) Sets list of fields that identify a column.
void	setComparisonField(ExperimentStatistic value) Sets the comparison field.
void	setOutputFormat(weka.experiment.ResultMatrix value) Sets the output format to use for generating the output.
void	setOutputHeader(boolean value) Sets whether to output the header of the result matrix as well.
void	setRow(BaseString[] value) Sets the list of fields that identify a row.
void	setSignificance(double value) Sets the significance level (0-1).
void	setSwapRowsAndColumns(boolean value) Sets whether to swap rows and columns.
void	setTestBase(int value) Sets the index of the test base.
void	setTester(weka.experiment.Tester value) Sets the Tester to use.
String	significanceTipText() Returns the tip text for this property.
String	swapRowsAndColumnsTipText() Returns the tip text for this property.
String	testBaseTipText() Returns the tip text for this property.
String	testerTipText() Returns the tip text for this property.

Methods inherited from class adams.flow.transformer.AbstractTransformer

backupState, execute, hasPendingOutput, input, output, postExecute, reset, restoreState, wrapUp

Methods inherited from class adams.flow.core.AbstractActor

annotationsTipText, canInspectOptions, canPerformSetUpCheck, cleanUp, compareTo, debug, destroy, equals, findVariables, findVariables, findVariables, forceVariables, forCommandLine, forName, getAdditionalInformation, getAnnotations, getDefaultName, getDetectedVariables, getErrorHandler, getFlowActors, getFullName, getName, getNextSibling, getParent, getPreviousSibling, getRoot, getSkip, getStopFlowOnError, getStopMessage, getStorageHandler, getVariables, handleError, hasErrorHandler, hasStopMessage, index, initialize, isBackedUp, isExecuted, isFinished, isHeadless, isStopped, nameTipText, performSetUpChecks, preExecute, pruneBackup, pruneBackup, setAnnotations, setErrorHandler, setHeadless, setName, setParent, setSkip, setStopFlowOnError, setUp, setVariables, shallowCopy, shallowCopy, sizeOf, skipTipText, stopExecution, stopExecution, stopFlowOnErrorTipText, updateDetectedVariables, updatePrefix, updateVariables, variableChanged

Methods inherited from class adams.core.option.AbstractOptionHandler

cleanUpOptions, debug, debugLevelTipText, finishInit, getDebugLevel, getOptionManager, isDebugOn, newOptionManager, setDebugLevel, toCommandLine, toString

Methods inherited from class adams.core.ConsoleObject

getDebugging, getSystemErr, getSystemOut

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

m_Tester

protected weka.experiment.Tester m_Tester

the tester class to use.

m_ComparisonField

protected ExperimentStatistic m_ComparisonField

the comparison field.

m_Significance

protected double m_Significance

the significance.

m_TestBase

protected int m_TestBase

the test base.

m_Row

protected BaseString[] m_Row

the row (= datasets).

m_Column

protected BaseString[] m_Column

the column (= classifiers).

m_SwapRowsAndColumns

protected boolean m_SwapRowsAndColumns

whether to swap rows and columns.

m_OutputFormat

protected weka.experiment.ResultMatrix m_OutputFormat

the output format.

m_OutputHeader

protected boolean m_OutputHeader

whether to output the header.

Constructor Detail

WekaExperimentEvaluation

public WekaExperimentEvaluation()

Method Detail

globalInfo

public String globalInfo()

Returns a string describing the object.

Specified by:

globalInfo in class AbstractOptionHandler

Returns:

a description suitable for displaying in the gui

defineOptions

public void defineOptions()

Adds options to the internal list of options.

Specified by:

defineOptions in interface OptionHandler

Overrides:

defineOptions in class AbstractActor

getQuickInfo

public String getQuickInfo()

Returns a quick info about the actor, which will be displayed in the GUI.

Specified by:

getQuickInfo in interface QuickInfoSupporter

Overrides:

getQuickInfo in class AbstractActor

Returns:

null if no info available, otherwise short string

setTester

public void setTester(weka.experiment.Tester value)

Sets the Tester to use.

Parameters:

value - the Tester

getTester

public weka.experiment.Tester getTester()

Returns the Tester in use.

Returns:

the Tester

testerTipText

public String testerTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setComparisonField

public void setComparisonField(ExperimentStatistic value)

Sets the comparison field.

Parameters:

value - the field

getComparisonField

public ExperimentStatistic getComparisonField()

Returns the comparison field.

Returns:

the string

comparisonFieldTipText

public String comparisonFieldTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setSignificance

public void setSignificance(double value)

Sets the significance level (0-1).

Parameters:

value - the significance

getSignificance

public double getSignificance()

Returns the current significance level (0-1).

Returns:

the significance

significanceTipText

public String significanceTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setTestBase

public void setTestBase(int value)

Sets the index of the test base.

Parameters:

value - the index

getTestBase

public int getTestBase()

Returns the index of the test base.

Returns:

the index

testBaseTipText

public String testBaseTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setRow

public void setRow(BaseString[] value)

Sets the list of fields that identify a row.

Parameters:

value - the list of fields

getRow

public BaseString[] getRow()

Returns the list of fields that identify a row.

Returns:

the list of fields

rowTipText

public String rowTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setColumn

public void setColumn(BaseString[] value)

Sets list of fields that identify a column.

Parameters:

value - the list of fields

getColumn

public BaseString[] getColumn()

Returns the list of fields that identify a column.

Returns:

the list of fields

columnTipText

public String columnTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setSwapRowsAndColumns

public void setSwapRowsAndColumns(boolean value)

Sets whether to swap rows and columns.

Parameters:

value - true if to swap rows and columns

getSwapRowsAndColumns

public boolean getSwapRowsAndColumns()

Returns whether to swap rows and columns.

Returns:

true if swapping rows and columns

swapRowsAndColumnsTipText

public String swapRowsAndColumnsTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setOutputFormat

public void setOutputFormat(weka.experiment.ResultMatrix value)

Sets the output format to use for generating the output.

Parameters:

value - the format

getOutputFormat

public weka.experiment.ResultMatrix getOutputFormat()

Returns the output format in use for generating the output.

Returns:

the format

outputFormatTipText

public String outputFormatTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

setOutputHeader

public void setOutputHeader(boolean value)

Sets whether to output the header of the result matrix as well.

Parameters:

value - true if to output the header as well

getOutputHeader

public boolean getOutputHeader()

Returns whether to output the header of the result matrix as well.

Returns:

true if to output the header as well

outputHeaderTipText

public String outputHeaderTipText()

Returns the tip text for this property.

Returns:

tip text for this property suitable for displaying in the GUI or for listing the options.

accepts

public Class[] accepts()

Returns the class that the consumer accepts.

Returns:

weka.experiment.Experiment.class

determineColumnNames

protected Vector<String> determineColumnNames(BaseString[] list,
                                              String defaultList,
                                              weka.core.Instances inst)

Returns a vector with column names of the dataset, listed in "list". If a column cannot be found or the list is empty the ones from the default list are returned.

Parameters:

list - list of attribute names

defaultList - the default list of attribute names

inst - the instances to get the attribute names from

Returns:

a vector containing attribute names

getData

protected weka.core.Instances getData(weka.experiment.Experiment exp)
                               throws Exception

Loads the experimental results.

Parameters:

exp - the experiment to evaluate

Returns:

the results

Throws:

Exception - If reading fails.

getTester

protected weka.experiment.Tester getTester(weka.experiment.Experiment exp,
                                           weka.core.Instances data)
                                    throws Exception

Sets up the testing algorithm and returns it.

Parameters:

exp - the experiment to evaluate

data - the experimental data

Returns:

the configured testing algorithm

Throws:

Exception - If something goes wrong, like testing algorithm of result matrix cannot be instantiated

evaluateExperiment

protected void evaluateExperiment(weka.experiment.Experiment exp)
                           throws Exception

Evaluates the experiment.

Parameters:

exp - the experiment to evaluate

Throws:

Exception - If something goes wrong, like loading data fails or comparison field invalid

doExecute

protected String doExecute()

Executes the flow item.

Specified by:

doExecute in class AbstractActor

Returns:

null if everything is fine, otherwise error message

generates

public Class[] generates()

Returns the class of objects that it generates.

Returns:

java.lang.String.class