Package adams.flow.transformer

Class WekaCrossValidationEvaluator

  • All Implemented Interfaces:
    adams.core.AdditionalInformationHandler, adams.core.CleanUpHandler, adams.core.Destroyable, adams.core.GlobalInfoSupporter, adams.core.logging.LoggingLevelHandler, adams.core.logging.LoggingSupporter, adams.core.option.OptionHandler, adams.core.QuickInfoSupporter, adams.core.Randomizable, adams.core.ShallowCopySupporter<adams.flow.core.Actor>, adams.core.SizeOfHandler, adams.core.Stoppable, adams.core.StoppableWithFeedback, adams.core.ThreadLimiter, adams.core.VariablesInspectionHandler, InstancesViewSupporter, adams.event.VariableChangeListener, adams.flow.core.Actor, adams.flow.core.ErrorHandler, adams.flow.core.InputConsumer, adams.flow.core.OutputProducer, Serializable, Comparable
    public class WekaCrossValidationEvaluator
extends AbstractCallableWekaClassifierEvaluator
implements adams.core.Randomizable, adams.core.ThreadLimiter, InstancesViewSupporter
    Cross-validates a classifier on an incoming dataset. The classifier setup being used in the evaluation is a callable 'Classifier' actor.

    Input/output:
    - accepts:
       weka.core.Instances
    - generates:
       adams.flow.container.WekaEvaluationContainer


    Container information:
    - adams.flow.container.WekaEvaluationContainer: Evaluation, Model, Prediction output, Original indices, Test data 

    -logging-level <OFF|SEVERE|WARNING|INFO|CONFIG|FINE|FINER|FINEST> (property: loggingLevel)
    The logging level for outputting errors and debugging output.
    default: WARNING
 
    -name <java.lang.String> (property: name)
    The name of the actor.
    default: WekaCrossValidationEvaluator
 
    -annotation <adams.core.base.BaseAnnotation> (property: annotations)
    The annotations to attach to this actor.
    default:
 
    -skip <boolean> (property: skip)
    If set to true, transformation is skipped and the input token is just forwarded
    as it is.
    default: false
 
    -stop-flow-on-error <boolean> (property: stopFlowOnError)
    If set to true, the flow execution at this level gets stopped in case this
    actor encounters an error; the error gets propagated; useful for critical
    actors.
    default: false
 
    -silent <boolean> (property: silent)
    If enabled, then no errors are output in the console; Note: the enclosing
    actor handler must have this enabled as well.
    default: false
 
    -output <weka.classifiers.evaluation.output.prediction.AbstractOutput> (property: output)
    The class for generating prediction output; if 'Null' is used, then an Evaluation
    object is forwarded instead of a String; not used when using parallel execution.
    default: weka.classifiers.evaluation.output.prediction.Null
 
    -always-use-container <boolean> (property: alwaysUseContainer)
    If enabled, always outputs an evaluation container.
    default: false
 
    -classifier <adams.flow.core.CallableActorReference> (property: classifier)
    The callable classifier actor to cross-validate on the input data.
    default: WekaClassifierSetup
 
    -no-predictions <boolean> (property: discardPredictions)
    If enabled, the collection of predictions during evaluation is suppressed,
     wich will conserve memory.
    default: false
 
    -seed <long> (property: seed)
    The seed value for the cross-validation (used for randomization).
    default: 1
 
    -folds <int> (property: folds)
    The number of folds to use in the cross-validation; use -1 for leave-one-out
    cross-validation (LOOCV).
    default: 10
    minimum: -1
 
    -num-threads <int> (property: numThreads)
    The number of threads to use for parallel execution; > 0: specific number
    of cores to use (capped by actual number of cores available, 1 = sequential
    execution); = 0: number of cores; < 0: number of free cores (eg -2 means
    2 free cores; minimum of one core is used)
    default: 1
 
    -use-views <boolean> (property: useViews)
    If enabled, views of the dataset are being used instead of actual copies,
     to conserve memory.
    default: false
 
    -generator <weka.classifiers.CrossValidationFoldGenerator> (property: generator)
    The scheme to use for generating the folds; the actor options take precedence
    over the scheme's ones.
    default: weka.classifiers.DefaultCrossValidationFoldGenerator
 
    -final-model <boolean> (property: finalModel)
    If enabled, a final model is built on the full dataset.
    default: false
    Author:
    fracpete (fracpete at waikato dot ac dot nz)
    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected WekaCrossValidationExecution m_CrossValidation
      for performing cross-validation.
      protected boolean m_FinalModel
      whether to create a final model.
      protected int m_Folds
      the number of folds.
      protected CrossValidationFoldGenerator m_Generator
      the fold generator.
      protected adams.flow.standalone.JobRunnerSetup m_JobRunnerSetup
      the jobrunner setup.
      protected int m_NumThreads
      the number of threads to use for parallel execution.
      protected long m_Seed
      the seed value.
      protected boolean m_UseViews
      whether to use views.

      • Fields inherited from class adams.flow.transformer.AbstractTransformer

        BACKUP_INPUT, BACKUP_OUTPUT, m_InputToken, m_OutputToken

      • Fields inherited from class adams.flow.core.AbstractActor

        m_Annotations, m_BackupState, m_DetectedObjectVariables, m_DetectedVariables, m_ErrorHandler, m_Executed, m_Executing, m_ExecutionListeningSupporter, m_FullName, m_LoggingPrefix, m_Name, m_Parent, m_ScopeHandler, m_Self, m_Silent, m_Skip, m_StopFlowOnError, m_StopMessage, m_Stopped, m_StorageHandler, m_VariablesUpdated

      • Fields inherited from class adams.core.option.AbstractOptionHandler

        m_OptionManager

      • Fields inherited from class adams.core.logging.LoggingObject

        m_Logger, m_LoggingIsEnabled, m_LoggingLevel

      • Fields inherited from interface adams.flow.core.Actor

        FILE_EXTENSION, FILE_EXTENSION_GZ

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      Class[] accepts()
      Returns the class that the consumer accepts.
      String classifierTipText()
      Returns the tip text for this property.
      void defineOptions()
      Adds options to the internal list of options.
      protected String doExecute()
      Executes the flow item.
      String finalModelTipText()
      Returns the tip text for this property.
      String foldsTipText()
      Returns the tip text for this property.
      Class[] generates()
      Returns the class of objects that it generates.
      String generatorTipText()
      Returns the tip text for this property.
      boolean getFinalModel()
      Returns whether to build a final model on the full dataset.
      int getFolds()
      Returns the number of folds.
      CrossValidationFoldGenerator getGenerator()
      Returns the scheme for generating the folds.
      int getNumThreads()
      Returns the number of threads to use for cross-validation.
      String getQuickInfo()
      Returns a quick info about the actor, which will be displayed in the GUI.
      long getSeed()
      Returns the seed value.
      boolean getUseViews()
      Returns whether to use views instead of dataset copies, in order to conserve memory.
      String globalInfo()
      Returns a string describing the object.
      String numThreadsTipText()
      Returns the tip text for this property.
      String outputTipText()
      Returns the tip text for this property.
      String seedTipText()
      Returns the tip text for this property.
      void setFinalModel​(boolean value)
      Sets whether to build a final model on the full dataset.
      void setFolds​(int value)
      Sets the number of folds.
      void setGenerator​(CrossValidationFoldGenerator value)
      Sets the scheme for generating the folds.
      void setNumThreads​(int value)
      Sets the number of threads to use for cross-validation.
      void setSeed​(long value)
      Sets the seed value.
      String setUp()
      Initializes the item for flow execution.
      void setUseViews​(boolean value)
      Sets whether to use views instead of dataset copies, in order to conserve memory.
      void stopExecution()
      Stops the execution.
      String useViewsTipText()
      Returns the tip text for this property.
      void wrapUp()
      Cleans up after the execution has finished.

      • Methods inherited from class adams.flow.transformer.AbstractTransformer

        backupState, currentInput, execute, hasInput, hasPendingOutput, input, output, postExecute, restoreState

      • Methods inherited from class adams.flow.core.AbstractActor

        annotationsTipText, canInspectOptions, canPerformSetUpCheck, cleanUp, compareTo, configureLogger, destroy, equals, finalUpdateVariables, findVariables, findVariables, forceVariables, forCommandLine, forName, forName, getAdditionalInformation, getAnnotations, getDefaultName, getDetectedVariables, getErrorHandler, getFlowActors, getFlowExecutionListeningSupporter, getFullName, getName, getNextSibling, getParent, getParentComponent, getPreviousSibling, getRoot, getScopeHandler, getSilent, getSkip, getStopFlowOnError, getStopMessage, getStorageHandler, getVariables, handleError, handleException, hasErrorHandler, hasStopMessage, index, initialize, isBackedUp, isExecuted, isExecuting, isFinished, isHeadless, isStopped, nameTipText, performSetUpChecks, performVariableChecks, preExecute, pruneBackup, pruneBackup, setAnnotations, setErrorHandler, setName, setParent, setSilent, setSkip, setStopFlowOnError, setVariables, shallowCopy, shallowCopy, silentTipText, sizeOf, skipTipText, stopExecution, stopFlowOnErrorTipText, updateDetectedVariables, updatePrefix, updateVariables, variableChanged

      • Methods inherited from class adams.core.option.AbstractOptionHandler

        cleanUpOptions, finishInit, getDefaultLoggingLevel, getOptionManager, loggingLevelTipText, newOptionManager, setLoggingLevel, toCommandLine, toString

      • Methods inherited from class adams.core.logging.LoggingObject

        getLogger, getLoggingLevel, initializeLogging, isLoggingEnabled

      • Methods inherited from interface adams.flow.core.Actor

        cleanUp, compareTo, destroy, equals, findVariables, getAnnotations, getDefaultName, getDetectedVariables, getErrorHandler, getFlowExecutionListeningSupporter, getFullName, getName, getNextSibling, getParent, getParentComponent, getPreviousSibling, getRoot, getScopeHandler, getSilent, getSkip, getStopFlowOnError, getStopMessage, getStorageHandler, getVariables, handleError, hasErrorHandler, hasStopMessage, index, isExecuted, isFinished, isHeadless, isStopped, setAnnotations, setErrorHandler, setName, setParent, setSilent, setSkip, setStopFlowOnError, setVariables, shallowCopy, shallowCopy, sizeOf, stopExecution, toCommandLine, variableChanged

      • Methods inherited from interface adams.core.AdditionalInformationHandler

        getAdditionalInformation

      • Methods inherited from interface adams.core.logging.LoggingLevelHandler

        getLoggingLevel, setLoggingLevel

      • Methods inherited from interface adams.core.logging.LoggingSupporter

        getLogger, isLoggingEnabled

      • Methods inherited from interface adams.core.option.OptionHandler

        cleanUpOptions, getOptionManager

      • Methods inherited from interface adams.core.VariablesInspectionHandler

        canInspectOptions

    • Field Detail

      • m_Folds

        protected int m_Folds
        the number of folds.

      • m_Seed

        protected long m_Seed
        the seed value.

      • m_NumThreads

        protected int m_NumThreads
        the number of threads to use for parallel execution.

      • m_UseViews

        protected boolean m_UseViews
        whether to use views.

      • m_FinalModel

        protected boolean m_FinalModel
        whether to create a final model.

      • m_JobRunnerSetup

        protected transient adams.flow.standalone.JobRunnerSetup m_JobRunnerSetup
        the jobrunner setup.

    • Constructor Detail

      • WekaCrossValidationEvaluator

        public WekaCrossValidationEvaluator()

    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the object.
        Specified by:
        globalInfo in interface adams.core.GlobalInfoSupporter
        Specified by:
        globalInfo in class adams.core.option.AbstractOptionHandler
        Returns:
        a description suitable for displaying in the gui

      • getQuickInfo

        public String getQuickInfo()
        Returns a quick info about the actor, which will be displayed in the GUI.
        Specified by:
        getQuickInfo in interface adams.flow.core.Actor
        Specified by:
        getQuickInfo in interface adams.core.QuickInfoSupporter
        Overrides:
        getQuickInfo in class AbstractCallableWekaClassifierEvaluator
        Returns:
        null if no info available, otherwise short string

      • outputTipText

        public String outputTipText()
        Returns the tip text for this property.
        Overrides:
        outputTipText in class AbstractWekaClassifierEvaluator
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setFolds

        public void setFolds​(int value)
        Sets the number of folds.
        Parameters:
        value - the folds, -1 for LOOCV

      • getFolds

        public int getFolds()
        Returns the number of folds.
        Returns:
        the folds

      • foldsTipText

        public String foldsTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setSeed

        public void setSeed​(long value)
        Sets the seed value.
        Specified by:
        setSeed in interface adams.core.Randomizable
        Parameters:
        value - the seed

      • getSeed

        public long getSeed()
        Returns the seed value.
        Specified by:
        getSeed in interface adams.core.Randomizable
        Returns:
        the seed

      • seedTipText

        public String seedTipText()
        Returns the tip text for this property.
        Specified by:
        seedTipText in interface adams.core.Randomizable
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setNumThreads

        public void setNumThreads​(int value)
        Sets the number of threads to use for cross-validation.
        Specified by:
        setNumThreads in interface adams.core.ThreadLimiter
        Parameters:
        value - the number of threads: -1 = # of CPUs/cores; 0/1 = sequential execution

      • getNumThreads

        public int getNumThreads()
        Returns the number of threads to use for cross-validation.
        Specified by:
        getNumThreads in interface adams.core.ThreadLimiter
        Returns:
        the number of threads: -1 = # of CPUs/cores; 0/1 = sequential execution

      • numThreadsTipText

        public String numThreadsTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setUseViews

        public void setUseViews​(boolean value)
        Sets whether to use views instead of dataset copies, in order to conserve memory.
        Specified by:
        setUseViews in interface InstancesViewSupporter
        Parameters:
        value - true if to use views

      • getUseViews

        public boolean getUseViews()
        Returns whether to use views instead of dataset copies, in order to conserve memory.
        Specified by:
        getUseViews in interface InstancesViewSupporter
        Returns:
        true if using views

      • useViewsTipText

        public String useViewsTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setGenerator

        public void setGenerator​(CrossValidationFoldGenerator value)
        Sets the scheme for generating the folds.
        Parameters:
        value - the generator

      • generatorTipText

        public String generatorTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setFinalModel

        public void setFinalModel​(boolean value)
        Sets whether to build a final model on the full dataset.
        Parameters:
        value - true if to build final model

      • getFinalModel

        public boolean getFinalModel()
        Returns whether to build a final model on the full dataset.
        Returns:
        true if to build final model

      • finalModelTipText

        public String finalModelTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • accepts

        public Class[] accepts()
        Returns the class that the consumer accepts.
        Specified by:
        accepts in interface adams.flow.core.InputConsumer
        Returns:
        weka.core.Instances.class

      • generates

        public Class[] generates()
        Returns the class of objects that it generates.
        Specified by:
        generates in interface adams.flow.core.OutputProducer
        Overrides:
        generates in class AbstractWekaClassifierEvaluator
        Returns:
        the output that it generates

      • setUp

        public String setUp()
        Initializes the item for flow execution.
        Specified by:
        setUp in interface adams.flow.core.Actor
        Overrides:
        setUp in class AbstractCallableWekaClassifierEvaluator
        Returns:
        null if everything is fine, otherwise error message

      • doExecute

        protected String doExecute()
        Executes the flow item.
        Specified by:
        doExecute in class adams.flow.core.AbstractActor
        Returns:
        null if everything is fine, otherwise error message

      • stopExecution

        public void stopExecution()
        Stops the execution. No message set.
        Specified by:
        stopExecution in interface adams.flow.core.Actor
        Specified by:
        stopExecution in interface adams.core.Stoppable
        Overrides:
        stopExecution in class adams.flow.core.AbstractActor

      • wrapUp

        public void wrapUp()
        Cleans up after the execution has finished.
        Specified by:
        wrapUp in interface adams.flow.core.Actor
        Overrides:
        wrapUp in class AbstractWekaClassifierEvaluator