Package adams.flow.transformer

Class Cleaner

  • All Implemented Interfaces:
    adams.core.AdditionalInformationHandler, adams.core.CleanUpHandler, adams.core.Destroyable, adams.core.GlobalInfoSupporter, adams.core.io.ModelFileHandler, adams.core.logging.LoggingLevelHandler, adams.core.logging.LoggingSupporter, adams.core.option.OptionHandler, adams.core.QuickInfoSupporter, adams.core.ShallowCopySupporter<adams.flow.core.Actor>, adams.core.SizeOfHandler, adams.core.Stoppable, adams.core.StoppableWithFeedback, adams.core.VariablesInspectionHandler, adams.event.VariableChangeListener, adams.flow.core.Actor, adams.flow.core.ErrorHandler, adams.flow.core.InputConsumer, adams.flow.core.ModelLoaderSupporter, adams.flow.core.OutputProducer, adams.multiprocess.JobRunnerSupporter, Serializable, Comparable
    public class Cleaner
extends adams.flow.transformer.AbstractTransformer
implements adams.flow.core.ModelLoaderSupporter, adams.multiprocess.JobRunnerSupporter
    In case of Instances objects, 'unclean' Instance objects get removed. When receiving an Instance object, a note is attached.
    The following order is used to obtain the model (when using AUTO):
    1. model file present?
    2. source actor present?
    3. storage item present?
    4. The cleaner is instantiated from the provided definition.

    Input/output:
    - accepts:
       adams.flow.container.CleaningContainer
       weka.core.Instance
       weka.core.Instances
    - generates:
       adams.flow.container.CleaningContainer


    Container information:
    - adams.flow.container.CleaningContainer: Instance, Instances, Checks, Cleaner 

    -logging-level <OFF|SEVERE|WARNING|INFO|CONFIG|FINE|FINER|FINEST> (property: loggingLevel)
    The logging level for outputting errors and debugging output.
    default: WARNING
    min-user-mode: Expert
 
    -name <java.lang.String> (property: name)
    The name of the actor.
    default: Cleaner
 
    -annotation <adams.core.base.BaseAnnotation> (property: annotations)
    The annotations to attach to this actor.
    default:
 
    -skip <boolean> (property: skip)
    If set to true, transformation is skipped and the input token is just forwarded
    as it is.
    default: false
 
    -stop-flow-on-error <boolean> (property: stopFlowOnError)
    If set to true, the flow execution at this level gets stopped in case this
    actor encounters an error; the error gets propagated; useful for critical
    actors.
    default: false
    min-user-mode: Expert
 
    -silent <boolean> (property: silent)
    If enabled, then no errors are output in the console; Note: the enclosing
    actor handler must have this enabled as well.
    default: false
    min-user-mode: Expert
 
    -cleaner <adams.data.cleaner.instance.AbstractCleaner> (property: cleaner)
    The Cleaner to use to clean Instances.
    default: adams.data.cleaner.instance.IQRCleaner -pre-filter weka.filters.AllFilter -filter \"weka.filters.unsupervised.attribute.InterquartileRange -R first-last -O 3.0 -E 6.0\"
 
    -model-loading-type <AUTO|FILE|SOURCE_ACTOR|STORAGE> (property: modelLoadingType)
    Determines how to load the model, in case of AUTO, first the model file
    is checked, then the callable actor and then the storage.
    default: AUTO
 
    -model <adams.core.io.PlaceholderFile> (property: modelFile)
    The file to load the model from, ignored if pointing to a directory.
    default: ${CWD}
 
    -model-actor <adams.flow.core.CallableActorReference> (property: modelActor)
    The callable actor (source) to obtain the model from, ignored if not present.
    default:
 
    -model-storage <adams.flow.control.StorageName> (property: modelStorage)
    The storage item to obtain the model from, ignored if not present.
    default: storage
 
    -cleaner-details-output <adams.core.io.PlaceholderFile> (property: cleanerDetailsOutput)
    The file to save the cleaner details to after training, in case the cleaner
    implements the adams.data.cleaner.CleanerDetails interface; ignored if pointing
    to a directory.
    default: ${CWD}
 
    -prefer-jobrunner <boolean> (property: preferJobRunner)
    If enabled, tries to offload the processing onto a adams.flow.standalone.JobRunnerInstance;
     applies only to training.
    default: false
    Author:
    dale (dale at waikato dot ac dot nz)
    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  Cleaner.CleanJob  

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static String BACKUP_ACTUALCLEANER
      the key for storing the current evaluator in the backup.
      protected AbstractCleaner m_ActualCleaner
      the cleaner used for training/evaluating.
      protected AbstractCleaner m_Cleaner
      the evaluator.
      protected adams.core.io.PlaceholderFile m_CleanerDetailsOutput
      the file to save the cleaner details to.
      protected adams.flow.standalone.JobRunnerInstance m_JobRunnerInstance
      the JobRunnerInstance to use.
      protected CleanerModelLoader m_ModelLoader
      the model loader.
      protected boolean m_PreferJobRunner
      whether to offload training into a JobRunnerInstance.

      • Fields inherited from class adams.flow.transformer.AbstractTransformer

        BACKUP_INPUT, BACKUP_OUTPUT, m_InputToken, m_OutputToken

      • Fields inherited from class adams.flow.core.AbstractActor

        m_Annotations, m_BackupState, m_DetectedObjectVariables, m_DetectedVariables, m_ErrorHandler, m_Executed, m_Executing, m_ExecutionListeningSupporter, m_FullName, m_LoggingPrefix, m_Name, m_Parent, m_ScopeHandler, m_Self, m_Silent, m_Skip, m_StopFlowOnError, m_StopMessage, m_Stopped, m_StorageHandler, m_VariablesUpdated

      • Fields inherited from class adams.core.option.AbstractOptionHandler

        m_OptionManager

      • Fields inherited from class adams.core.logging.LoggingObject

        m_Logger, m_LoggingIsEnabled, m_LoggingLevel

      • Fields inherited from interface adams.flow.core.Actor

        FILE_EXTENSION, FILE_EXTENSION_GZ

    • Constructor Summary

      Constructors 
      Constructor Description
      Cleaner()  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      Class[] accepts()
      Returns the class that the consumer accepts.
      protected Hashtable<String,​Object> backupState()
      Backs up the current state of the actor before update the variables.
      String cleanerDetailsOutputTipText()
      Returns the tip text for this property.
      String cleanerTipText()
      Returns the tip text for this property.
      void defineOptions()
      Adds options to the internal list of options.
      protected String doExecute()
      Executes the flow item.
      Class[] generates()
      Returns the class of objects that it generates.
      AbstractCleaner getCleaner()
      Returns the cleaner in use.
      adams.core.io.PlaceholderFile getCleanerDetailsOutput()
      Returns the file for the cleaner details.
      adams.flow.core.CallableActorReference getModelActor()
      Returns the filter source actor.
      adams.core.io.PlaceholderFile getModelFile()
      Returns the file to load the model from.
      adams.flow.core.AbstractModelLoader.ModelLoadingType getModelLoadingType()
      Returns the loading type.
      adams.flow.control.StorageName getModelStorage()
      Returns the filter storage item.
      boolean getPreferJobRunner()
      Returns whether to offload processing to a JobRunner instance if available.
      String getQuickInfo()
      Returns a quick info about the actor, which will be displayed in the GUI.
      String globalInfo()
      Returns a string describing the object.
      protected void initialize()
      Initializes the members.
      String modelActorTipText()
      Returns the tip text for this property.
      String modelFileTipText()
      Returns the tip text for this property.
      String modelLoadingTypeTipText()
      Returns the tip text for this property.
      String modelStorageTipText()
      Returns the tip text for this property.
      String preferJobRunnerTipText()
      Returns the tip text for this property.
      protected void pruneBackup()
      Removes entries from the backup.
      protected void reset()
      Resets the scheme.
      protected void restoreState​(Hashtable<String,​Object> state)
      Restores the state of the actor before the variables got updated.
      void setCleaner​(AbstractCleaner value)
      Sets the cleaner to use.
      void setCleanerDetailsOutput​(adams.core.io.PlaceholderFile value)
      Sets the file for the cleaner details.
      void setLoggingLevel​(adams.core.logging.LoggingLevel value)
      Sets the logging level.
      void setModelActor​(adams.flow.core.CallableActorReference value)
      Sets the filter source actor.
      void setModelFile​(adams.core.io.PlaceholderFile value)
      Sets the file to load the model from.
      void setModelLoadingType​(adams.flow.core.AbstractModelLoader.ModelLoadingType value)
      Sets the loading type.
      void setModelStorage​(adams.flow.control.StorageName value)
      Sets the filter storage item.
      void setPreferJobRunner​(boolean value)
      Sets whether to offload processing to a JobRunner instance if available.
      String setUp()
      Initializes the item for flow execution.
      protected String setUpCleaner()
      Configures the cleaner.
      void stopExecution()
      Stops the execution.
      void wrapUp()
      Cleans up after the execution has finished.

      • Methods inherited from class adams.flow.transformer.AbstractTransformer

        currentInput, execute, hasInput, hasPendingOutput, input, output, postExecute

      • Methods inherited from class adams.flow.core.AbstractActor

        annotationsTipText, canInspectOptions, canPerformSetUpCheck, cleanUp, compareTo, configureLogger, destroy, equals, finalUpdateVariables, findVariables, findVariables, forceVariables, forCommandLine, forName, forName, getAdditionalInformation, getAnnotations, getDefaultName, getDetectedVariables, getErrorHandler, getFlowActors, getFlowExecutionListeningSupporter, getFullName, getName, getNextSibling, getParent, getParentComponent, getPreviousSibling, getRoot, getScopeHandler, getSilent, getSkip, getStopFlowOnError, getStopMessage, getStorageHandler, getVariables, handleError, handleException, hasErrorHandler, hasStopMessage, index, isBackedUp, isExecuted, isExecuting, isFinished, isHeadless, isStopped, nameTipText, performSetUpChecks, performVariableChecks, preExecute, pruneBackup, setAnnotations, setErrorHandler, setName, setParent, setSilent, setSkip, setStopFlowOnError, setVariables, shallowCopy, shallowCopy, silentTipText, sizeOf, skipTipText, stopExecution, stopFlowOnErrorTipText, updateDetectedVariables, updatePrefix, updateVariables, variableChanged

      • Methods inherited from class adams.core.option.AbstractOptionHandler

        cleanUpOptions, finishInit, getDefaultLoggingLevel, getOptionManager, loggingLevelTipText, newOptionManager, toCommandLine, toString

      • Methods inherited from class adams.core.logging.LoggingObject

        getLogger, getLoggingLevel, initializeLogging, isLoggingEnabled

      • Methods inherited from interface adams.flow.core.Actor

        cleanUp, compareTo, destroy, equals, findVariables, getAnnotations, getDefaultName, getDetectedVariables, getErrorHandler, getFlowExecutionListeningSupporter, getFullName, getName, getNextSibling, getParent, getParentComponent, getPreviousSibling, getRoot, getScopeHandler, getSilent, getSkip, getStopFlowOnError, getStopMessage, getStorageHandler, getVariables, handleError, hasErrorHandler, hasStopMessage, index, isExecuted, isFinished, isHeadless, isStopped, setAnnotations, setErrorHandler, setName, setParent, setSilent, setSkip, setStopFlowOnError, setVariables, shallowCopy, shallowCopy, sizeOf, stopExecution, toCommandLine, variableChanged

      • Methods inherited from interface adams.core.AdditionalInformationHandler

        getAdditionalInformation

      • Methods inherited from interface adams.core.logging.LoggingLevelHandler

        getLoggingLevel

      • Methods inherited from interface adams.core.logging.LoggingSupporter

        getLogger, isLoggingEnabled

      • Methods inherited from interface adams.core.option.OptionHandler

        cleanUpOptions, getOptionManager

      • Methods inherited from interface adams.core.VariablesInspectionHandler

        canInspectOptions

    • Field Detail

      • BACKUP_ACTUALCLEANER

        public static final String BACKUP_ACTUALCLEANER
        the key for storing the current evaluator in the backup.
        See Also:
        Constant Field Values

      • m_ActualCleaner

        protected AbstractCleaner m_ActualCleaner
        the cleaner used for training/evaluating.

      • m_CleanerDetailsOutput

        protected adams.core.io.PlaceholderFile m_CleanerDetailsOutput
        the file to save the cleaner details to.

      • m_PreferJobRunner

        protected boolean m_PreferJobRunner
        whether to offload training into a JobRunnerInstance.

      • m_JobRunnerInstance

        protected transient adams.flow.standalone.JobRunnerInstance m_JobRunnerInstance
        the JobRunnerInstance to use.

    • Constructor Detail

      • Cleaner

        public Cleaner()

    • Method Detail

      • globalInfo

        public String globalInfo()
        Returns a string describing the object.
        Specified by:
        globalInfo in interface adams.core.GlobalInfoSupporter
        Specified by:
        globalInfo in class adams.core.option.AbstractOptionHandler
        Returns:
        a description suitable for displaying in the gui

      • defineOptions

        public void defineOptions()
        Adds options to the internal list of options.
        Specified by:
        defineOptions in interface adams.core.option.OptionHandler
        Overrides:
        defineOptions in class adams.flow.core.AbstractActor

      • initialize

        protected void initialize()
        Initializes the members.
        Overrides:
        initialize in class adams.flow.core.AbstractActor

      • setLoggingLevel

        public void setLoggingLevel​(adams.core.logging.LoggingLevel value)
        Sets the logging level.
        Specified by:
        setLoggingLevel in interface adams.core.logging.LoggingLevelHandler
        Overrides:
        setLoggingLevel in class adams.core.option.AbstractOptionHandler
        Parameters:
        value - the level

      • setCleaner

        public void setCleaner​(AbstractCleaner value)
        Sets the cleaner to use.
        Parameters:
        value - the cleaner

      • getCleaner

        public AbstractCleaner getCleaner()
        Returns the cleaner in use.
        Returns:
        the cleaner

      • cleanerTipText

        public String cleanerTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setModelLoadingType

        public void setModelLoadingType​(adams.flow.core.AbstractModelLoader.ModelLoadingType value)
        Sets the loading type. In case of AbstractModelLoader.ModelLoadingType.AUTO, first file, then callable actor, then storage.
        Specified by:
        setModelLoadingType in interface adams.flow.core.ModelLoaderSupporter
        Parameters:
        value - the type

      • getModelLoadingType

        public adams.flow.core.AbstractModelLoader.ModelLoadingType getModelLoadingType()
        Returns the loading type. In case of AbstractModelLoader.ModelLoadingType.AUTO, first file, then callable actor, then storage.
        Specified by:
        getModelLoadingType in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        the type

      • modelLoadingTypeTipText

        public String modelLoadingTypeTipText()
        Returns the tip text for this property.
        Specified by:
        modelLoadingTypeTipText in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setModelFile

        public void setModelFile​(adams.core.io.PlaceholderFile value)
        Sets the file to load the model from.
        Specified by:
        setModelFile in interface adams.core.io.ModelFileHandler
        Specified by:
        setModelFile in interface adams.flow.core.ModelLoaderSupporter
        Parameters:
        value - the model file

      • getModelFile

        public adams.core.io.PlaceholderFile getModelFile()
        Returns the file to load the model from.
        Specified by:
        getModelFile in interface adams.core.io.ModelFileHandler
        Specified by:
        getModelFile in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        the model file

      • modelFileTipText

        public String modelFileTipText()
        Returns the tip text for this property.
        Specified by:
        modelFileTipText in interface adams.core.io.ModelFileHandler
        Specified by:
        modelFileTipText in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setModelActor

        public void setModelActor​(adams.flow.core.CallableActorReference value)
        Sets the filter source actor.
        Specified by:
        setModelActor in interface adams.flow.core.ModelLoaderSupporter
        Parameters:
        value - the source

      • getModelActor

        public adams.flow.core.CallableActorReference getModelActor()
        Returns the filter source actor.
        Specified by:
        getModelActor in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        the source

      • modelActorTipText

        public String modelActorTipText()
        Returns the tip text for this property.
        Specified by:
        modelActorTipText in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setModelStorage

        public void setModelStorage​(adams.flow.control.StorageName value)
        Sets the filter storage item.
        Specified by:
        setModelStorage in interface adams.flow.core.ModelLoaderSupporter
        Parameters:
        value - the storage item

      • getModelStorage

        public adams.flow.control.StorageName getModelStorage()
        Returns the filter storage item.
        Specified by:
        getModelStorage in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        the storage item

      • modelStorageTipText

        public String modelStorageTipText()
        Returns the tip text for this property.
        Specified by:
        modelStorageTipText in interface adams.flow.core.ModelLoaderSupporter
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setCleanerDetailsOutput

        public void setCleanerDetailsOutput​(adams.core.io.PlaceholderFile value)
        Sets the file for the cleaner details.
        Parameters:
        value - the file

      • getCleanerDetailsOutput

        public adams.core.io.PlaceholderFile getCleanerDetailsOutput()
        Returns the file for the cleaner details.
        Returns:
        the file

      • cleanerDetailsOutputTipText

        public String cleanerDetailsOutputTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • setPreferJobRunner

        public void setPreferJobRunner​(boolean value)
        Sets whether to offload processing to a JobRunner instance if available.
        Specified by:
        setPreferJobRunner in interface adams.multiprocess.JobRunnerSupporter
        Parameters:
        value - if true try to find/use a JobRunner instance

      • getPreferJobRunner

        public boolean getPreferJobRunner()
        Returns whether to offload processing to a JobRunner instance if available.
        Specified by:
        getPreferJobRunner in interface adams.multiprocess.JobRunnerSupporter
        Returns:
        if true try to find/use a JobRunner instance

      • preferJobRunnerTipText

        public String preferJobRunnerTipText()
        Returns the tip text for this property.
        Specified by:
        preferJobRunnerTipText in interface adams.multiprocess.JobRunnerSupporter
        Returns:
        tip text for this property suitable for displaying in the GUI or for listing the options.

      • getQuickInfo

        public String getQuickInfo()
        Returns a quick info about the actor, which will be displayed in the GUI.
        Specified by:
        getQuickInfo in interface adams.flow.core.Actor
        Specified by:
        getQuickInfo in interface adams.core.QuickInfoSupporter
        Overrides:
        getQuickInfo in class adams.flow.core.AbstractActor
        Returns:
        null if no info available, otherwise short string

      • pruneBackup

        protected void pruneBackup()
        Removes entries from the backup.
        Overrides:
        pruneBackup in class adams.flow.core.AbstractActor

      • backupState

        protected Hashtable<String,​Object> backupState()
        Backs up the current state of the actor before update the variables.
        Overrides:
        backupState in class adams.flow.transformer.AbstractTransformer
        Returns:
        the backup

      • restoreState

        protected void restoreState​(Hashtable<String,​Object> state)
        Restores the state of the actor before the variables got updated.
        Overrides:
        restoreState in class adams.flow.transformer.AbstractTransformer
        Parameters:
        state - the backup of the state to restore from

      • reset

        protected void reset()
        Resets the scheme.
        Overrides:
        reset in class adams.flow.core.AbstractActor

      • accepts

        public Class[] accepts()
        Returns the class that the consumer accepts.
        Specified by:
        accepts in interface adams.flow.core.InputConsumer
        Returns:
        adams.flow.container.CleaningContainer.class, weka.core.Instance.class, weka.core.Instances.class

      • generates

        public Class[] generates()
        Returns the class of objects that it generates.
        Specified by:
        generates in interface adams.flow.core.OutputProducer
        Returns:
        adams.flow.container.CleaningContainer.class

      • setUp

        public String setUp()
        Initializes the item for flow execution.
        Specified by:
        setUp in interface adams.flow.core.Actor
        Overrides:
        setUp in class adams.flow.core.AbstractActor
        Returns:
        null if everything is fine, otherwise error message

      • setUpCleaner

        protected String setUpCleaner()
        Configures the cleaner.
        Returns:
        null if successful, otherwise error message

      • doExecute

        protected String doExecute()
        Executes the flow item.
        Specified by:
        doExecute in class adams.flow.core.AbstractActor
        Returns:
        null if everything is fine, otherwise error message

      • stopExecution

        public void stopExecution()
        Stops the execution. No message set.
        Specified by:
        stopExecution in interface adams.flow.core.Actor
        Specified by:
        stopExecution in interface adams.core.Stoppable
        Overrides:
        stopExecution in class adams.flow.core.AbstractActor

      • wrapUp

        public void wrapUp()
        Cleans up after the execution has finished.
        Specified by:
        wrapUp in interface adams.flow.core.Actor
        Overrides:
        wrapUp in class adams.flow.transformer.AbstractTransformer