Attending BigMine 2015 workshop as part of KDD 2015 in Sydney to give a demo on ADAMS.

More details here.

It's been a bit quieter with updates this past week, mainly because I've been preparing material for my presentation/demo in Sydney at the BigMine 2015 workshop on Monday (http://bigdata-mining.org/bigmine-15/).

Fixes

• fixed a performance bottleneck in the GenericArrayEditor when dealing with large arrays (100s of elements)

Changes

• WekaSelectDataset (source): file chooser uses bookmarks panel now

• Instance Explorer: file chooser uses bookmarks panel now

• added multi-monitor support under the hood, i.e., child windows/dialogs show up on the same monitor as its parent (eg graphical actors in the Flow editor)

• added support for "atomic file move" operation to MoveFile transformer and several Rat inputs/outputs

• MoveFile can forward the moved file now instead of the input token (if enabled)

Additions

• flow debugging: Heatmap and Trail data structures now have their own renderer and exporter

• basic zoom functionality has been added to the Flow editor (zooms actor icons and text)

• Wizard (and flow) for creating compatible datasets for WEKA

• added EscapeLatexCharacters conversion to the adams-latex module

Have a good weekend!

It's been quite busy here again, with lots of things happening. Most notably, there is now a new data structure available for managing tracked objects: Trail.

Fixes

• WekaStreamFilter now handles Instances objects correctly

• adams-events/adams-rats: the default scheduler instantiation is now centralized and synchronized to avoid exception that scheduler has already been instantiated (race condition)

• debugging renderer fixed: WekaInstancesRender displayed more than one instance in the table

Changes

• Once/TriggerOnce control actors can be reset now using a monitor variable

• HeatmapDisplay actor now works inside a TabView

• Inspect control actor now allows the automatic closing of the frame after accept/skip was clicked (useful when only using once)

• MathExpression source/transformer now offer rouding (ie output of integer instead of double)

• timeseries now use 1 milli-second as resolution rather than 0.1 second

• FileInUse boolean condition now works on Unix as well

• added IsAndroid boolean condition

• Console sink now allows new lines (n) and tabs (t) in the prefix string

• the GenericObjectEditor now places the popup with the class tree better; especially on Windows, it no longer gets stuck behind the taskbar

Additions

• new conversion

  • Point2DToString/StringToPoint2D

  • QuadrilateralLocationToString/StringToQuadrilateralLocation

  • QuadrilateralLocationCenter

• added support for timestamp strings with milli-seconds: BaseDateTimeMsec experimental support for spreadsheet cells that use a float instead of a double as internal data representation (for conserving memory)

• WaitForFile transformer waits for a file to become fully available (eg when being written or uploaded; confer FileInUse boolean condition)

• Trail data structure for handling tracked objects, supported by the following actors:

  • source: NewTrail

  • transformers; AddTrailBackground, AddTrailStep, GetTrailBackground, TrailFileReader, TrailFileWriter, TrailFilter

  • sink: TrailDisplay

• Trail viewer for loading, saving, filtering trails

That's it for now and have a good weekend!

Despite being a very busy start of term last week, I managed to get in a number of changes. Mainly due to an upcoming collaboration with the Biology department here and being invited to talk at a workshop in August.

Changes

• Support for OSX builds (-app) has been removed. Users can use the bin/run.sh shell script for starting ADAMS from a terminal.

• Java code generation (eg when exporting a flow via ActorExecutionClassProducer) has seen a few readability improvements, which also makes it easy to modify the code: lists instead of arrays are being used now for assembling sub-actors; variable naming has been improved as well

• The Multi-Explorer now copies pretty much everything apart from result histories when using the "copy" button to copy a session.

• The FFmpeg sink was moved to the new adams-video addons module (see below)

• Reports in the Preview Browser now have a preview text area that shows the value of the currently selected report field; useful when the value is quite long or an error message spread out over several lines.

• The ReportDisplay sink now enables the user to use a custom entry name for the report being displaying, using a variable (eg file name)

Additions

• A new addons module got added, aimed at video processing: adams-video

  • WebcamImage source allows you to get a live-feed from a webcam attached to the computer

  • ListWebcams source outputs the names of all the webcams attached to the computer

  • WebcamInfo outputs some information on a webcam

  • MjpegImagesSequence allows the processing of each frame from a MJPEG movie

  • MovieImageSequence allows processing of frames every X msec from a large number of movie file formats

  • TrackObjects allows you to track objects in image sequences, e.g., obtained from movies

  • TransformTrackedObject transformer allows you to transform a tracked object using a callable transformer, e.g., for blurring a face that is being tracked

  • The ExtractTrackedObject transformer extracts the tracked object as separated image for further processing, e.g., feature generation

• The FileInUse boolean condition allows to check whether a file is currently in use by another process on the Windows platform

• The SelectCharset source allows the user to select a character set interactively, e.g., when loading files (= encoding).

• The adams-moa module has seen a few additions as well for regression (MOARegressorSetup, MOARegressing, MOARegressorEvaluator, MOATrainRegressor) and clustering (MOAClusterVisualization sink).

Credits

A lot of the code for processing movies, tracking objects and accessing webcams was taken from BoofCV (http://boofcv.org/) and Xuggle (http://www.xuggle.com/).

Last week wasn't overly busy with changes, so I've waited till now for an update.

Fixes

• DumpFile sink now offers buffering of X tokens before writing to disk in order to improve I/O performance

• ProgressBar now uses the correct font for calculating the offset for the percentage being display

• quick info of the NewSpreadSheet source no longer displays the columns twice, but sheet name and columns

Changes

• Added a section on setting up IntelliJ IDEA for development in the manual

• Added video on how to setup IntelliJ IDEA for development (recording time was 1:47, almost feature film length)

• Added a button for closing tabs to Flow editor, Image viewer, Image processor, Spreadsheet viewer (see close_tab.png attachment)

• The GenericObjectEditor (GOE) can now load/save setups using cmdlines (.cmdline) and XStream XML serialization (.xml) as well, on top of Java serialized objects (.model), using the load/save buttons in the bottom left corner of any GOE window

• Wizards now allow loading/saving the current setup from any page and for the overall wizard - useful when setting up default parameters for various scenarios

• WekaChooseAttributes now offers the displayed message as customizable option

• WekaInstancesInfo can output all attributes now as well; supports output of array as well

• ProgressBar now offers custom prefix and suffix for surrounding the percentage being displayed

• generating code from flows, e.g., using the ActorExecutionClassProducer in the export functionality of the flow editor, now outputs much nicer source code, mimicking the nesting of actors using {...} blocks and using shortened class names wherever possible - gone is the spaghetti code and also some unnecessary dead code!

Additions

• HttpRequest source uses JSoup library for sending GET/POST requests to URLs; handles cookies as well - can be used for web scraping

• conversions: SplitOptions, JoinOptions

• WekaInstancesPlot allows you to plot two attributes against each other using Weka's plotting facility

• WekaSelectDataset source works like SelectFile, but uses Weka's file chooser component

• PlotAttributeVsAttribute got added to the main menu, allowing you to generate plots like the matrix plot in the Explorer's Visualize tab, but you choose what attributes to plot against each other

Removals

• HttpPost transformer now obsolete, due to introduction of HttpRequest source

The next few weeks will most likely be a bit quieter, since B semester is starting up and I'll be busy with my students.

I've compiled a video that shows you how to set up a development environment using maven and IntelliJ IDEA for ADAMS (base and addons modules):

IntelliJ IDEA is now the preferred IDE for ADAMS.

With the ADAMS release finally out, I was able to tackle cleaning out old stuff and updating libraries.

Old stuff

• Emptied Conversion.props files, which are used to keep backward compatibility of old flows in case class names or option names change. Upside is that loading of flows should be quicker again, downside that old flows might no longer work (simply load them with ADAMS 0.4.10 and save them again)

• Removed classes that have been marked as deprecated:

  • adams-core/src/main/java/adams/data/conversion/MakeFilename.java

  • adams-core/src/main/java/adams/data/conversion/TimestampToDouble.java

  • adams-core/src/main/java/adams/flow/processor/FixDeprecatedCommandlineTransformers.java

  • adams-core/src/main/java/adams/flow/transformer/AnyToCommandline.java

  • adams-core/src/main/java/adams/flow/transformer/CommandlineToAny.java

  • adams-core/src/main/java/adams/flow/transformer/FileSize.java

  • adams-core/src/test/java/adams/data/conversion/MakeFilenameTest.java

  • adams-core/src/test/java/adams/data/conversion/TimestampToDoubleTest.java

  • adams-core/src/test/java/adams/flow/processor/FixDeprecatedCommandlineTransformersTest.java

  • adams-core/src/test/java/adams/flow/transformer/AnyToCommandlineTest.java

  • adams-core/src/test/java/adams/flow/transformer/FileSizeTest.java

  • adams-imaging/src/main/java/adams/data/image/transformer/Crop.java

  • adams-imaging/src/main/java/adams/flow/sink/ImageMagickWriter.java

  • adams-imaging/src/main/java/adams/flow/sink/JAIWriter.java

  • adams-imaging/src/main/java/adams/flow/transformer/ImageMagickReader.java

  • adams-imaging/src/main/java/adams/flow/transformer/ImageMetaDataExtractor.java

  • adams-imaging/src/main/java/adams/flow/transformer/JAIReader.java

  • adams-imaging/src/main/java/adams/flow/transformer/SetImagePixel.java

  • adams-imaging/src/test/java/adams/data/image/transformer/CropTest.java

  • adams-imaging/src/test/java/adams/flow/sink/ImageMagickWriterTest.java

  • adams-imaging/src/test/java/adams/flow/sink/JAIWriterTest.java

  • adams-imaging/src/test/java/adams/flow/transformer/ImageMagickReaderTest.java

  • adams-imaging/src/test/java/adams/flow/transformer/ImageMetaDataExtractorTest.java

  • adams-imaging/src/test/java/adams/flow/transformer/JAIReaderTest.java

  • adams-imaging/src/test/java/adams/flow/transformer/SetImagePixelTest.java

  • adams-moa/src/main/java/adams/flow/processor/FixDeprecatedMOAClassifier.java

  • adams-moa/src/main/java/adams/flow/processor/FixDeprecatedMOAClusterer.java

  • adams-moa/src/main/java/adams/flow/transformer/MOAClassifier.java

  • adams-moa/src/main/java/adams/flow/transformer/MOAClusterer.java

  • adams-moa/src/test/java/adams/flow/transformer/MOAClassifierTest.java

  • adams-moa/src/test/java/adams/flow/transformer/MOAClustererTest.java

  • adams-net/src/main/java/adams/flow/processor/FixDeprecatedEmail.java

  • adams-net/src/main/java/adams/flow/sink/Email.java

  • adams-net/src/test/java/adams/flow/sink/EmailTest.java

  • adams-spreadsheet/src/main/java/adams/data/conversion/AggregateSpreadSheet.java

  • adams-spreadsheet/src/main/java/adams/flow/processor/FixDeprecatedStringToSpreadSheet.java

  • adams-spreadsheet/src/main/java/adams/flow/transformer/StringToSpreadSheet.java

  • adams-spreadsheet/src/test/java/adams/data/conversion/AggregateSpreadSheetTest.java

  • adams-spreadsheet/src/test/java/adams/flow/transformer/StringToSpreadSheetTest.java

  • adams-weka/src/main/java/adams/flow/processor/FixDeprecatedWekaClassifier.java

  • adams-weka/src/main/java/adams/flow/processor/FixDeprecatedWekaClusterer.java

  • adams-weka/src/main/java/adams/flow/transformer/WekaClassifier.java

  • adams-weka/src/main/java/adams/flow/transformer/WekaClusterer.java

  • adams-weka/src/test/java/adams/flow/transformer/CommandlineToAnyTest.java

  • adams-weka/src/test/java/adams/flow/transformer/WekaClassifierTest.java

  • adams-weka/src/test/java/adams/flow/transformer/WekaClustererTest.java

Libraries

I updated the following libraries to the specified versions:

• Jython 2.7.0 (Python scripting)

• WEKA timeseries 2015.05.19

• jackcess 2.1.2 (MS Access)

• Apache CXF 3.1.1 (webservices)

• jetty 9.2.10.v20150310 (webserver)

• postgresql 9.4-1201-jdbc41

• JSch 0.1.53 (sftp, scp, ssh, ...)

• mysql-jdbc-connector 5.1.35

• javax.mail 1.4.7 (sending emails)

• RSyntaxTextArea 2.5.7 (eg Groovy/Python syntax highlighting)

• JFlex 1.4.3 (parser generation)

• json-path 2.0.0

• sqlite-jdbc 3.8.10.1

• joda-time 2.8.1

• jide-oss 3.6.9 (eg directory chooser)

• quartz 1.8.6 (cronjobs)

• Apache POI 3.12 (MS Excel documents)

• metadata-extractor 2.8.1 (image meta-data)

• Apache pdfbox 1.8.9 (eg PDF text extraction)

• twitter4j 3.0.6

In case you feel adventurous, give the new snapshots a go! ;-)

ADAMS 0.4.10 has been released. More details and download links are available from here.

ADAMS 0.4.9 has been released. More details and download links are available from here.

It's been a busy week with a lot of little changes before the upcoming release.

Fixes

• The GPD classifier (package weka.classifiers.functions) now uses the GaussianProcesses classifier as fallback, in case it cannot build a model (avoids predicting only NaNs)

• The internal model used by WekaForecasting (adams-timeseries) gets correctly reset now

• Adding actors after the SequenceSource is no longer broken

Changes

• The PropertySheetPanelPage and WekaPropertySheetPanelPage wizard pages now have an optional button panel, for loading/saving the current properties

• The wizard framework now uses the logging infrastructure as well

• The ParameterPanelPage now offers the REGEXP option, which allows the entering of regular expressions

• The Listeners menu item got removed from the Flow editor tree popup menu

• The main menu of ADAMS has been rearranged, with the Machine learning menu getting decluttered, moving stuff into Visualization and Tools

• The Flow editor's menu got consolidated as well, with the Debug and Execution menu getting merged into the new Run menu

• The toolbar of the Flow editor is now configurable as well (eg through its preferences)

• The introduction of the BaseClassname wrapper for class names, make it easier in the GUI to give feedback on an incorrectly entered classname - affects the following actors: Cast, ArrayProcess, ArrayGenerate, SelectObjects, WekaSelectObjects

• The default class for the Cast control actor is now Unknown, which makes the actor now always show up in the actor tree

Additions

• The Dark Lord, an attribute selection algorithm using a genetic algorithm, has been added to the main menu

• Actor processors added for listing the usage of callable actors, variables and storage items

• Find usages is a new menu item in the popup menu Flow editor tree, which allows you to list occurrences of variables, storage and callable actors throughout the flow - and allows you to jump to them (you may need to remove your $HOME/.adams/FlowEditor.props or %USERPROFILE%/_adams/FlowEditor.props file to make this appear)

• The SelectDirectoryPage and SelectMultipleDirectoriesPage pages got added to the wizard

• The ListTables source (adams-core) allows you to list tables from a database, e.g., when you want to iterate over them and extract some information

• With the DatabaseMetaData source (adams-spreadsheet), you can output various types of meta-data information about a database connection/tables/columns.

• The SourceReset source actor allows you to reset all sub-actors whenever a monitored variable changes (like the TransformerReset and SinkReset actors)

• The Flow editor now sports a New from clipboard menu item, which allows you to paste the current actor on the clipboard straight into a new tab

• Main menu items Margin curve and Cost curve got added (adams-weka)

• That's it for now. At the moment, I'm in the process of updating the screenshots in the manual(s). Sometime in the next few days, I'll be making a new release.