The new Weka release prompted an update and some other changes that occurred during the crazy busy period before Xmas.

Fixes

• adams-spreadsheet: fixed a memory leak when generating and displaying lots of spreadsheets with the SpreadSheetDisplay sink.

• adams-weka and adams-weka-lts: the attribute summary panel now displays min/max/mean/stdev for date attributes using the attribute's date format rather than as number of milli-seconds.

Changes

• The ReportFileReader transformer can output reports as an array now.

• Any variables in the extension option of the ReplaceFileExtension conversion get automatically expanded now.

• adams-spectral-2dim-core: The SampleDataFileReader can output the sample data objects now as an array as well.

• adams-weka: upgraded Weka to 3.9.5-fork-0.0.2

• adams-weka-lts: upgraded Weka to 3.9.0-fork-0.0.12

• adams-meka: upgraded to Meka 1.9.4

Additions

• The Preview browser now has additional actions for local files: open, delete, print.

• The CombineArrays and CombineCollections sources make it easy to combine multiple arrays/collections in storage to be combined into a single one.

• The FileNameCompare comparator, which can be used with the Sort transformer, compares file-like objects, using only the name-part, ignored the path altogether. This allows interleaving files from multiple directories using their name alone.

Bunch of tweaks/fixes and, thanks to working on an image segmentation project, also quite a few additions related to image processing. The Preprocess panel in the Weka Investigator got more user friendly, allowing you to convert some attribute types as well as reordering attributes without using a filter (well, filters are used behind the scene to achieve it).

Fixes

• The debug view of a flow with external actors is no longer displayed in a separate page, but by using a splitview of the same panel (debug=top, actualflow=bottom). This fixes the display problem when debugging flows with external actors.

• The history available through the right-click menu of chooser panels for files and dirs now disambigues the captions of its menu items by prefixing them with parent dir(s).

• In the Debug panel of the Flow editor, once the inspection dialog has been closed, it no longer pops up when selecting another storage item.

• The DirectoryValueDefinition and FileValueDefinition used by EnterManyValues now apply the file chooser title option.

• adams-meta: The ProcessActor transformer now forwards the original actor in case the ModifyingProcessor did not actually modify the actor.

• adams-weka: The PCA tab no longer changes the class attribute of the chosen dataset, but works with a copy of the dataset instead.

Changes

• Added the rounding type RINT and added support for rounding at a specified decimal, available in Round and MathExpression conversions/actors.

• adams-json:

  • The transformers GetJsonKeys, GetJsonValue and SetJsonValue now accept strings (representing JSON) as input as well.

  • The JsonToReport conversion now allows specifying a prefix for the field names.

  • Upgraded gson dependency to 2.8.6

• adams-net: Upgraded requests4j to 0.2.1

• adams-spreadsheet: The SpreadSheetSelectSubset transformer can now preview the select row(s) with a chosen preview generator.

• adams-scripts: improved checks on Windows for Java executable and error messages are now more helpful (command-prompt window doesn't just disappear anymore).

• adams-meka: upgraded to 1.9.3

• adams-ml: The ActualVsPredictedPlot now allows the user to specify a custom paintlet (e.g., the new MetaDataValuePaintlet).

• adams-weka:

  • Classifier errors in the Weka Investigator now support using custom paintlets.

  • Upgraded Weka to 3.9.4-fork-0.0.3 to take advantage of the ConverterFileChooser upgrades

• adams-weka-lts:

  • Upgraded Weka to 3.9.0-fork-0.0.11 to take advantage of the ConverterFileChooser upgrades

• adams-imaging: The PNGImageReader now uses a color provider to replace the colors of PNGs with a palette (the reader does not have access to the color palette itself, unfortunately).

Additions

• The ArrayToReport conversion turns any array into a Report object, using the array index, suffix and prefix to generate field names.

• The MetaDataValuePaintlet paints a meta-data value (with circle as fallback if value not present) at the position of the data point (centered).

• Added colors providers for color blindness (8, 12, 15 and 24 colors), which can be used in plots. They providers are based on palettes from http://mkweb.bcgsc.ca/colorblind/palettes.mhtml

• The RelativeDir transformer turns paths into relative ones by removing the specified path prefix.

• The IterateFilesWithProgressBar flow template generates a sub-flow that iterates files from a directory that matched the provided regular expression. Uses a ProgressBar for providing feedback to the user.

• adams-imaging:

  • Added PNGImageReader for reading PNG images using the pngj library.

  • Added basic support for image segmentation annotations: ImageSegmentationFileReader, ImageSegmentationAnnotator and ImageSegmentationFileWriter. The flow adams-imaging-image_segmentation_annotation.flow uses these actors.

  • The GrayOrIndexedImageWriter image writer turns gray or ones with an indexed palette into color ones using the specified color provider. Useful for making image segmentation images human-viewable.

  • The ReplaceColor BufferedImage transformer plugin allows swapping out colors in an image.

  • The ReplacePredominantColor BufferedImage transformer allows replacing the dominant color (e.g., the background) with another color.

  • The ImageObjectAnnotator transformer supersedes the ImageAnnotator actor, with a more flexible architecture underneath the and undo/redo support.

  • The preview plugin ColorDistributionHandler displays a table with pixel counts per color for an image.

  • With the ImageSegmentationContainerOperation transformer operations on segmentation containers like comparing predictions with annotations are now possible.

  • The ObjectAnnotationsToImageSegmentationLayers conversion generates image segmentation layers from object shape annotations, using the labels of the object as layer name.

• adams-spectral-2dmin-core: Added the filters for updating a spectrum's report based on on other report values: ConditionalReportField, ReportFieldFromStringValue, ReportFieldFromStringValueWithLookup.

• adams-spreadsheet:

  • Added support for JFreeChart bar charts.

  • Added support for reordering rows in a spreadsheet using SpreadSheetReorderRows.

• adams-weka and adams-weka-lts:

  • The FromPredictions evaluation plugin for classification (in the Weka Investigator) allows you to load a spreadsheet and generate the usual outputs available from a regular evaluation run.

  • The Scatter plot tab in the Weka Investigator allows you to plot two attributes from a dataset against each other using various paintlets.

  • The Reorder attributes, Checked to date, Checked to nominal and Checked to string actions got added to the drop down list of actions of Attributes box in the Preprocess tab.

  • The AnyToString filter turns any attribute into a string one.

  • The StringToDate filter turns string attributes into date ones, using the specified parse format (eg "yyyy-MM-dd HH:mm:ss").

• adams-visualstats: The ScatterPaintletMetaDataValue paintlet for scatter plots (like PCA/PLS in Weka Investigator) allows the plotting of a meta-data value as string rather than using Xs or circles (uses circle as fallback if value not present).

Have a great Xmas break!

Lately, it has been quieter than usual with the updates on the ADAMS front. However, that doesn't mean development has stopped. Quite the opposite. At the moment, most of the work goes into our User-friendly Deep Learning framework. It consists of a Python API backend, Python/Java API clients and ADAMS workflows (in additional modules outside the usual ADAMS repos).

Fixes

• The SelectObjects source actor now handles enums as well (couldn't instantiate them from strings if attached as variable).

• using patched version of jide-oss now (3.6.18.1). to avoid problems under OSX

• The ArrayToChunks transformer no longer misses to output the last chunk (under certain circumstances).

• The ExternalFlow standalone actor now updates the programmatic variables for the external flows before executing them.

• adams-imaging: The SimpleCrop cropping algorithm now ensures that the coordinates are within the bounds of the image.

Changes

• The EnterValue source can now display vertical buttons as well when presenting multiple values.

• When using the PlainText renderer for actor annotations in the Flow editor, the sub-string TODO will automatically switch the color to RED (default is BLUE).

• The Convert transformer now displays any additional information that the conversion provides as well.

• In the Flow editor, the actor path is now displayed in the right section of the statusbar, independent of other status messages.

• The BooleanExpression and MathematicalExpression parsers now support random number generation via rand/randint (seeded and unseeded) methods.

• The ExternalFlow standalone can execute flows in headless and non-interactive modes now.

• adams-audio: The AudioPlayback sink can playback MP3 now.

• adams-net: upgraded requests4j to 0.2.0

• adams-imaging: The ImageObjectInfo transformer now has the ability to forward the bitmap stored in the LocatedObject (if any) by using BITMAP as info type.

• adams-pdf: The PDFCreate transformer can stop during PDF generation now if the flow gets stopped - useful for large PDFs.

• adams-spreadsheet:

  • The SpreadSheetSelectSubset transformer now allows double-clicking a single row to select it and close the dialog.i Also added option for multi-selection (default) or single-selection.

  • The SpreadSheetDisplay sink now offers previews for the selected rows.

Additions

• The SimpleMenu flow template generates (vertical) button-based menu using the supplied strings as captions for the buttons

• The FileProcessingWithProgressBar flow template generates a Trigger with a FileSystemSearch that updates the user on the progress of the files being processed with a ProgressBar.

• The FileBrowser sink displays a file or directory with the system's file browser.

• The FileValueDefinition plugin for the EnterManyValues source allows better control over how to select files (extensions, all files filter). The DirectoryValueDefinition allows customizing the title of the dialog.

• The HasVariableValue boolean condition is a simple way of checking a variable against a specific string value.

• The new Attached variables tab in the Flow editor allows one to see all the variables attached to the selected actor and their current values in a table.

• adams-imaging:

  • The ImageLabeler transformer can be used for image classification labeling tasks.

  • The following algorithms for generating negative regions have been added: MaxDimensions (for limiting the upper size of regions), RandomRegions (generates randomly placed and sized regions).

  • The ColorCounts BufferedImage feature generator counts the occurrences of every single color in an image and outputs them alongside the hex value of the color.

• adams-audio: The MP3ToWave sink converts an MP3 file to a WAV one.

• adams-spectral-2dim-core: Added the WaveNumberExpression filter for modifying a spectrum's wave numbers via a mathematical expression.

Over last few months, I was working mainly on image processing projects, which explains the large amount of imaging related additions and changes. One other big change was happening underneath the hood, abstracting class loading and deep copies, in order to get Weka package support working again (Weka's isolating classloader doesn't make it easy integrating Weka in other applications).

Fixes

• Added the ArrayRSquared array statistic.

• The ConfirmationDialog transformer no longer blocks the Flow editor window, i.e., it is now possible to stop the flow via the Flow editor's menu.

• The control actors CloseCallableDisplay and CallableActorScreenshot now allow selecting top-level GridView and TabView instances as well.

• Validating a flow now sets all the programmatic variables (like 'flow_dir') beforehand (if possible) to ensure correct execution.

• The GOE editor for BaseClassname now lists all classes, including non-ADAMS and abstract ones, but excluding anonymous ones.

• adams-imaging:

  • Keeping of all objects did not work properly in the ImageObjectFilter transformer, as indices could change due to filtering, resulting in an inconsistent mix of values in the report.

  • Fixed the compare with itself comparison of the DetermineOverlappingObjects transformer.

  • The logging of interactions is now properly filtered by the interaction loggers used by the ImageAnnotator.

  • fixed collecting of matches in AreaRatio and IntersectOverUnionRatio object overlap schemes: the order of objects could leave out matches as the currently highest value was used as threshold rather than the user-supplied threshold.

• adams-weka and adams-weka-lts: The R^2 now uses the same approach like R for calculating the statistic.

• adams-ml: The INTERSECT mode of MultiOutlierDetector now works correctly (used by RemoveOutliers control actor).

Changes

• The Flow control actor now has a read-only flag. When checked, it will prompt the user wether to continue with editing, just viewing or cancelling the operation altogether.

• The ConfirmationDialog now allows specifying a custom cancel token as well.

• Upgraded java-cup to 11b-20160615 and switched from exec-maven-plugin to parsergen-maven-plugin for compiling java-cup/jflex parsers.

• The MapToVariables transformer can suppress non-primitive objects getting turned into variables with the skipNonPrimitive option.

• The SetVariable transformer now allows forwarding the current variable value instead of the input token with the outputValue option. Useful when constructing file names on the fly via variable expansion.

• major class instantiation overhaul to add support for Weka packages again: adams.core.classmanager.ClassManager and the concrete CustomClassManager classes (ADAMS and Weka) are now responsible for instantiating a class from a classname and for generating deep copies of objects

• The EnterValue source now can specify how large (cols/rows) the text box is, e.g., for entering multi-line text.

• adams-imaging:

  • The CompareObjectLocations transformer now has the ability to forward screenshots of comparison to a queue and run in non-interactive mode.

  • The STARTING_WITH option for loading associated meta-data with the ImageFileReader now skips all the files that have an extension that the image reader itself manages.

  • The preview handlers ObjectLocationsFromReport and ObjectLocationsFromSpreadsheet can remove overlapping objects now as well.

  • upgraded Apache's commons-imaging to 1.0-alpha1

  • removed Apache Sanselan

• adams-spreadsheet:

  • The LookUpAdd transformer now has an optional conversion for the value being stored in the look-up table.

  • The SpreadSheetInfo transformer can output the unique values across the sheet now as well (SHEET_VALUES).

• adams-visualstats: The Histogram and ScatterDisplay sinks can export the plot data now as CSV files.

• adams-weka and adams-weka-lts: The WekaAttributeSelection transformer now reduces/transforms the test set from a WekaTrainTestSetContainer as well (if not performing cross-valication) and stores the result in the outgoing container.

• adams-net: upgraded requests4j library to 0.1.7

• adams-rsync: upgraded rysnc4j library to 3.1.3-1

• adams-ml-app: added adams-nlp module as dependency

Additions

• The DirectorySupplier source allows selecting/outputting of multiple directories, analog to the FileSupplier source.

• The FilterMap can be used for filtering Map objects using the specified filter.

• Added inspection handlers for java.util.Map and java.util.List to the Flow debugger.

• The EnterManyValues source allows users now to select enumeration values via the EnumValueDefinition class.

• adams-imaging:

  • added the following object filters: AttachMetaData (attaches single meta-data key-value pair to all located objects), MakeSquare (turns the bounding box into a square one), RemovePolygons (removes polygon information from objects, eg for enforcing bounding box display).

  • The ClipBoundingBoxes image object filter fixes bounding boxes that stick out of the image boundaries (or removes them completely if width/height are zero).

  • The TransformMetaData image object filter allows you to pass meta-data through callable actors, e.g., for cleaning up.

  • The RenameLabels image object filter allows you to quickly rename labels, e.g., for collapsing multiple categories into a single one.

  • Added CocoAnnotationsReportReader and CocoAnnotationsHandler (Preview browser), which supercede their Detectron counter parts. The report reader is optimized for speed and the Preview browser handler caches the JSON files it read (and monitors, if these should change), allowing for rapid browsing of directories with lots of images and multiple JSON files.

  • The GrayOrIndexedColorizer BufferedImage transformer and the GrayOrIndexedImageHandler allow overriding the unique colors in the image with custom ones using the supplied color provider (eg when viewing PNGs used for image segmentation).

  • The AnnotationsAndPredictions draw operations allows overlaying annotations and predictions using reports from storage, simplifying the combining.

  • added support for reading PNGs as spreadsheet (eg for reading indexed PNG files) using the PNGSpreadSheetReader.

  • The ImageContainerToSpreadSheet conversion turns any image into a spreadsheet of its (A)RGB values.

  • Added the KeepHighestMetaDataValue scheme for removing overlapping objects (eg for post-processing predictions) - used in conjunction with the RemoveOverlappingImageObjects transformer.

• adams-ml: Added the MeanAbsoluteError outlier detector (used by the RemoveOutliers control actor).

The biggest change was moving the debug view of the Flow editor into the side panel, making it less confusing when debugging multiple flows. The Preview browser can now reuse previews (and therefore keeping the setup in the view), making flicking through images much more responsive. Apart from that, a bunch of bug fixes and little improvements.

Fixes

• The Preview browser now handles errors better that occur while generating previews.

• File choosers with a filter text field now only show items that match the filter and the selected file format.

• The Debug view in the Flow editor has been integrated into the side panel of the editor window rather than being display as separate frame (this also applies to all the flow execution listeners). This makes it easier to debug several flows in parallel, as the view changes based on what flow has been selected in the editor. It is still possible to detach the view and display it in a separate frame via the right-click popup menu on the tab title.

• adams-moa: compiled a custom (unofficial) version of MOA to bundle with ADAMS to fix compatibility issue of jclasslocator library versions between MOA (0.0.12) and ADAMS (0.0.15).

• adams-weka and adams-weka-lts:

  • The WekaSpreadSheetToPredictions transformer now handles spreadsheets with string labels in the actual/predicted columns, not just when containing numeric label indices.

  • The ConfusionMatrix output generators in the Classify tab of the Weka Investigator now maintain the order of the class labels from the dataset.

  • When deserializing a session, the data tables (eg in the Preprocessing tab) get resized using optimial column widths, instead of using a width of only a few pixels.

  • Setting/updating/removing column filters on the Data tab in the Weka Investigator now gets applied to the correct column.

  • The Weka Multi-Experimenter no longer generates a NullPointerException.

Changes

• The generic search panel now allows toggling between regular expression search and simple sub-string search via its popup menu of the text field.

• Containers now make use of the more intelligent plain text rendering available through the adams.data.textrenderer.TextRenderer class hierarchy. Speeds up debugging of large objects, like spreadsheets.

• Added support for caching previews and reusing them in the Preview browser. However, reuse is currently only supported by image preview handlers.

• adams-imaging:

  • The ObjectLocationsSpreadSheetReader report reader can add polygon coordinates to the report as well (as long as X and Y coordinates are present in two separate columns).

  • The ByMetaDataNumericValue object filter now allows using x/y/width/height as key as well.

  • The left-click processors in the ImageViewer can now specify what additional keys have to be pressed (shift/alt/ctrl/meta).

  • The following content handlers in the Preview browser now support an optional alternative location for meta-data: ObjectCentersFromReport, ObjectLocationsFromReport, ObjectLocationsFromSpreadSheet.

• adams-ml: The ConfusionMatrix transformer now offers the classLabels option for enforcing the order of labels in the matrix.

• adams-weka and adams-weka-lts:

  • The WekaSpreadSheetToPredictions transformer now allows sorting of labels with a specified comparator.

  • The SimpleArffLoader now supports the sparse ARFF format as well.

• adams-weka:

  • upgraded multisearch-weka-package to 2020.2.17

  • upgraded Weka to 3.9.4-fork-0.0.1

• adams-weka-lts:

  • upgraded Weka to 3.9.0-fork-0.0.9

• adams-spectral-2dim-core: The WekaFilter post-processor can now turn off wrapping in a SpectrumFilter (on by default).

Additions

• adams-imaging:

  • Added the MergeGrid multi-image operation for combining images generated by the SubImages/Grid operation. Combines the annotations as well, but does not remove duplicate or overlapping objects.

  • Added new left-click processors for the ImageViewer: ViewObjects, DeleteObjects, AddMetaData.

  • Added report writer for object locations: ObjectLocationsSpreadSheetWriter

  • The object filter AttachOverlappingMetaData allows the copying of meta-data between overlapping objects, eg reference data. The meta-data is copied from a report in storage.

• adams-weka and adams-weka-lts: The NominalToNumeric filter allows you to either use the interal representation of the labels as numeric values or you can extract the numeric part from the label via regular expression replacement.

For a while now, I've been working on getting ADAMS running on Docker, with some of the snapshots now being deployed weekly to the public Docker hub. However, these images may not be suitable (or simply too large) and I therefore launched another little tool today called adamsflow2docker. This tool streamlines the process of deploying a worker flow (e.g., for processing data) within a Docker image.

Links to the adamsflow2docker tool and more detailed instructions on running ADAMS from within Docker are available from here.

For a long time, I have been mulling over on how to make it easier on building ADAMS applications. Most importantly, without having to write Maven configuration files and/or requiring compilation. The initial approach (on the ADAMS homepage) was to let the user select a number of modules and then generate a download with a complete Maven environment. However, this required that the user not only had Java installed, but also Maven. Not something your average data scientist would have readily available.

Enter bootstrapp, my just released Java library for bootstrapping Maven applications. With this library, you only have to supply the Maven artifacts that should make up the application. It will also generate simple start-up scripts for launching the application (Linux/Mac/Windows).

Using this generic approach under the hood, I put together a similar library for ADAMS, instant-adams. This makes it easy to put together your own, custom applications now, with only those modules and libraries that you really need.

The following command-line generates an application using the ADAMS modules adams-weka, adams-groovy and adams-excel, plus the kfGroovy Weka package:

java -jar instant-adams-0.0.1-spring-boot.jar \
  -M adams-weka,adams-groovy,adams-excel \
  -V 20.2.0-SNAPSHOT \
  -d nz.ac.waikato.cms.weka:kfGroovy:1.0.12 \
  -o ./out \
  -v -Xmx1g

In terms of ADAMS version, you can either use the ones from the daily builds (Y.M.x-SNAPSHOT, Y=2-digit year, M=month, x=patch level, usually 0) or ones from releases (e.g., 20.1.1). Be aware, only the releases are kept indefinitely, daily builds are only available for a short number of days (or number of builds).

The above command-line generates scripts which start up ADAMS with 1GB of heap size (-v -Xmx1g).

Please note, since all ADAMS artifacts are managed by one of our inhouse servers and are not available from Maven Central, the library downloads a custom Maven configuration file behind the scenes to download these artifacts through our server instead. If necessary, This can be changed by pointing to a custom Maven user settings file using the -u/--maven_user_settings option.

Feedback welcome!

The Flow editor now displays the debug view as a side panel rather than in a separate frame, which makes it less confusing when debugging several flows at the same time. ;-)

ADAMS 20.1.1 has been released. More details and download links are available from here.

Minor update post before the 20.1.1 bugfix release (which includes these changes)...

Fixes

• The enum editor in the GenericObjectEditor now allows entering multiple values as text.

• adams-addons-all: The static class discovery now works properly.

• adams-weka and adams-weka-lts: Re-enabled the residual plot plugins in the Weka Investigator.

• adams-imaging: The SelectObjects image plugin (used for annotating images) now rejects selected objects that have either width or height equal to zero (which can occur easily when using a stylus).

Changes

• Object editor: added a button with drop-down menu to the PropertySheet panel for quick access to variable actions, rather than right-clicking on label; also functions as indicator whether variable is attached by changing color.

• adams-tensorflow: switched from wai.tfrecords to wai.annotations library for generating TFRecords.

• adams-weka: upgraded Weka to 3.9.4-fork-0.0.1

• adams-weka and adams-weka-lts: The split action in the Weka Investigator has been generalized and now allows the use of any split generator available.

Additions

• adams-spectral-2dim-core: The ArrayToSpectrum conversion converts a float array of amplitudes back into a Spectrum object.

• adams-weka and adams-weka-lts: Added the Discard predictions menu item to the result history of the Classify tab in the Weka Investigator.