Updates 2018-11-09

Not really a theme this time, some libraries got upgraded to fix security vulnerabilities, a general framework for file-based dataset generation (with CNTK Faster-RCNN a concrete implementation), the Weka Investigator got two new tabs (Fast ICA, Data query) and other bits and bobs.

Fixes

  • adams-visualstats: setting x/y index in the scatter plot no longer shifts the panel (getX/getY where overriding the methods of the JComponent class)
  • adams-weka and adams-weka-lts: The WekaTrainClusterer transformer now displays the selected post-processor correctly in its quick info.

Changes

  • upgraded commons-compress dependency to 1.18 due to security vulnerability: https://nvd.nist.gov/vuln/detail/CVE-2018-11771
  • The Copy transformer now has option for specifying type of copy.
  • The StringMatcher transformer as a new flag outputIndices for outputting the 1-based indices instead of the strings.
  • The File commander now has an extra column listing the file extension, to make sorting by file type easier.
  • The Preview browser now offers a simple note-taking functionality: adding notes via popup menu in files list; clearing/viewing/saving of notes via main menu.
  • adams-pdf: upgraded pdfbox to 2.0.12+ to fix https://nvd.nist.gov/vuln/detail/CVE-2018-11797
  • adams-rsync: upgraded rsync4j to 3.1.2-13 (OpenSSH 7.9 binaries for Windows)
  • adams-weka:
    • upgraded Weka to 3.9.3-fork-0.0.2
    • upgraded matrix-algorithms to 0.0.12
    • The ClusterCenters clusterer post-processor now allows computation of median for numeric attributes and most common label for nominal ones.
  • adams-weka-lts:
    • upgraded Weka to 3.9.0-fork-0.0.7
    • upgraded matrix-algorithms to 0.0.12
    • The ClusterCenters clusterer post-processor now allows computation of median for numeric attributes and most common label for nominal ones.
  • adams-spectral-3way-core:
    • SimpleXYZReader and SimpleXYZWriter now support gzip compression
    • feature generators are now more robust when trying to access incorrect report values.
  • adams-spreadsheet:
    • SpreadSheetAddFormulaColumn now has a placeholder for the last row
    • SpreadSheetAddFormulaRow now has a placeholder for the last column
    • The expression language for LookUpUpdate now has has/remove methods for checking whether a variable is present and to remove it respectively.

Additions

  • Added a transformer for reversing array order: ArrayReverse
  • adams-imaging: transformer for cleaning up overlapping images: DeleteOverlappingImageObjects
  • adams-cntk: added preview handler for CNTK FasterRCNN ROIs: CNTKFasterRcnnRois
  • adams-weka and adams-weka-lts:
    • added experimental FastICA (independent component analysis) support to the Investigator as new tab (ICA).
    • added the WekaClustererPostProcessor transformer to allow for multiple post-processing of a container output by a WekaTrainClusterer actor.
    • added the following clusterer post-processors: AddCluster (adds the predicted cluster), ClusterCounts (records how many instances get clustered in a certain cluster).
    • added new tab to the Weka Investigator for sql-like queries: Data query
  • adams-spectral-2dim-core: added experimental FastICA support to the SpectrumExplorer.
  • adams-ml: added simple framework for preparing/generating datasets from files, using: PrepareFileBasedDataset and GenerateFileBasedDataset.
  • adams-cntk: added file-based dataset generator plugin CNTKFasterRCNN.
  • adams-spectral-3way-core:
    • added Shift filter, which allows shifting of x, y or z values.
    • Upgraded multiway-algorithms dependency to 0.0.8

Updates 2018/10/19

Quite a bit of work underneath the hood has happened, UI improvements relating to the Flow editor, improved previewing for spectral data types and lots of little changes and fixes.

Fixes

  • The ListStructureModifyingActors actor processor now skips disabled actors (used when debugging).
  • GOE editors that handle actor references like callable actors now skip any actors that are disabled.
  • Rearranged the Flow editor tree popup menu a bit to streamline usage: combined external actor operations; moved less commonly used operations down.
  • Added support for Flow editor tree popup menu extensions, ie menu items that get inserted after/before an existing menu item (Tree.PopupMenuExtensions in FlowEditor.props settings file).
  • The FileInfoHandler of the Preview Browser switched to using java.nio.file.Files.XXX methods (due to bug in java.io.File.XXX methods, see https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6203387); added owner, posix permissions, regular file, symbolic link to the output.
  • Recent refactoring of the generic plot panel and its axes, resulted in invalid formattters getting assigned. This resulted in the Memory monitor x axis showing the HH:mm plus the Java Epoch.
  • BOMs (byte order marks; used by UTF variants) are now handled properly by the CsvSpreadSheetReader, the generic loading of text files and the various TextFileReader schemes.
  • adams-imaging:
    • Annotating objects is no longer generating negative heights or widths when not starting selection rectangle in top-right corner.
    • The Objects sub-image algorithm now ensures that the sub-image dimensions do not exceed the original image.
  • adams-ml: The ConfusionMatrix transformer now handles unclassified predictions (aka missing values).
  • adams-spreadsheet:
    • SpreadSheetDisplay now adheres to the useOptimalColumnWidth flag.
    • Spreadsheet file viewer: now hides rows with missing values when applying filters; Copy column now copies only the visible cell values instead of all.
    • The SpreadSheetMerge transformer now removes duplicate unique columns correctly (failed to remove correct ones if no prefix/ID was used), when using the keepOnlySingleUniqueID flag.
  • adams-weka and adams-weka-lts:
    • WekaPredictionsToSpreadSheet (also used in generating the confusion matrix view in the Investigator) now ensures that labels are treated as strings now, to avoid numeric conversions for labels like '00' or '07'.
    • WekaPredictionsToSpreadSheet, WekaPredictionsToInstances and the Investigator now correctly align the original instances with the predictions of cross-validation when using a grouped split generator.
  • adams-spectral-2dim-core: The OpusSpectrumReaderExt now applies the scaling factor for Y defined in the file when reading the amplitudes.

Changes

  • Switched from Swing widgets to ADAMS ones: JButton, JCheckBox, JComboBox, JTextArea, JTextField, JTextPane. This makes it easier to introduce additional functionality in the top-level class.
  • The Command and Exec sources and the Exec sink now support setting of custom environment variables.
  • Added option for conversion to be applied to values, to have better control over types for hashset-handlers: HashSetInit (standalone/transformer), HashSetAdd (transformer), HashSet (boolean condition).
  • The parsers for BooleanExpression, MathematicalExpression and StringExpression now allow expression variables (not to be confused with flow variables) to be strings consisting of alphanumeric chars and "_" (but must start with uppercase letter to distinguish them from built-in functions), as well as enclosing via single quotes instead of just "[]".
  • Report tables are now using a case-insensitive sort on the field names.
  • The NewReport source now uses BaseClassname instead of String for the report classname.
  • BaseClassname now has its own GOE editor, which allows you to search for classes.
  • The GenericArrayEditor now allows the sorting (asc/desc) of the elements.
  • The SetContainerValue control actor now allows storage of non-standard, additional meta-data via the force flag.
  • adams-rats: The Exec input and output now support setting of custom environment variables.
  • adams-spreadsheet: The LookUpUpdate parser now allows enclosing of expression variables in "[]" as well, aligning it with the other parsers.
  • adams-spectral-3way-core: upgraded multiway-algorithms dependency to 0.0.7.
  • adams-spectral-2dim-core:
    • Added -lenient flag to the InstanceGenerator transformer to only log errors instead of returning them (and possibly stopping the flow).
    • The spectrum preview in the Preview browser and the object renderer when debugging a flow now display spectrum and sample data via a tabbed pane, rather than just the spectrum.
  • adams-compress: upgraded unrar dependency to 3.0.0
  • adams-net:
    • removed obsolete org.apache.httpcomponents/httpclient dependency
    • upgraded tika-core dependency to 1.19.1

Additions

  • Added the PullUpActor tree editor popup menu and keyboard action, which moves all sub-actors of a mutable actor handler (like Tee or Trigger) up one level and removes the enclosing actor handler.
  • Added little test dialog for date formatting/parsing in Developer user mode in the Maintenance menu.
  • adams-ml: added basic filter framework for spreadsheet-based datasets, SavitzkyGolay, PLS, MultiStream, MultiBatch.
  • adams-weka and adams-weka-lts: added OuterProductAnalysis filter (Fabricio S.Terra, Raphael A.Viscarra Rossel, Jose A.M.Dematte (2019). Spectral fusion by Outer Product Analysis (OPA) to improve predictions of soil organic C. Geoderma. 335:35-46. URL https://doi.org/10.1016/j.geoderma.2018.08.005.).
  • adams-spectral-2dim-core:
    • Added new AbstractMultiSpectrumFilter schemes: Add and Subtract.
    • Source for listing sample IDs of sample data with no associated spectra: OrphanedSampleDataIdSupplier
    • Added handlers in the Preview browser for spectrum formats: ASCIIXY, CML, DPT, JCampDX, JCampDX2, Json, MPS, Opus, OpusExt, Relab, SPA, SPC, SpecLib.

Updates 2018/09/16

It has been a while, I know. Been visiting clients and preparing to go live with some large ADAMS-based systems. On the data transformation side of things, there are a few more PLS variants available. For image processing, there is now better support for VIA/VGG annotations and data available. The experimental CNTK Java integration has been removed, too hard to maintain as it is tied to a specific CNTK version.

Fixes

  • The Flow editor no longer saves expanded external flows.

Changes

  • The Copy (plain text) operation in the Flow editor tree popup menu now uses the compact format as well.
  • The Flow control actor now allows to specify a restart manager. This allows the monitoring of flow(s)/setup file(s) by a flow that runs as a background service (daemon or Windows service), triggering its restart.
  • Added a GOE editor for the BaseInterval class.
  • adams-rsync: updated the rsync4j library to 3.1.12-12
  • adams-weka: upgraded multisearch library to 2018.8.16
  • adams-weka and adams-lts:
    • upgraded matrix-algorithms to 0.0.11
    • the filter PartitionedMultiFilter2 now allows specification of prefixes for the filters rather than just using 'filtered'
  • adams-imaging:
    • MergeObjectLocations can now just merge all locations when using minOverlapRatio=0.0 and overlapAction=KEEP
    • ViaAnnotations transformer has been removed.
  • adams-cntk: removed the experimental CNTK Java bindings, as they too hard to maintain and strictly limited to a specific release of CNTK.
  • adams-cntk-weka and adams-cntk-weka-lts: removed any code that relied on the experimental CNTK Java bindings.

Additions

  • Added menu item Copy actor path to the Flow editor's popup menu.
  • Added class hierarchy for conversions that swap out objects and concrete SwapObjects implementation for replacing one object with another, using their command-lines.
  • adams-meta:
    • added actor wrappers for auto-generated actors, which can get removed using the RemoveAutogeneratedActors processor: AutogeneratedStandalone, AutogeneratedSource, AutogeneratedTransformer, AutogeneratedSink
    • added actor wrappers for inactive actors (actors that don't get executed, but kept for reference), which can get activated again using the ReactivateActors processor: InactiveStandalone, InactiveSource, InactiveTransformer, InactiveSink
  • adams-net:
    • added MarkdownToHTML conversion for converting Markdown string into HTML string
  • adams-weka and adams-lts:
    • added more PLS algorithms: DIPLS, PRM, VCPLS
    • added SwapPLS conversion for replacing PLS algorithm (with optional migration of number of components)
  • adams-imaging:
  • adams-spectral-2dim-core: added the RowWiseSpreadSheetSampleDataReader sample data reader.

Updates 2018/07/26

The last few weeks I have been busy preparing the migration of a large, commercial ADAMS-based application for processing spectral data to a newer version. This new version uses Rats instead of cronjobs to improve processing speed of data and making predictions. Due to these flows now containing several 10,000 actors, a lot of speed improvements were necessary to make them easier to view and edit in the Flow editor. That introduced also a number of bugs (a lot of rewriting happened behind the scenes), but it has stabilized a lot now.

Fixes

  • The undo is now fixed in the Flow editor, with the flow no longer disappearing when performing an undo operation. Due to changes of the underlying data structure introduced the last couple of weeks, for speeding up the read/write of flows from/to disk.
  • Using now nested lists for copying/pasting actors on the clipboard for better performance, also using a faster check whether nodes are on clipboard for pasting.
  • The popup menu in the GOE failed on java.io.File objects due to a wrong cast to PlaceholderFile, which got fixed.
  • The GOE could end in an endless loop, firing property events that hammered the Swing thread, resulting in a lock up of the user interface. Removing an unnecessary call for firing property events, the lock up disappeared.
  • Reading of flows in the old nested format now works again.
  • File choosers that allow the editing of options of the selected file type (reader/writer) now ensure that the selected filter and the last handler used are of the same class when showing the dialog, resetting the last handler if that's not the case. Otherwise, cases can occur where the selected filter is not presented in the GenericObjectEditor.
  • adams-weka and adams-weka-lts:
    • The WekaFilter transformer no longer re-initializes the filter if loading from disk/actor/storage and initializeOnce is checked.
    • The CrossValidationFoldGenerator classes now report back the actual number of folds in use (eg when using -1 for LOO, this depends on the data), which in turn makes the WekaCrossValidationExecution work properly when using GroupedCrossValidationFoldGenerator in conjunction with LOO.
    • search methods that implement RankedOutputSearch (eg PrincipalComponents) now output the attribute importance from the ranking instead of the attribute index
  • adams-rats:
    • The Rat actor now disables the performLazySetup option if the sub-flow contains at least one Breakpoint actor (debugging doesn't work in conjunction with lazy setup).
    • The Subscribe rat input now ignores any notifications if the Rat actor is paused.
    • The RatControl actor how has optional support for bulk actions, ie checked Rat actors can be paused/resumed/stopped/started.
  • adams-spectral-2dim-core: fixed selection of instance generator when exporting visible spectra in the Spectrum Explorer: it is now possible to select from a class hierarchy

Changes

  • Introduced the adams.flow.processor.ActorProcessor interface, replacing the usage of the AbstractActorProcessor class.
  • The DeserializeToStorage standalone now accepts pairs of storage name / file name to reduce the number of actors necessary when loading lots of models. Storage name and/or file name can be variables as well.
  • The Flow editor now prompts whether to perform checks before saving a when it is considered large (default is 200 actors); occurs on a per-flow basis and can be reverted by simply checking/unchecking the checkbox in the menu.
  • Actors that implement the AtomicExecution interface (eg Tee or Branch) now allow the user to specify a timeout after which the sub-flow gets stopped nonetheless (when finishedBeforeStopping is set).
  • Upgraded lanterna to 3.0.1 (for providing an ncurses-like terminal interface).
  • adams-spreadsheet: The SpreadSheetSetCell transformer can use a cell finder now to locate the cells to update with the specified value as well (eg for updating all cells with missing values or a value less than a certain minimum).
  • adams-weka and adams-weka-lts: switched to updated Weka versions (3.9.2-fork-0.0.2 and 3.9.0-fork-0.0.5 respectively) that include patch for MyHeap usage in weka.core.neighboursearch.LinearNNSearch, avoiding the the number of elements cannot exceed the initially set maximum limit exception.

Additions

  • added Clear clipboard to menu of Flow editor
  • added MapToReport conversion
  • Added the ListObjectCommandlines actor processor for listing object command-lines of a specific class.
  • Added actor processor for updating the number of threads used by classes: ChangeNumThreads
  • adams-spreadsheet:
    • The SpreadSheetCellSelector transformer lets you select cells in a spreadsheet to forward in the flow.
    • The SpreadSheetRandomSystematicSample transformer picks a random systematic sample of rows from the incoming spreadsheet.
    • added Clear clipboard to menu of Spreadsheet file viewer
  • adams-weka and adams-weka-lts:
    • The PredictionTrend output generator for the classification tab of the Weka Investigator plots actual and predicted values in a single plot, sorted by actual. This allows spotting of trends.
    • Added NIPALS and SparsePLS
  • adams-meta: The ProcessorActor transformer allows you to process an incoming flow with a specific actor processor (eg listing variables or command-lines).
  • adams-spectral-2dim-core: added instance generator based solely on report fields and with dedicated class field: FieldInstanceGeneratorWithClass

Updates 2018/06/26

Finished some major refactoring related to loading and saving flows. Well, actually replacing an ugly hack from two years ago with a clean implementation. Properly decoupling user interface and readers/writers for flows. That and some further optimizations in the Flow editor resulted in nice speed improvements: a flow with 38,000 actors now loads in about 3 seconds on my laptop. BTW this is the largest flow that I have generated so far!

Further CPU profiling also revealed some other code locations which could be optimized, improving startup time of flows.

Fixes

  • The Preview browser now properly updates the display when selecting a different handler for serialized object.
  • SetManyVariables standalone and transformer now use a list internally, which improves performance of the addVariablePair method when setting hundreds of variables.
  • User interface improvements in the Flow editor that relate to long running background tasks, they no longer block the interface.
  • General Flow editor improvements to speed things up with flows that have 10s of thousands of actors.
  • The Switch and IfThenElse actor now clean up the current token once execution finishes, to avoid unnecessary memory usage.

Changes

  • The Image viewer now uses a multi-page instead of multi-tab interface as well.
  • Copy/paste of flow nodes now uses a custom DataFlavour on the clipboard to speed up checks whether pasting is available; added "Copy (plain text)" for old approach.
  • Switched to sizeofag 1.0.4. This enabled the Inspect memory sub menu in the Flow editor, displaying the memory consumption of the selected actor (detailed per class or just overall size).
  • An alternative visualvm executable can be specified now by the user through preferences (or the JVisualVM.props properties file). JVisualVM got dropped from JDKs starting with Java 9. The project is now maintained separately and you can download it from here: https://visualvm.github.io/
  • adams-rsync: upgraded rsync4j to 3.1.2-10.
  • adams-rats: a Rat actor can free up memory now with the wrapUpAfterExecution flag, which is available in conjunction with performLazySetup.
  • adams-weka and adams-weka-lts: upgraded to matrix-algorithms 0.0.4.
  • adams-webservice: upgraded Apache CXF to 3.2.5.

Additions

  • Added simple source actor for just keep the flow running: Idle. This is useful when using an event-based flow (eg using Rat actors) rather than a data driven one.
  • The ForceFlush control actor enforces actors implementing FlushSupporter (currently DumpFile and WekaInstanceDumper) to flush their buffers.
  • With the ArrayToChunks transformer you can split an array into sub-arrays of a specified length.
  • adams-weka and adams-weka-lts: meta-classifier that can use a fallback model for making predictions: Fallback
  • adams-rats: the ExecuteRats control actor allows execution of Rat actors in MANUAL mode whenever a token passes through.
  • adams-compress: added support for Zstd de/compression library (https://en.wikipedia.org/wiki/Zstd).
  • adams-spectral-2dim-core: added the MaxWaves outlier detector for enforcing an upper limit on wave numbers.
  • adams-spectral-3way-core: added support for feature generation through the ThreeWayDataFeatureGenerator transformer and its associated class hierarchy of feature generators.

Updates 2018/05/25

A lot of work went into the 3-way data module, as well as proper support for polygons in the image annotation framework. The audio module also got a bit of love.

Furthermore, some user interfaces got switched from multi-tab to multi-page ones, freeing up vertical screen real estate (most people of have wides-screen monitors!).

Fixes

  • External actors that use a file monitor can once again use variables in the file name for the external flow.
  • The flow panel and flow tree now get focused better, to enable a smoother keyboard interaction.
  • adams-imaging: The ImageAnnotator now uses a scrollpane to display more buttons than fit on screen.
  • adams-weka and adams-weka-lts: the MultiSearch meta-classifier now allows you to select the search algorithm for the parameter space.

Changes

  • CsvSpreadSheetWriter now allows an empty quote character, which will suppress any quoting.
  • Switched the following applications from multi-tab interface to a multi-page one in order to manage more tabs/pages at the same time without decreasing the vertical screen real estate (these pages can be detached/reattached):
    • Flow editor
    • adams-imaging: Image processor
    • adams-heatmap: Heatmap viewer
    • adams-spreadsheet: Spreadsheet file viewer

Additions

  • The implicit variable flow_filename_restore contains a file name based on the full file name of the flow, but with the .props extension. To be used for interactive actors like EnterManyValues to restore the entered values.
  • Added Favorite actor (here, after, beneath) keyboard actions (Ctrl+E, Ctrl+Alt+E, Ctrl+Shift+E).
  • adams-spectral-2dim-core: added meta spectrum reader that reads spectra from zip files: ZippedSpectrumReader.
  • adams-spectral-3way-core:
    • added heatmap viewer for 3-day data, like 3D fluorescent spectra (EEM)
    • added experimental support for building and applying the multi-way algorithms available from: https://github.com/waikato-datamining/multiway-algorithms
    • added lots of conversion to and fro tensors, 3-way data and spreadsheets
  • adams-audio:
    • With the AudioRecorder source it is possible to record audio from a microphone.
    • The WaveToAmplitudes conversion extracts amplitudes from Wave object, absolute or normalized
    • Added Resample wave filter for using different sample rate.
    • Added Cut wave filter for cutting out a portion of a wave.
    • Added sink for playing back audio files: AudioPlayback
  • adams-compress: added support for xz compression via Xz/UnXz transformers and XzTextFileReader.
  • adams-imaging:
    • Added the Grayscale buffered image transformer with customizable lumiance parameters via a class hierarchy (supporting BT 601, 709 and 2020).
    • The Rotate object filter rotates objects by the specified angle.
    • ViaAnnotations transformer generates annotations for VIA (VGG Image Annotator - http://www.robots.ox.ac.uk/~vgg/software/via/)
    • Added support for polygon annotations via the PolygonPainter (default is RectanglePainter).

Updates 2018/04/20

Spent a lot of time trying to get things working with Java 9. The usual backwards compatibility mantra got a serious beating with Java 9. ADAMS now compiles with Java 9, but I still recommend to stick with Java 8 for a while.

Futhermore, there is now a separate module for ffmpeg called adams-ffmpeg, rather than being tucked away in the adams-video module. In addition to that, I added an experimental module for audio processing called adams-audio.

Fixes

  • Now compiles with Java 9.
  • Auto-registering of flows now registers after the flow ID has been initialized rather than with 0.
  • adams-imaging:
    • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now use the user-specified font.
    • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now use the user-specified font.

Changes

  • Upgraded sizeofag to 1.0.2.

  • Upgraded jclasslocator to 0.0.10.

  • Removed archetype module.

  • Added keyboard shortcuts to the Flow editor for bringing up the actor tree to search for an actor to insert, skips the GOE dialog: Ctrl+J (here), Ctrl+Alt+J (after), Ctrl+Shift+J (beneath)

  • adams-spectral-2dim-core: The Evaluator transformer now allows to turn off

    cloning of container storing the evaluation results.

  • adams-webservices: updated Apache CXF to 3.2.4 to make it compile with Java 9 (10 doesn't work yet, but the upcoming 3.2.5 release has a fix for that).

  • adams-meka: Upgraded to MEKA 1.9.2.

  • adams-imaging:

    • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now offer x/y offsets for the label.
    • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now offer x/y offsets for the label.
    • ImageObjectInfo now allows to output the INDEX_STRING and INDEX_INT as well.

Additions

  • Added support for publish/subscribe: InitPublishSubscribe standalone for initializing pub/sub data structure in storage and the Publish sink for broadcasting the data to all the subscribers of the specified pub/sub data structure.
  • Added platform specific file-in-use checks: Windows and NonWindows (Default combines both).
  • Added the JsonObjectToMap conversion to turn a JSON object into a simple map.
  • Added the ExtractSimpleContainerContent conversion to retrieve the content of simple containers.
  • Added the Go to menu item to the Flow editor popup menu, which allows you to jump to the reference actor (eg a callable actor).
  • adams-rats:
    • added support for using sockets for sending and receiving data, aptly named Socket for input and output (adams-rats-sockets.flow).
    • added support for publish/subscribe with Subscribe rat input and Publish rat output. Simpler and more efficient than queues for notifying multiple Rat actors at once.
    • added Django web application for remote control of Rat actors that are available through the RatControl actor, using a REST webservice for communication.
  • adams-weka: The SocketFacade meta-classifier communicates via sockets with a prediction scheme, can be used to interface with any programming language as long as communication occurs via sockets.
  • adams-imaging:
    • added BasicStatistics feature generator (for buffered images).
  • adams-audio: new module for some basic audio processing of WAV files, generating spectrograms (as image or tabular data).
  • added separate adams-ffmpeg module, to make ffmpeg functionality available in adams-audio and adams-video.
  • adams-ml: added basic framework for calculating summary statistics with the SummaryStatistics transformer.
  • adams-spectral-2dim-core: added the SimpleSpectralInstanceGenerator instances generator, which does not output a class attribute in the generated Weka Instances object.

Updates 2018-03-26

Ran out of time on Friday to post this update, so here it goes...

Finally managed to sort my Java2D rendering issues (basically blocking my X11 session when displaying images with Java), by switching to OpenGL rendering by supplying this option to the JVM: -Dsun.java2d.opengl=true

The following modules were removed:

  • adams-imaging-imagej
  • adams-dl4j
  • adams-dl4j-insight
  • adams-dl4j-weka

Why remove deeplearning4j? Well, we just didn't have much luck with that framework and CNTK (despite being in Python) was much easier to work with (and the API didn't change all the time).

Fixes

  • The DumpFile sink no longer attempts to write to disk if the output file is pointing to directory (eg due to an invalid variable)
  • PlaceholderFile/Directory: getAbsolutePath() now checks whether the path is a Windows one (X:...) and if that is that case, just returns it (otherwise the CWD gets prepended).
  • The following overlay paintlets now always use the panel's container manager: MeanOverlayPaintlet, MedianOverlayPaintlet, StdDevOverlayPaintlet, LimitPaintlet
  • When enclosing a referenced actor (eg below CallableActors) in the Flow editor, it now triggers the updating of the reference throughout the flow.
  • Fixed how the bin directory of JRE/JDK is determined, now works with Java 9+.
  • adams-weka and adams-weka-lts:
    • The Export output... functionality in the Weka Investigator no longer exports model files from the predictions or model tabs, but spreadsheets and text files.
    • The search panel of the WekaInstancesDisplay now shows up when used in conjunction with the DisplayPanelManager.
  • adams-spectral-2dim-core:
    • The RowWiseSpreadSheetSpectrumReader and ColumnWiseSpreadSheetReader are now more robust when parsing numeric values in the spreadsheet, in case there is incorrect data present. They can also process multiple sheets, in case the spreadsheet reader supports that.

Changes

  • When saving a flow, the modules that were present at that time will get stored in the comments. When loading a flow with the modules listed, these modules are then checked against the current environment now, outputting a warning if modules missing.
  • When editing an actor in the Flow editor, the children no longer get discarded when both actors, new and old, implement MutableActorHandler (like Tee and Trigger).
  • The gradient color generators BiColorGenerator and MultiColorGenerator now support a custom alpha value to obtain transparent colors.
  • adams-weka-lts: now uses 3.9.0-fork-0.0.4, which had a few classes backported from 3.9.2 in order to make it compile and work under Java 9.
  • adams-spreadsheet: moved cell rendering customization (ie neg/pos background colors) into CellRenderingCustomizer class hierarchy.
  • adams-imaging: The ImageObjectOverlay can store meta-data values in the outgoing report of the objects with the highest overlaps as well, e.g., associated labels.
  • adams-ml: The ConfusionMatrix transformer can take a column with probabilities now into account to generate the counts. Furthermore, instead of counts, it can also generate overall percentages or row percentages now.

Additions

  • The StringArraySplit transformer splits string array into sub-arrays using the specified regular expression. E.g., can be used to split log file into individual entries using the timestamp as split criterion.
  • Added a color based on the X11 color palette (https://en.wikipedia.org/wiki/X11_color_names), called X11ColorProvider.
  • With the PercentilePaintlet it is possible to plot a horizontal line at the specified percentile for each sequence.
  • adams-imaging: added the Objects sub-images generator, which uses the specified object finder to locate the sub-images of interest and then extracts them (available through the SubImages BufferedImageTransformer).
  • adams-spreadsheet:
    • added the Percentile row/column statistic.
    • added the NumericRangeCellRenderingCustomizer for highlighting cells in specified columns that fall in defined intervals.
  • adams-ml: added the ConfsionMatrixCellRenderingCustomizer to make confusion matrix spreadsheets more readable (as generated by the ConfusionMatrix transformer).
  • adams-weka and adams-weka-lts:
    • Investigator, Multi-Experimenter, WekaRandomSplit, WekaCrossValidationSplit and WekaCrossValidationEvaluator now allow you to select the type of split generator you want to use. You can either use Weka's default splitting mechanism (Default...) or you can use the new Grouped... ones. The latter one is useful when dealing with data that contains duplicates or near-duplicates that can be grouped via a regular expression applied to a nominal or string attribute. An example is scanning the same sample with multiple instruments. A sample should never be present in train and test, as it gives a way to optimistic estimate (in other words: you are sort of cheating).
    • The Investigator now offers confusion matrix output in the classify tab, which offers various display and highlighting options.
  • adams-python:
    • Added new module to adams-base, to make interaction with Python easier.
    • PythonEnvironment standalone allows you to specify python/pip commands to use (eg from a virtual environment).
    • The PythonExec source executes a Python script and forwards the output that the script generates.
    • With PipExec you can run pip commnands and post-process the output the command generates.
    • The PythonExpandScript transformer expands variables and placholders in a script template and saves it to a new file.

Updates 2018/03/02

A few changes in terms of modules occurred: all -stable modules have been renamed to -lts (long term support) and moved to the new adams-lts repository (https://github.com/waikato-datamining/adams-lts). Furthermore, experimental support for the MongoDB NoSQL database has been added.

Fixes

  • The CheckActorReferenceUsage flow processor now skips actors that are disabled, in other words, a disabled callable actor won't get moaned about if not used.
  • Added missing setting of variable BASEDIR in the following batch files: daemon.bat, exec.bat, logging.bat
  • Updating of properties, e.g., with the UpdateProperty transformer, now handles arrays properly: the value string is interpreted as blank-separated list and the array constructed from those elements.
  • The dialog size of showInputDialog method (e.g., when renaming an actor) is now adjusting a bit more to the input.
  • The flow used by System performance (from the Help menu) is now working again properly.
  • adams-weka and adams-weka-lts: introduced custom aggregation of Evaluation objects (Relative absolute error and Root relative squared error will differ a bit due to different handling of training priors - still buggy in Weka ATM), affecting WekaAggregateEvaluations, WekaCrossValidationExecution, RemoveOutliers and WekaSpreadSheetToPredictions.
  • adams-rats: The FileLister rat input now empties the queue when returning an array.
  • adams-spreadsheet: The parser for the expression used by SpreadSheetQuery handles negative numbers now correctly.

Changes

  • Updated the section in the manual on how to install an ADAMS flow as daemon (Linux, systemd) and service (Windows, NSSM).
  • Added methods 'replaceext' (replacing file extension) and 'ext' (extracting file extension) to parsers BooleanExpression, MathematicalExpression and StringExpression.
  • Added help menu items in the Flow editor for: variables, boolean/math/string expressions
  • Combined help display of GenericObjectEditor and Class help.
  • adams-spreadsheet: The formula of conversions SpreadSheetAddFormulaColumn and SpreadSheetAddFormulaRow can contain variables now that get expanded before inserting it into the spreadsheet.
  • adams-rats: The ChangeRatState control actor no longer checks whether the specified Rat actors are present if a variable is attached to the "rats" property (the one listing the names).
  • adams-weka and adams-weka-lts:
    • added mxexpression-weka-package dependency, offering filter and classifier that use mathematical expression to update target attribute or make prediction, using the mXparser library (https://sourceforge.net/projects/mxparser/).
    • Class help in the GUI is now in HTML; help in GenericObjectEditor is now showing full help.
    • The WekaCrossValidationEvaluator now has an option (finalModel) to build a model on the full dataset. Automatically implies that a container is being output.

Additions

  • The GroupByRegExp transformer allows generating groups of strings from array using a regular expression and a grouping expression, e.g., "(.*)-RGB.*" as regexp and "$1" for grouping them.
  • Added some of Eric Eaton's Java utility classes located at: http://www.seas.upenn.edu/~eeaton/software/Utils/Utils.zip
  • Added conversions MathExpression and StringExpression for better efficiency when processing spreadsheets (using SpreadSheetConvertCells instead of SpreadSheetTransformCells).
  • The File monitor tool allows you to monitor files that get appended, like log files, outputting all the new data added to the files.
  • adams-spectral-2dim-core:
    • The NewSpectrum source allows the creation of a new spectrum.
    • With GetSpectrumAmplitude the amplitude of the specified wave number can be retrieved.
    • SetSpectrumAmplitude allows you to set the amplitude of the specified wave number or simply add a new spetrum point.
  • Added module adams-mongodb for basic MongoDB support.
  • adams-weka and adams-weka-lts: added ability to upload currently best setup in classifier-based genetic algorithms via a setup upload scheme, e.g., in a MySQL database.

Updates 2018/02/12

Apart from some bugfixes, the biggest change is the addition of the adams-rest module. This module allows you to quickly create REST webservices, a simpler alternative to SOAP-based ones. Using the GenericServer, you can simply slot together any functionality that you want, using classes that implement the adams.flow.rest.RESTPlugin interface. See below for more details on what REST functionality is already available.

Fixes

  • adams-spreadsheet: Fixed handling of null values for date/time columns in the SpreadSheetDbReader source.
  • Improved positioning of dialogs/frames on screen, leaving (by default) 50 pixels at the bottom of the screen reserved (GUIHelper.props, property ScreenBorder.Bottom). This avoids dialogs like the GenericObjectEditor to appear behind the taskbar on Windows.
  • adams-rats: The FileLister rat input had a bug when adding the moved files to its output list when skipping files that were in use, generating an IndexOutOfBoundsException.

Changes

  • The expressions in the debug panel of the Flow editor now allow string expressions as well, rather than just simple strings with variables.
  • Added ability to delete via regular expression: DeleteVariable, DeleteStorageValue
  • The StringCut transformer now uses Index instead of int for the character positions, allowing the use of last.
  • adams-imaging: The conversions *RectangleToString and StringToRectangle can parse the x0y0x1y1 format now as well (default is xywh).
  • adams-net: The HttpRequest transformer now accepts byte arrays as well as input.
  • adams-timeseries: Updated timeseriesForecasting dependency to 1.0.25

Additions

  • SwapVariables allows swapping of values of two variables.
  • Run to here was added to the Flow editor menu, which can be used while debugging a flow and you don't want to step through all the actors.
  • Additional boolean conditions for tokens: IsBoolean, IsByte, IsShort, IsInteger, IsLong, IsFloat, IsDouble
  • Added JSON support for reports: conversions JsonToReport and ReportToJson, DefaultSimpleJsonReportReader and DefaultSimpleJsonReportWriter.
  • Added the PathSplit conversion for splitting path into its individual parts: drive letter or server from UCN path (if applicable), directories, file name.
  • adams-imaging:
    • Added modified version of CountColor, which performs linescan from left/right for counting and stops as soon as it hits another color (used for objects located in center of image): CountColorOutside
    • Added the LocatedObjectToRectangle conversion, which turns the position of the object into a BaseRectangle object.
    • The GetImageObjectMetaData transformer retrieves the meta-data map of a located object.
  • The SimpleStringReplace transformer uses simple String.replace(..., ...) for replacing sub-strings. Gets around problems with trying to specify the regular expression of StringReplace when trying to only do a simple string replacement.
  • adams-net:
    • Added URLEncode and URLDecode conversions for creating URL-conform strings and turning them back into regular strings (e.g., "hello world" into "hello+world").
    • Added StringArrayToURLParameters and URLParametersToStringArray conversions to turn string array of key-value pairs and a string that can be appended to a URL (and vice versa).
  • Added adams-rest module, for providing REST webservice support.
  • adams-rats:
    • Added generic REST support via the adams.flow.rest.RatsServer REST server (same as GenericServer, but aware of the Rat actor it belongs to).
    • Generic (text) upload capability using the adams.flow.rest.text.RatsTextUpload REST plugin.
    • Added support for controlling Rat actors (accessible via RatControl/RegisterFlow) via REST service: control.RatControl
  • adams-spectral-2dim-core:
    • Added conversions to turn spectrum into JSON string and vice versa: SpectrumToJson, JsonToSpectrum.
    • Added JSON support for reports: DefaultSimpleJsonSampleDataReader and DefaultSimpleJsonSampleDataWriter.
  • adams-spectral-2dim-webservice:
    • Added REST webservice plugins for spectra: DeleteSpectrum (delete from DB), GetSpectrum (load from DB), PutSpectrum (store in DB), TransformSpectrum (transform spectrum using specific filter)
  • adams-spreadsheet: With the SpreadSheetSubsetFromGroup transformer it is possible to retrieve specific rows from groups within spreadsheet.