Updates 2018/09/16

It has been a while, I know. Been visiting clients and preparing to go live with some large ADAMS-based systems. On the data transformation side of things, there are a few more PLS variants available. For image processing, there is now better support for VIA/VGG annotations and data available. The experimental CNTK Java integration has been removed, too hard to maintain as it is tied to a specific CNTK version.

Fixes

  • The Flow editor no longer saves expanded external flows.

Changes

  • The Copy (plain text) operation in the Flow editor tree popup menu now uses the compact format as well.
  • The Flow control actor now allows to specify a restart manager. This allows the monitoring of flow(s)/setup file(s) by a flow that runs as a background service (daemon or Windows service), triggering its restart.
  • Added a GOE editor for the BaseInterval class.
  • adams-rsync: updated the rsync4j library to 3.1.12-12
  • adams-weka: upgraded multisearch library to 2018.8.16
  • adams-weka and adams-lts:
    • upgraded matrix-algorithms to 0.0.11
    • the filter PartitionedMultiFilter2 now allows specification of prefixes for the filters rather than just using 'filtered'
  • adams-imaging:
    • MergeObjectLocations can now just merge all locations when using minOverlapRatio=0.0 and overlapAction=KEEP
    • ViaAnnotations transformer has been removed.
  • adams-cntk: removed the experimental CNTK Java bindings, as they too hard to maintain and strictly limited to a specific release of CNTK.
  • adams-cntk-weka and adams-cntk-weka-lts: removed any code that relied on the experimental CNTK Java bindings.

Additions

  • Added menu item Copy actor path to the Flow editor's popup menu.
  • Added class hierarchy for conversions that swap out objects and concrete SwapObjects implementation for replacing one object with another, using their command-lines.
  • adams-meta:
    • added actor wrappers for auto-generated actors, which can get removed using the RemoveAutogeneratedActors processor: AutogeneratedStandalone, AutogeneratedSource, AutogeneratedTransformer, AutogeneratedSink
    • added actor wrappers for inactive actors (actors that don't get executed, but kept for reference), which can get activated again using the ReactivateActors processor: InactiveStandalone, InactiveSource, InactiveTransformer, InactiveSink
  • adams-net:
    • added MarkdownToHTML conversion for converting Markdown string into HTML string
  • adams-weka and adams-lts:
    • added more PLS algorithms: DIPLS, PRM, VCPLS
    • added SwapPLS conversion for replacing PLS algorithm (with optional migration of number of components)
  • adams-imaging:
  • adams-spectral-2dim-core: added the RowWiseSpreadSheetSampleDataReader sample data reader.

Updates 2018/07/26

The last few weeks I have been busy preparing the migration of a large, commercial ADAMS-based application for processing spectral data to a newer version. This new version uses Rats instead of cronjobs to improve processing speed of data and making predictions. Due to these flows now containing several 10,000 actors, a lot of speed improvements were necessary to make them easier to view and edit in the Flow editor. That introduced also a number of bugs (a lot of rewriting happened behind the scenes), but it has stabilized a lot now.

Fixes

  • The undo is now fixed in the Flow editor, with the flow no longer disappearing when performing an undo operation. Due to changes of the underlying data structure introduced the last couple of weeks, for speeding up the read/write of flows from/to disk.
  • Using now nested lists for copying/pasting actors on the clipboard for better performance, also using a faster check whether nodes are on clipboard for pasting.
  • The popup menu in the GOE failed on java.io.File objects due to a wrong cast to PlaceholderFile, which got fixed.
  • The GOE could end in an endless loop, firing property events that hammered the Swing thread, resulting in a lock up of the user interface. Removing an unnecessary call for firing property events, the lock up disappeared.
  • Reading of flows in the old nested format now works again.
  • File choosers that allow the editing of options of the selected file type (reader/writer) now ensure that the selected filter and the last handler used are of the same class when showing the dialog, resetting the last handler if that's not the case. Otherwise, cases can occur where the selected filter is not presented in the GenericObjectEditor.
  • adams-weka and adams-weka-lts:
    • The WekaFilter transformer no longer re-initializes the filter if loading from disk/actor/storage and initializeOnce is checked.
    • The CrossValidationFoldGenerator classes now report back the actual number of folds in use (eg when using -1 for LOO, this depends on the data), which in turn makes the WekaCrossValidationExecution work properly when using GroupedCrossValidationFoldGenerator in conjunction with LOO.
    • search methods that implement RankedOutputSearch (eg PrincipalComponents) now output the attribute importance from the ranking instead of the attribute index
  • adams-rats:
    • The Rat actor now disables the performLazySetup option if the sub-flow contains at least one Breakpoint actor (debugging doesn't work in conjunction with lazy setup).
    • The Subscribe rat input now ignores any notifications if the Rat actor is paused.
    • The RatControl actor how has optional support for bulk actions, ie checked Rat actors can be paused/resumed/stopped/started.
  • adams-spectral-2dim-core: fixed selection of instance generator when exporting visible spectra in the Spectrum Explorer: it is now possible to select from a class hierarchy

Changes

  • Introduced the adams.flow.processor.ActorProcessor interface, replacing the usage of the AbstractActorProcessor class.
  • The DeserializeToStorage standalone now accepts pairs of storage name / file name to reduce the number of actors necessary when loading lots of models. Storage name and/or file name can be variables as well.
  • The Flow editor now prompts whether to perform checks before saving a when it is considered large (default is 200 actors); occurs on a per-flow basis and can be reverted by simply checking/unchecking the checkbox in the menu.
  • Actors that implement the AtomicExecution interface (eg Tee or Branch) now allow the user to specify a timeout after which the sub-flow gets stopped nonetheless (when finishedBeforeStopping is set).
  • Upgraded lanterna to 3.0.1 (for providing an ncurses-like terminal interface).
  • adams-spreadsheet: The SpreadSheetSetCell transformer can use a cell finder now to locate the cells to update with the specified value as well (eg for updating all cells with missing values or a value less than a certain minimum).
  • adams-weka and adams-weka-lts: switched to updated Weka versions (3.9.2-fork-0.0.2 and 3.9.0-fork-0.0.5 respectively) that include patch for MyHeap usage in weka.core.neighboursearch.LinearNNSearch, avoiding the the number of elements cannot exceed the initially set maximum limit exception.

Additions

  • added Clear clipboard to menu of Flow editor
  • added MapToReport conversion
  • Added the ListObjectCommandlines actor processor for listing object command-lines of a specific class.
  • Added actor processor for updating the number of threads used by classes: ChangeNumThreads
  • adams-spreadsheet:
    • The SpreadSheetCellSelector transformer lets you select cells in a spreadsheet to forward in the flow.
    • The SpreadSheetRandomSystematicSample transformer picks a random systematic sample of rows from the incoming spreadsheet.
    • added Clear clipboard to menu of Spreadsheet file viewer
  • adams-weka and adams-weka-lts:
    • The PredictionTrend output generator for the classification tab of the Weka Investigator plots actual and predicted values in a single plot, sorted by actual. This allows spotting of trends.
    • Added NIPALS and SparsePLS
  • adams-meta: The ProcessorActor transformer allows you to process an incoming flow with a specific actor processor (eg listing variables or command-lines).
  • adams-spectral-2dim-core: added instance generator based solely on report fields and with dedicated class field: FieldInstanceGeneratorWithClass

Updates 2018/06/26

Finished some major refactoring related to loading and saving flows. Well, actually replacing an ugly hack from two years ago with a clean implementation. Properly decoupling user interface and readers/writers for flows. That and some further optimizations in the Flow editor resulted in nice speed improvements: a flow with 38,000 actors now loads in about 3 seconds on my laptop. BTW this is the largest flow that I have generated so far!

Further CPU profiling also revealed some other code locations which could be optimized, improving startup time of flows.

Fixes

  • The Preview browser now properly updates the display when selecting a different handler for serialized object.
  • SetManyVariables standalone and transformer now use a list internally, which improves performance of the addVariablePair method when setting hundreds of variables.
  • User interface improvements in the Flow editor that relate to long running background tasks, they no longer block the interface.
  • General Flow editor improvements to speed things up with flows that have 10s of thousands of actors.
  • The Switch and IfThenElse actor now clean up the current token once execution finishes, to avoid unnecessary memory usage.

Changes

  • The Image viewer now uses a multi-page instead of multi-tab interface as well.
  • Copy/paste of flow nodes now uses a custom DataFlavour on the clipboard to speed up checks whether pasting is available; added "Copy (plain text)" for old approach.
  • Switched to sizeofag 1.0.4. This enabled the Inspect memory sub menu in the Flow editor, displaying the memory consumption of the selected actor (detailed per class or just overall size).
  • An alternative visualvm executable can be specified now by the user through preferences (or the JVisualVM.props properties file). JVisualVM got dropped from JDKs starting with Java 9. The project is now maintained separately and you can download it from here: https://visualvm.github.io/
  • adams-rsync: upgraded rsync4j to 3.1.2-10.
  • adams-rats: a Rat actor can free up memory now with the wrapUpAfterExecution flag, which is available in conjunction with performLazySetup.
  • adams-weka and adams-weka-lts: upgraded to matrix-algorithms 0.0.4.
  • adams-webservice: upgraded Apache CXF to 3.2.5.

Additions

  • Added simple source actor for just keep the flow running: Idle. This is useful when using an event-based flow (eg using Rat actors) rather than a data driven one.
  • The ForceFlush control actor enforces actors implementing FlushSupporter (currently DumpFile and WekaInstanceDumper) to flush their buffers.
  • With the ArrayToChunks transformer you can split an array into sub-arrays of a specified length.
  • adams-weka and adams-weka-lts: meta-classifier that can use a fallback model for making predictions: Fallback
  • adams-rats: the ExecuteRats control actor allows execution of Rat actors in MANUAL mode whenever a token passes through.
  • adams-compress: added support for Zstd de/compression library (https://en.wikipedia.org/wiki/Zstd).
  • adams-spectral-2dim-core: added the MaxWaves outlier detector for enforcing an upper limit on wave numbers.
  • adams-spectral-3way-core: added support for feature generation through the ThreeWayDataFeatureGenerator transformer and its associated class hierarchy of feature generators.

Updates 2018/05/25

A lot of work went into the 3-way data module, as well as proper support for polygons in the image annotation framework. The audio module also got a bit of love.

Furthermore, some user interfaces got switched from multi-tab to multi-page ones, freeing up vertical screen real estate (most people of have wides-screen monitors!).

Fixes

  • External actors that use a file monitor can once again use variables in the file name for the external flow.
  • The flow panel and flow tree now get focused better, to enable a smoother keyboard interaction.
  • adams-imaging: The ImageAnnotator now uses a scrollpane to display more buttons than fit on screen.
  • adams-weka and adams-weka-lts: the MultiSearch meta-classifier now allows you to select the search algorithm for the parameter space.

Changes

  • CsvSpreadSheetWriter now allows an empty quote character, which will suppress any quoting.
  • Switched the following applications from multi-tab interface to a multi-page one in order to manage more tabs/pages at the same time without decreasing the vertical screen real estate (these pages can be detached/reattached):
    • Flow editor
    • adams-imaging: Image processor
    • adams-heatmap: Heatmap viewer
    • adams-spreadsheet: Spreadsheet file viewer

Additions

  • The implicit variable flow_filename_restore contains a file name based on the full file name of the flow, but with the .props extension. To be used for interactive actors like EnterManyValues to restore the entered values.
  • Added Favorite actor (here, after, beneath) keyboard actions (Ctrl+E, Ctrl+Alt+E, Ctrl+Shift+E).
  • adams-spectral-2dim-core: added meta spectrum reader that reads spectra from zip files: ZippedSpectrumReader.
  • adams-spectral-3way-core:
    • added heatmap viewer for 3-day data, like 3D fluorescent spectra (EEM)
    • added experimental support for building and applying the multi-way algorithms available from: https://github.com/waikato-datamining/multiway-algorithms
    • added lots of conversion to and fro tensors, 3-way data and spreadsheets
  • adams-audio:
    • With the AudioRecorder source it is possible to record audio from a microphone.
    • The WaveToAmplitudes conversion extracts amplitudes from Wave object, absolute or normalized
    • Added Resample wave filter for using different sample rate.
    • Added Cut wave filter for cutting out a portion of a wave.
    • Added sink for playing back audio files: AudioPlayback
  • adams-compress: added support for xz compression via Xz/UnXz transformers and XzTextFileReader.
  • adams-imaging:
    • Added the Grayscale buffered image transformer with customizable lumiance parameters via a class hierarchy (supporting BT 601, 709 and 2020).
    • The Rotate object filter rotates objects by the specified angle.
    • ViaAnnotations transformer generates annotations for VIA (VGG Image Annotator - http://www.robots.ox.ac.uk/~vgg/software/via/)
    • Added support for polygon annotations via the PolygonPainter (default is RectanglePainter).

Updates 2018/04/20

Spent a lot of time trying to get things working with Java 9. The usual backwards compatibility mantra got a serious beating with Java 9. ADAMS now compiles with Java 9, but I still recommend to stick with Java 8 for a while.

Futhermore, there is now a separate module for ffmpeg called adams-ffmpeg, rather than being tucked away in the adams-video module. In addition to that, I added an experimental module for audio processing called adams-audio.

Fixes

  • Now compiles with Java 9.
  • Auto-registering of flows now registers after the flow ID has been initialized rather than with 0.
  • adams-imaging:
    • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now use the user-specified font.
    • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now use the user-specified font.

Changes

  • Upgraded sizeofag to 1.0.2.

  • Upgraded jclasslocator to 0.0.10.

  • Removed archetype module.

  • Added keyboard shortcuts to the Flow editor for bringing up the actor tree to search for an actor to insert, skips the GOE dialog: Ctrl+J (here), Ctrl+Alt+J (after), Ctrl+Shift+J (beneath)

  • adams-spectral-2dim-core: The Evaluator transformer now allows to turn off

    cloning of container storing the evaluation results.

  • adams-webservices: updated Apache CXF to 3.2.4 to make it compile with Java 9 (10 doesn't work yet, but the upcoming 3.2.5 release has a fix for that).

  • adams-meka: Upgraded to MEKA 1.9.2.

  • adams-imaging:

    • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now offer x/y offsets for the label.
    • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now offer x/y offsets for the label.
    • ImageObjectInfo now allows to output the INDEX_STRING and INDEX_INT as well.

Additions

  • Added support for publish/subscribe: InitPublishSubscribe standalone for initializing pub/sub data structure in storage and the Publish sink for broadcasting the data to all the subscribers of the specified pub/sub data structure.
  • Added platform specific file-in-use checks: Windows and NonWindows (Default combines both).
  • Added the JsonObjectToMap conversion to turn a JSON object into a simple map.
  • Added the ExtractSimpleContainerContent conversion to retrieve the content of simple containers.
  • Added the Go to menu item to the Flow editor popup menu, which allows you to jump to the reference actor (eg a callable actor).
  • adams-rats:
    • added support for using sockets for sending and receiving data, aptly named Socket for input and output (adams-rats-sockets.flow).
    • added support for publish/subscribe with Subscribe rat input and Publish rat output. Simpler and more efficient than queues for notifying multiple Rat actors at once.
    • added Django web application for remote control of Rat actors that are available through the RatControl actor, using a REST webservice for communication.
  • adams-weka: The SocketFacade meta-classifier communicates via sockets with a prediction scheme, can be used to interface with any programming language as long as communication occurs via sockets.
  • adams-imaging:
    • added BasicStatistics feature generator (for buffered images).
  • adams-audio: new module for some basic audio processing of WAV files, generating spectrograms (as image or tabular data).
  • added separate adams-ffmpeg module, to make ffmpeg functionality available in adams-audio and adams-video.
  • adams-ml: added basic framework for calculating summary statistics with the SummaryStatistics transformer.
  • adams-spectral-2dim-core: added the SimpleSpectralInstanceGenerator instances generator, which does not output a class attribute in the generated Weka Instances object.

Updates 2018-03-26

Ran out of time on Friday to post this update, so here it goes...

Finally managed to sort my Java2D rendering issues (basically blocking my X11 session when displaying images with Java), by switching to OpenGL rendering by supplying this option to the JVM: -Dsun.java2d.opengl=true

The following modules were removed:

  • adams-imaging-imagej
  • adams-dl4j
  • adams-dl4j-insight
  • adams-dl4j-weka

Why remove deeplearning4j? Well, we just didn't have much luck with that framework and CNTK (despite being in Python) was much easier to work with (and the API didn't change all the time).

Fixes

  • The DumpFile sink no longer attempts to write to disk if the output file is pointing to directory (eg due to an invalid variable)
  • PlaceholderFile/Directory: getAbsolutePath() now checks whether the path is a Windows one (X:...) and if that is that case, just returns it (otherwise the CWD gets prepended).
  • The following overlay paintlets now always use the panel's container manager: MeanOverlayPaintlet, MedianOverlayPaintlet, StdDevOverlayPaintlet, LimitPaintlet
  • When enclosing a referenced actor (eg below CallableActors) in the Flow editor, it now triggers the updating of the reference throughout the flow.
  • Fixed how the bin directory of JRE/JDK is determined, now works with Java 9+.
  • adams-weka and adams-weka-lts:
    • The Export output... functionality in the Weka Investigator no longer exports model files from the predictions or model tabs, but spreadsheets and text files.
    • The search panel of the WekaInstancesDisplay now shows up when used in conjunction with the DisplayPanelManager.
  • adams-spectral-2dim-core:
    • The RowWiseSpreadSheetSpectrumReader and ColumnWiseSpreadSheetReader are now more robust when parsing numeric values in the spreadsheet, in case there is incorrect data present. They can also process multiple sheets, in case the spreadsheet reader supports that.

Changes

  • When saving a flow, the modules that were present at that time will get stored in the comments. When loading a flow with the modules listed, these modules are then checked against the current environment now, outputting a warning if modules missing.
  • When editing an actor in the Flow editor, the children no longer get discarded when both actors, new and old, implement MutableActorHandler (like Tee and Trigger).
  • The gradient color generators BiColorGenerator and MultiColorGenerator now support a custom alpha value to obtain transparent colors.
  • adams-weka-lts: now uses 3.9.0-fork-0.0.4, which had a few classes backported from 3.9.2 in order to make it compile and work under Java 9.
  • adams-spreadsheet: moved cell rendering customization (ie neg/pos background colors) into CellRenderingCustomizer class hierarchy.
  • adams-imaging: The ImageObjectOverlay can store meta-data values in the outgoing report of the objects with the highest overlaps as well, e.g., associated labels.
  • adams-ml: The ConfusionMatrix transformer can take a column with probabilities now into account to generate the counts. Furthermore, instead of counts, it can also generate overall percentages or row percentages now.

Additions

  • The StringArraySplit transformer splits string array into sub-arrays using the specified regular expression. E.g., can be used to split log file into individual entries using the timestamp as split criterion.
  • Added a color based on the X11 color palette (https://en.wikipedia.org/wiki/X11_color_names), called X11ColorProvider.
  • With the PercentilePaintlet it is possible to plot a horizontal line at the specified percentile for each sequence.
  • adams-imaging: added the Objects sub-images generator, which uses the specified object finder to locate the sub-images of interest and then extracts them (available through the SubImages BufferedImageTransformer).
  • adams-spreadsheet:
    • added the Percentile row/column statistic.
    • added the NumericRangeCellRenderingCustomizer for highlighting cells in specified columns that fall in defined intervals.
  • adams-ml: added the ConfsionMatrixCellRenderingCustomizer to make confusion matrix spreadsheets more readable (as generated by the ConfusionMatrix transformer).
  • adams-weka and adams-weka-lts:
    • Investigator, Multi-Experimenter, WekaRandomSplit, WekaCrossValidationSplit and WekaCrossValidationEvaluator now allow you to select the type of split generator you want to use. You can either use Weka's default splitting mechanism (Default...) or you can use the new Grouped... ones. The latter one is useful when dealing with data that contains duplicates or near-duplicates that can be grouped via a regular expression applied to a nominal or string attribute. An example is scanning the same sample with multiple instruments. A sample should never be present in train and test, as it gives a way to optimistic estimate (in other words: you are sort of cheating).
    • The Investigator now offers confusion matrix output in the classify tab, which offers various display and highlighting options.
  • adams-python:
    • Added new module to adams-base, to make interaction with Python easier.
    • PythonEnvironment standalone allows you to specify python/pip commands to use (eg from a virtual environment).
    • The PythonExec source executes a Python script and forwards the output that the script generates.
    • With PipExec you can run pip commnands and post-process the output the command generates.
    • The PythonExpandScript transformer expands variables and placholders in a script template and saves it to a new file.

Updates 2018/03/02

A few changes in terms of modules occurred: all -stable modules have been renamed to -lts (long term support) and moved to the new adams-lts repository (https://github.com/waikato-datamining/adams-lts). Furthermore, experimental support for the MongoDB NoSQL database has been added.

Fixes

  • The CheckActorReferenceUsage flow processor now skips actors that are disabled, in other words, a disabled callable actor won't get moaned about if not used.
  • Added missing setting of variable BASEDIR in the following batch files: daemon.bat, exec.bat, logging.bat
  • Updating of properties, e.g., with the UpdateProperty transformer, now handles arrays properly: the value string is interpreted as blank-separated list and the array constructed from those elements.
  • The dialog size of showInputDialog method (e.g., when renaming an actor) is now adjusting a bit more to the input.
  • The flow used by System performance (from the Help menu) is now working again properly.
  • adams-weka and adams-weka-lts: introduced custom aggregation of Evaluation objects (Relative absolute error and Root relative squared error will differ a bit due to different handling of training priors - still buggy in Weka ATM), affecting WekaAggregateEvaluations, WekaCrossValidationExecution, RemoveOutliers and WekaSpreadSheetToPredictions.
  • adams-rats: The FileLister rat input now empties the queue when returning an array.
  • adams-spreadsheet: The parser for the expression used by SpreadSheetQuery handles negative numbers now correctly.

Changes

  • Updated the section in the manual on how to install an ADAMS flow as daemon (Linux, systemd) and service (Windows, NSSM).
  • Added methods 'replaceext' (replacing file extension) and 'ext' (extracting file extension) to parsers BooleanExpression, MathematicalExpression and StringExpression.
  • Added help menu items in the Flow editor for: variables, boolean/math/string expressions
  • Combined help display of GenericObjectEditor and Class help.
  • adams-spreadsheet: The formula of conversions SpreadSheetAddFormulaColumn and SpreadSheetAddFormulaRow can contain variables now that get expanded before inserting it into the spreadsheet.
  • adams-rats: The ChangeRatState control actor no longer checks whether the specified Rat actors are present if a variable is attached to the "rats" property (the one listing the names).
  • adams-weka and adams-weka-lts:
    • added mxexpression-weka-package dependency, offering filter and classifier that use mathematical expression to update target attribute or make prediction, using the mXparser library (https://sourceforge.net/projects/mxparser/).
    • Class help in the GUI is now in HTML; help in GenericObjectEditor is now showing full help.
    • The WekaCrossValidationEvaluator now has an option (finalModel) to build a model on the full dataset. Automatically implies that a container is being output.

Additions

  • The GroupByRegExp transformer allows generating groups of strings from array using a regular expression and a grouping expression, e.g., "(.*)-RGB.*" as regexp and "$1" for grouping them.
  • Added some of Eric Eaton's Java utility classes located at: http://www.seas.upenn.edu/~eeaton/software/Utils/Utils.zip
  • Added conversions MathExpression and StringExpression for better efficiency when processing spreadsheets (using SpreadSheetConvertCells instead of SpreadSheetTransformCells).
  • The File monitor tool allows you to monitor files that get appended, like log files, outputting all the new data added to the files.
  • adams-spectral-2dim-core:
    • The NewSpectrum source allows the creation of a new spectrum.
    • With GetSpectrumAmplitude the amplitude of the specified wave number can be retrieved.
    • SetSpectrumAmplitude allows you to set the amplitude of the specified wave number or simply add a new spetrum point.
  • Added module adams-mongodb for basic MongoDB support.
  • adams-weka and adams-weka-lts: added ability to upload currently best setup in classifier-based genetic algorithms via a setup upload scheme, e.g., in a MySQL database.

Updates 2018/02/12

Apart from some bugfixes, the biggest change is the addition of the adams-rest module. This module allows you to quickly create REST webservices, a simpler alternative to SOAP-based ones. Using the GenericServer, you can simply slot together any functionality that you want, using classes that implement the adams.flow.rest.RESTPlugin interface. See below for more details on what REST functionality is already available.

Fixes

  • adams-spreadsheet: Fixed handling of null values for date/time columns in the SpreadSheetDbReader source.
  • Improved positioning of dialogs/frames on screen, leaving (by default) 50 pixels at the bottom of the screen reserved (GUIHelper.props, property ScreenBorder.Bottom). This avoids dialogs like the GenericObjectEditor to appear behind the taskbar on Windows.
  • adams-rats: The FileLister rat input had a bug when adding the moved files to its output list when skipping files that were in use, generating an IndexOutOfBoundsException.

Changes

  • The expressions in the debug panel of the Flow editor now allow string expressions as well, rather than just simple strings with variables.
  • Added ability to delete via regular expression: DeleteVariable, DeleteStorageValue
  • The StringCut transformer now uses Index instead of int for the character positions, allowing the use of last.
  • adams-imaging: The conversions *RectangleToString and StringToRectangle can parse the x0y0x1y1 format now as well (default is xywh).
  • adams-net: The HttpRequest transformer now accepts byte arrays as well as input.
  • adams-timeseries: Updated timeseriesForecasting dependency to 1.0.25

Additions

  • SwapVariables allows swapping of values of two variables.
  • Run to here was added to the Flow editor menu, which can be used while debugging a flow and you don't want to step through all the actors.
  • Additional boolean conditions for tokens: IsBoolean, IsByte, IsShort, IsInteger, IsLong, IsFloat, IsDouble
  • Added JSON support for reports: conversions JsonToReport and ReportToJson, DefaultSimpleJsonReportReader and DefaultSimpleJsonReportWriter.
  • Added the PathSplit conversion for splitting path into its individual parts: drive letter or server from UCN path (if applicable), directories, file name.
  • adams-imaging:
    • Added modified version of CountColor, which performs linescan from left/right for counting and stops as soon as it hits another color (used for objects located in center of image): CountColorOutside
    • Added the LocatedObjectToRectangle conversion, which turns the position of the object into a BaseRectangle object.
    • The GetImageObjectMetaData transformer retrieves the meta-data map of a located object.
  • The SimpleStringReplace transformer uses simple String.replace(..., ...) for replacing sub-strings. Gets around problems with trying to specify the regular expression of StringReplace when trying to only do a simple string replacement.
  • adams-net:
    • Added URLEncode and URLDecode conversions for creating URL-conform strings and turning them back into regular strings (e.g., "hello world" into "hello+world").
    • Added StringArrayToURLParameters and URLParametersToStringArray conversions to turn string array of key-value pairs and a string that can be appended to a URL (and vice versa).
  • Added adams-rest module, for providing REST webservice support.
  • adams-rats:
    • Added generic REST support via the adams.flow.rest.RatsServer REST server (same as GenericServer, but aware of the Rat actor it belongs to).
    • Generic (text) upload capability using the adams.flow.rest.text.RatsTextUpload REST plugin.
    • Added support for controlling Rat actors (accessible via RatControl/RegisterFlow) via REST service: control.RatControl
  • adams-spectral-2dim-core:
    • Added conversions to turn spectrum into JSON string and vice versa: SpectrumToJson, JsonToSpectrum.
    • Added JSON support for reports: DefaultSimpleJsonSampleDataReader and DefaultSimpleJsonSampleDataWriter.
  • adams-spectral-2dim-webservice:
    • Added REST webservice plugins for spectra: DeleteSpectrum (delete from DB), GetSpectrum (load from DB), PutSpectrum (store in DB), TransformSpectrum (transform spectrum using specific filter)
  • adams-spreadsheet: With the SpreadSheetSubsetFromGroup transformer it is possible to retrieve specific rows from groups within spreadsheet.

Updates 2018/01/12

I hope everyone had a great start to the new year. It's been a busy start here as there were some major changes in terms of Weka support within ADAMS.

The patched versions of Weka that ADAMS uses, are now available as separate branches from the following github repository:

waikato-datamining/weka

The adams-weka module has been cloned into adams-weka-stable, which will continue to use our patched version of Weka 3.9.0. adams-weka will use more up-to-date versions of the developer version of Weka from now on, starting with a patched version of 3.9.2.

The reason for this split was to be able to offer a stable Weka version to our commercial clients (not much fun if your serialized models are not compatible with a newer version and you have to rebuild all your models), but regular ADAMS users can still enjoy using the latest version of Weka. The spectral frameworks will continue to use the 3.9.0 version.

Another change happened, affecting the start up scripts for ADAMS. To avoid collisions with user's local installations of Weka and their respective packages, the Launcher class used by the start up scripts now sets a custom WEKA_HOME environment variable, pointing to a directory below the project's home directory ($HOME/.adams or %USERPROFILE%_adams), taking the Weka version into account. Once again, this avoids problems with conflicting Weka packages between different versions of Weka, when using different versions of ADAMS. This approach of separating Weka is something that I have used for my weka-virtualenv tool as well.

Fixes

  • adams-weka:
    • Fixed making predictions with the PLS1 instances analysis algorithm, used by the PLS supervised filter.

Changes

  • Added support for LONG to IncVariable and IncStorageValue transformers.
  • It is now possible to define a dialog width for the EnterManyValues source actor.
  • adams-weka:
    • Upgraded multisearch-weka-package to 2017.10.1
    • Upgraded partialLeastSquares dependency to 1.0.5
  • adams-groovy: removed Weka (and related) dependencies.
  • adams-jython: removed Weka (and related) dependencies.
  • adams-rsync: upgraded rsync4j to 3.1.2-7

Additions

  • Added better support for collections with the NewCollection source and CollectionInsert transformer.
  • Created copy of adams-weka as adams-weka-stable, with adams-weka now using Weka 3.9.2.
  • Created copy of adams-cntk-weka as adams-cntk-weka-stable, using adams-weka-stable dependencies.