Finished some major refactoring related to loading and saving flows. Well, actually replacing an ugly hack from two years ago with a clean implementation. Properly decoupling user interface and readers/writers for flows. That and some further optimizations in the Flow editor resulted in nice speed improvements: a flow with 38,000 actors now loads in about 3 seconds on my laptop. BTW this is the largest flow that I have generated so far!

Further CPU profiling also revealed some other code locations which could be optimized, improving startup time of flows.

Fixes

• The Preview browser now properly updates the display when selecting a different handler for serialized object.

• SetManyVariables standalone and transformer now use a list internally, which improves performance of the addVariablePair method when setting hundreds of variables.

• User interface improvements in the Flow editor that relate to long running background tasks, they no longer block the interface.

• General Flow editor improvements to speed things up with flows that have 10s of thousands of actors.

• The Switch and IfThenElse actor now clean up the current token once execution finishes, to avoid unnecessary memory usage.

Changes

• The Image viewer now uses a multi-page instead of multi-tab interface as well.

• Copy/paste of flow nodes now uses a custom DataFlavour on the clipboard to speed up checks whether pasting is available; added "Copy (plain text)" for old approach.

• Switched to sizeofag 1.0.4. This enabled the Inspect memory sub menu in the Flow editor, displaying the memory consumption of the selected actor (detailed per class or just overall size).

• An alternative visualvm executable can be specified now by the user through preferences (or the JVisualVM.props properties file). JVisualVM got dropped from JDKs starting with Java 9. The project is now maintained separately and you can download it from here: https://visualvm.github.io/

• adams-rsync: upgraded rsync4j to 3.1.2-10.

• adams-rats: a Rat actor can free up memory now with the wrapUpAfterExecution flag, which is available in conjunction with performLazySetup.

• adams-weka and adams-weka-lts: upgraded to matrix-algorithms 0.0.4.

• adams-webservice: upgraded Apache CXF to 3.2.5.

Additions

• Added simple source actor for just keep the flow running: Idle. This is useful when using an event-based flow (eg using Rat actors) rather than a data driven one.

• The ForceFlush control actor enforces actors implementing FlushSupporter (currently DumpFile and WekaInstanceDumper) to flush their buffers.

• With the ArrayToChunks transformer you can split an array into sub-arrays of a specified length.

• adams-weka and adams-weka-lts: meta-classifier that can use a fallback model for making predictions: Fallback

• adams-rats: the ExecuteRats control actor allows execution of Rat actors in MANUAL mode whenever a token passes through.

• adams-compress: added support for Zstd de/compression library (https://en.wikipedia.org/wiki/Zstd).

• adams-spectral-2dim-core: added the MaxWaves outlier detector for enforcing an upper limit on wave numbers.

• adams-spectral-3way-core: added support for feature generation through the ThreeWayDataFeatureGenerator transformer and its associated class hierarchy of feature generators.

A lot of work went into the 3-way data module, as well as proper support for polygons in the image annotation framework. The audio module also got a bit of love.

Furthermore, some user interfaces got switched from multi-tab to multi-page ones, freeing up vertical screen real estate (most people of have wides-screen monitors!).

Fixes

• External actors that use a file monitor can once again use variables in the file name for the external flow.

• The flow panel and flow tree now get focused better, to enable a smoother keyboard interaction.

• adams-imaging: The ImageAnnotator now uses a scrollpane to display more buttons than fit on screen.

• adams-weka and adams-weka-lts: the MultiSearch meta-classifier now allows you to select the search algorithm for the parameter space.

Changes

• CsvSpreadSheetWriter now allows an empty quote character, which will suppress any quoting.

• Switched the following applications from multi-tab interface to a multi-page one in order to manage more tabs/pages at the same time without decreasing the vertical screen real estate (these pages can be detached/reattached):

  • Flow editor

  • adams-imaging: Image processor

  • adams-heatmap: Heatmap viewer

  • adams-spreadsheet: Spreadsheet file viewer

Additions

• The implicit variable flow_filename_restore contains a file name based on the full file name of the flow, but with the .props extension. To be used for interactive actors like EnterManyValues to restore the entered values.

• Added Favorite actor (here, after, beneath) keyboard actions (Ctrl+E, Ctrl+Alt+E, Ctrl+Shift+E).

• adams-spectral-2dim-core: added meta spectrum reader that reads spectra from zip files: ZippedSpectrumReader.

• adams-spectral-3way-core:

  • added heatmap viewer for 3-day data, like 3D fluorescent spectra (EEM)

  • added experimental support for building and applying the multi-way algorithms available from: https://github.com/waikato-datamining/multiway-algorithms

  • added lots of conversion to and fro tensors, 3-way data and spreadsheets

• adams-audio:

  • With the AudioRecorder source it is possible to record audio from a microphone.

  • The WaveToAmplitudes conversion extracts amplitudes from Wave object, absolute or normalized

  • Added Resample wave filter for using different sample rate.

  • Added Cut wave filter for cutting out a portion of a wave.

  • Added sink for playing back audio files: AudioPlayback

• adams-compress: added support for xz compression via Xz/UnXz transformers and XzTextFileReader.

• adams-imaging:

  • Added the Grayscale buffered image transformer with customizable lumiance parameters via a class hierarchy (supporting BT 601, 709 and 2020).

  • The Rotate object filter rotates objects by the specified angle.

  • ViaAnnotations transformer generates annotations for VIA (VGG Image Annotator - http://www.robots.ox.ac.uk/~vgg/software/via/)

  • Added support for polygon annotations via the PolygonPainter (default is RectanglePainter).

Spent a lot of time trying to get things working with Java 9. The usual backwards compatibility mantra got a serious beating with Java 9. ADAMS now compiles with Java 9, but I still recommend to stick with Java 8 for a while.

Futhermore, there is now a separate module for ffmpeg called adams-ffmpeg, rather than being tucked away in the adams-video module. In addition to that, I added an experimental module for audio processing called adams-audio.

Fixes

• Now compiles with Java 9.

• Auto-registering of flows now registers after the flow ID has been initialized rather than with 0.

• adams-imaging:

  • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now use the user-specified font.

  • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now use the user-specified font.

Changes

• Upgraded sizeofag to 1.0.2.

• Upgraded jclasslocator to 0.0.10.

• Removed archetype module.

• Added keyboard shortcuts to the Flow editor for bringing up the actor tree to search for an actor to insert, skips the GOE dialog: Ctrl+J (here), Ctrl+Alt+J (after), Ctrl+Shift+J (beneath)

• adams-spectral-2dim-core: The Evaluator transformer now allows to turn off

  cloning of container storing the evaluation results.

• adams-webservices: updated Apache CXF to 3.2.4 to make it compile with Java 9 (10 doesn't work yet, but the upcoming 3.2.5 release has a fix for that).

• adams-meka: Upgraded to MEKA 1.9.2.

• adams-imaging:

  • The draw operations ObjectCentersFromReport and ObjectLocationsFromReport now offer x/y offsets for the label.

  • The image overlays ObjectCentersOverlayFromReport and ObjectLocationsOverlayFromReport now offer x/y offsets for the label.

  • ImageObjectInfo now allows to output the INDEX_STRING and INDEX_INT as well.

Additions

• Added support for publish/subscribe: InitPublishSubscribe standalone for initializing pub/sub data structure in storage and the Publish sink for broadcasting the data to all the subscribers of the specified pub/sub data structure.

• Added platform specific file-in-use checks: Windows and NonWindows (Default combines both).

• Added the JsonObjectToMap conversion to turn a JSON object into a simple map.

• Added the ExtractSimpleContainerContent conversion to retrieve the content of simple containers.

• Added the Go to menu item to the Flow editor popup menu, which allows you to jump to the reference actor (eg a callable actor).

• adams-rats:

  • added support for using sockets for sending and receiving data, aptly named Socket for input and output (adams-rats-sockets.flow).

  • added support for publish/subscribe with Subscribe rat input and Publish rat output. Simpler and more efficient than queues for notifying multiple Rat actors at once.

  • added Django web application for remote control of Rat actors that are available through the RatControl actor, using a REST webservice for communication.

• adams-weka: The SocketFacade meta-classifier communicates via sockets with a prediction scheme, can be used to interface with any programming language as long as communication occurs via sockets.

• adams-imaging:

  • added BasicStatistics feature generator (for buffered images).

• adams-audio: new module for some basic audio processing of WAV files, generating spectrograms (as image or tabular data).

• added separate adams-ffmpeg module, to make ffmpeg functionality available in adams-audio and adams-video.

• adams-ml: added basic framework for calculating summary statistics with the SummaryStatistics transformer.

• adams-spectral-2dim-core: added the SimpleSpectralInstanceGenerator instances generator, which does not output a class attribute in the generated Weka Instances object.

Ran out of time on Friday to post this update, so here it goes...

Finally managed to sort my Java2D rendering issues (basically blocking my X11 session when displaying images with Java), by switching to OpenGL rendering by supplying this option to the JVM: -Dsun.java2d.opengl=true

The following modules were removed:

• adams-imaging-imagej

• adams-dl4j

• adams-dl4j-insight

• adams-dl4j-weka

Why remove deeplearning4j? Well, we just didn't have much luck with that framework and CNTK (despite being in Python) was much easier to work with (and the API didn't change all the time).

Fixes

• The DumpFile sink no longer attempts to write to disk if the output file is pointing to directory (eg due to an invalid variable)

• PlaceholderFile/Directory: getAbsolutePath() now checks whether the path is a Windows one (X:...) and if that is that case, just returns it (otherwise the CWD gets prepended).

• The following overlay paintlets now always use the panel's container manager: MeanOverlayPaintlet, MedianOverlayPaintlet, StdDevOverlayPaintlet, LimitPaintlet

• When enclosing a referenced actor (eg below CallableActors) in the Flow editor, it now triggers the updating of the reference throughout the flow.

• Fixed how the bin directory of JRE/JDK is determined, now works with Java 9+.

• adams-weka and adams-weka-lts:

  • The Export output... functionality in the Weka Investigator no longer exports model files from the predictions or model tabs, but spreadsheets and text files.

  • The search panel of the WekaInstancesDisplay now shows up when used in conjunction with the DisplayPanelManager.

• adams-spectral-2dim-core:

  • The RowWiseSpreadSheetSpectrumReader and ColumnWiseSpreadSheetReader are now more robust when parsing numeric values in the spreadsheet, in case there is incorrect data present. They can also process multiple sheets, in case the spreadsheet reader supports that.

Changes

• When saving a flow, the modules that were present at that time will get stored in the comments. When loading a flow with the modules listed, these modules are then checked against the current environment now, outputting a warning if modules missing.

• When editing an actor in the Flow editor, the children no longer get discarded when both actors, new and old, implement MutableActorHandler (like Tee and Trigger).

• The gradient color generators BiColorGenerator and MultiColorGenerator now support a custom alpha value to obtain transparent colors.

• adams-weka-lts: now uses 3.9.0-fork-0.0.4, which had a few classes backported from 3.9.2 in order to make it compile and work under Java 9.

• adams-spreadsheet: moved cell rendering customization (ie neg/pos background colors) into CellRenderingCustomizer class hierarchy.

• adams-imaging: The ImageObjectOverlay can store meta-data values in the outgoing report of the objects with the highest overlaps as well, e.g., associated labels.

• adams-ml: The ConfusionMatrix transformer can take a column with probabilities now into account to generate the counts. Furthermore, instead of counts, it can also generate overall percentages or row percentages now.

Additions

• The StringArraySplit transformer splits string array into sub-arrays using the specified regular expression. E.g., can be used to split log file into individual entries using the timestamp as split criterion.

• Added a color based on the X11 color palette (https://en.wikipedia.org/wiki/X11_color_names), called X11ColorProvider.

• With the PercentilePaintlet it is possible to plot a horizontal line at the specified percentile for each sequence.

• adams-imaging: added the Objects sub-images generator, which uses the specified object finder to locate the sub-images of interest and then extracts them (available through the SubImages BufferedImageTransformer).

• adams-spreadsheet:

  • added the Percentile row/column statistic.

  • added the NumericRangeCellRenderingCustomizer for highlighting cells in specified columns that fall in defined intervals.

• adams-ml: added the ConfsionMatrixCellRenderingCustomizer to make confusion matrix spreadsheets more readable (as generated by the ConfusionMatrix transformer).

• adams-weka and adams-weka-lts:

  • Investigator, Multi-Experimenter, WekaRandomSplit, WekaCrossValidationSplit and WekaCrossValidationEvaluator now allow you to select the type of split generator you want to use. You can either use Weka's default splitting mechanism (Default...) or you can use the new Grouped... ones. The latter one is useful when dealing with data that contains duplicates or near-duplicates that can be grouped via a regular expression applied to a nominal or string attribute. An example is scanning the same sample with multiple instruments. A sample should never be present in train and test, as it gives a way to optimistic estimate (in other words: you are sort of cheating).

  • The Investigator now offers confusion matrix output in the classify tab, which offers various display and highlighting options.

• adams-python:

  • Added new module to adams-base, to make interaction with Python easier.

  • PythonEnvironment standalone allows you to specify python/pip commands to use (eg from a virtual environment).

  • The PythonExec source executes a Python script and forwards the output that the script generates.

  • With PipExec you can run pip commnands and post-process the output the command generates.

  • The PythonExpandScript transformer expands variables and placholders in a script template and saves it to a new file.

A few changes in terms of modules occurred: all -stable modules have been renamed to -lts (long term support) and moved to the new adams-lts repository (https://github.com/waikato-datamining/adams-lts). Furthermore, experimental support for the MongoDB NoSQL database has been added.

Fixes

• The CheckActorReferenceUsage flow processor now skips actors that are disabled, in other words, a disabled callable actor won't get moaned about if not used.

• Added missing setting of variable BASEDIR in the following batch files: daemon.bat, exec.bat, logging.bat

• Updating of properties, e.g., with the UpdateProperty transformer, now handles arrays properly: the value string is interpreted as blank-separated list and the array constructed from those elements.

• The dialog size of showInputDialog method (e.g., when renaming an actor) is now adjusting a bit more to the input.

• The flow used by System performance (from the Help menu) is now working again properly.

• adams-weka and adams-weka-lts: introduced custom aggregation of Evaluation objects (Relative absolute error and Root relative squared error will differ a bit due to different handling of training priors - still buggy in Weka ATM), affecting WekaAggregateEvaluations, WekaCrossValidationExecution, RemoveOutliers and WekaSpreadSheetToPredictions.

• adams-rats: The FileLister rat input now empties the queue when returning an array.

• adams-spreadsheet: The parser for the expression used by SpreadSheetQuery handles negative numbers now correctly.

Changes

• Updated the section in the manual on how to install an ADAMS flow as daemon (Linux, systemd) and service (Windows, NSSM).

• Added methods 'replaceext' (replacing file extension) and 'ext' (extracting file extension) to parsers BooleanExpression, MathematicalExpression and StringExpression.

• Added help menu items in the Flow editor for: variables, boolean/math/string expressions

• Combined help display of GenericObjectEditor and Class help.

• adams-spreadsheet: The formula of conversions SpreadSheetAddFormulaColumn and SpreadSheetAddFormulaRow can contain variables now that get expanded before inserting it into the spreadsheet.

• adams-rats: The ChangeRatState control actor no longer checks whether the specified Rat actors are present if a variable is attached to the "rats" property (the one listing the names).

• adams-weka and adams-weka-lts:

  • added mxexpression-weka-package dependency, offering filter and classifier that use mathematical expression to update target attribute or make prediction, using the mXparser library (https://sourceforge.net/projects/mxparser/).

  • Class help in the GUI is now in HTML; help in GenericObjectEditor is now showing full help.

  • The WekaCrossValidationEvaluator now has an option (finalModel) to build a model on the full dataset. Automatically implies that a container is being output.

Additions

• The GroupByRegExp transformer allows generating groups of strings from array using a regular expression and a grouping expression, e.g., "(.*)-RGB.*" as regexp and "$1" for grouping them.

• Added some of Eric Eaton's Java utility classes located at: http://www.seas.upenn.edu/~eeaton/software/Utils/Utils.zip

• Added conversions MathExpression and StringExpression for better efficiency when processing spreadsheets (using SpreadSheetConvertCells instead of SpreadSheetTransformCells).

• The File monitor tool allows you to monitor files that get appended, like log files, outputting all the new data added to the files.

• adams-spectral-2dim-core:

  • The NewSpectrum source allows the creation of a new spectrum.

  • With GetSpectrumAmplitude the amplitude of the specified wave number can be retrieved.

  • SetSpectrumAmplitude allows you to set the amplitude of the specified wave number or simply add a new spetrum point.

• Added module adams-mongodb for basic MongoDB support.

• adams-weka and adams-weka-lts: added ability to upload currently best setup in classifier-based genetic algorithms via a setup upload scheme, e.g., in a MySQL database.

Apart from some bugfixes, the biggest change is the addition of the adams-rest module. This module allows you to quickly create REST webservices, a simpler alternative to SOAP-based ones. Using the GenericServer, you can simply slot together any functionality that you want, using classes that implement the adams.flow.rest.RESTPlugin interface. See below for more details on what REST functionality is already available.

Fixes

• adams-spreadsheet: Fixed handling of null values for date/time columns in the SpreadSheetDbReader source.

• Improved positioning of dialogs/frames on screen, leaving (by default) 50 pixels at the bottom of the screen reserved (GUIHelper.props, property ScreenBorder.Bottom). This avoids dialogs like the GenericObjectEditor to appear behind the taskbar on Windows.

• adams-rats: The FileLister rat input had a bug when adding the moved files to its output list when skipping files that were in use, generating an IndexOutOfBoundsException.

Changes

• The expressions in the debug panel of the Flow editor now allow string expressions as well, rather than just simple strings with variables.

• Added ability to delete via regular expression: DeleteVariable, DeleteStorageValue

• The StringCut transformer now uses Index instead of int for the character positions, allowing the use of last.

• adams-imaging: The conversions *RectangleToString and StringToRectangle can parse the x0y0x1y1 format now as well (default is xywh).

• adams-net: The HttpRequest transformer now accepts byte arrays as well as input.

• adams-timeseries: Updated timeseriesForecasting dependency to 1.0.25

Additions

• SwapVariables allows swapping of values of two variables.

• Run to here was added to the Flow editor menu, which can be used while debugging a flow and you don't want to step through all the actors.

• Additional boolean conditions for tokens: IsBoolean, IsByte, IsShort, IsInteger, IsLong, IsFloat, IsDouble

• Added JSON support for reports: conversions JsonToReport and ReportToJson, DefaultSimpleJsonReportReader and DefaultSimpleJsonReportWriter.

• Added the PathSplit conversion for splitting path into its individual parts: drive letter or server from UCN path (if applicable), directories, file name.

• adams-imaging:

  • Added modified version of CountColor, which performs linescan from left/right for counting and stops as soon as it hits another color (used for objects located in center of image): CountColorOutside

  • Added the LocatedObjectToRectangle conversion, which turns the position of the object into a BaseRectangle object.

  • The GetImageObjectMetaData transformer retrieves the meta-data map of a located object.

• The SimpleStringReplace transformer uses simple String.replace(..., ...) for replacing sub-strings. Gets around problems with trying to specify the regular expression of StringReplace when trying to only do a simple string replacement.

• adams-net:

  • Added URLEncode and URLDecode conversions for creating URL-conform strings and turning them back into regular strings (e.g., "hello world" into "hello+world").

  • Added StringArrayToURLParameters and URLParametersToStringArray conversions to turn string array of key-value pairs and a string that can be appended to a URL (and vice versa).

• Added adams-rest module, for providing REST webservice support.

• adams-rats:

  • Added generic REST support via the adams.flow.rest.RatsServer REST server (same as GenericServer, but aware of the Rat actor it belongs to).

  • Generic (text) upload capability using the adams.flow.rest.text.RatsTextUpload REST plugin.

  • Added support for controlling Rat actors (accessible via RatControl/RegisterFlow) via REST service: control.RatControl

• adams-spectral-2dim-core:

  • Added conversions to turn spectrum into JSON string and vice versa: SpectrumToJson, JsonToSpectrum.

  • Added JSON support for reports: DefaultSimpleJsonSampleDataReader and DefaultSimpleJsonSampleDataWriter.

• adams-spectral-2dim-webservice:

  • Added REST webservice plugins for spectra: DeleteSpectrum (delete from DB), GetSpectrum (load from DB), PutSpectrum (store in DB), TransformSpectrum (transform spectrum using specific filter)

• adams-spreadsheet: With the SpreadSheetSubsetFromGroup transformer it is possible to retrieve specific rows from groups within spreadsheet.

I hope everyone had a great start to the new year. It's been a busy start here as there were some major changes in terms of Weka support within ADAMS.

The patched versions of Weka that ADAMS uses, are now available as separate branches from the following github repository:

waikato-datamining/weka

The adams-weka module has been cloned into adams-weka-stable, which will continue to use our patched version of Weka 3.9.0. adams-weka will use more up-to-date versions of the developer version of Weka from now on, starting with a patched version of 3.9.2.

The reason for this split was to be able to offer a stable Weka version to our commercial clients (not much fun if your serialized models are not compatible with a newer version and you have to rebuild all your models), but regular ADAMS users can still enjoy using the latest version of Weka. The spectral frameworks will continue to use the 3.9.0 version.

Another change happened, affecting the start up scripts for ADAMS. To avoid collisions with user's local installations of Weka and their respective packages, the Launcher class used by the start up scripts now sets a custom WEKA_HOME environment variable, pointing to a directory below the project's home directory ($HOME/.adams or %USERPROFILE%_adams), taking the Weka version into account. Once again, this avoids problems with conflicting Weka packages between different versions of Weka, when using different versions of ADAMS. This approach of separating Weka is something that I have used for my weka-virtualenv tool as well.

Fixes

• adams-weka:

  • Fixed making predictions with the PLS1 instances analysis algorithm, used by the PLS supervised filter.

Changes

• Added support for LONG to IncVariable and IncStorageValue transformers.

• It is now possible to define a dialog width for the EnterManyValues source actor.

• adams-weka:

  • Upgraded multisearch-weka-package to 2017.10.1

  • Upgraded partialLeastSquares dependency to 1.0.5

• adams-groovy: removed Weka (and related) dependencies.

• adams-jython: removed Weka (and related) dependencies.

• adams-rsync: upgraded rsync4j to 3.1.2-7

Additions

• Added better support for collections with the NewCollection source and CollectionInsert transformer.

• Created copy of adams-weka as adams-weka-stable, with adams-weka now using Weka 3.9.2.

• Created copy of adams-cntk-weka as adams-cntk-weka-stable, using adams-weka-stable dependencies.

ADAMS 17.12.0 has been released. More details and download links are available from here.

Last updates post before I'm getting stuck in making a new release...

A lot of things have happened in the image processing space again and a neat new feature in spreadsheet and instances tables (Spreadheet viewer, Weka Investigator) is being able to set filter strings or regular expressions per table column. This is column specific filtering, whereas the search box filters across the whole table.

Fixes

• The compact format for saving flows no longer stores the default actors along-side (in some cases).

• The Preview browser now transfers the search terms as well when opening a new window.

• Date-based chooser panels: setting the current date value now before displaying the dialog and automatically disposing the dialog once closed.

• adams-net: SimpleApacheSendEmail can handle attachments now.

• adams-spreadsheet: In the LookUpUpdate rules, variables can contain "." now as well; fixed replacing of '' around variables (replaced [] instead).

• adams-weka:

  • fixed rendering of missing values when sorting the data in the data tab of the Weka Investigator,

  • The evaluation statistics Bias, RPD, R^2 and SDR are now skipping missing value predictions.

• adams-rats: The FileLister input now has a flag to turn on reporting of moving errors.

• adams-video: The FixedIntervalBufferedImageSampler image sampler now handles the offset correctly when set to "-Inf".

Changes

• Upgraded snake yaml to 1.19.

• Menu items can have their category (ie the menu that they appear in) now reassigned via the Main.props file: Category-<classname>=<menu name>

• Added the following programmatic variables to the flows: project_name, project_home, project_modules.

• The Continue control actor got renamed to Block, which is a bit more intuitive in the flow environment (though it works like the continue control statement in Java).

• Added placeholders for TODAY, TOMORROW, YESTERDAY (using the start of day) to BaseDate, BaseDateTime and BaseDateTimeMsec.

• LocalScopeTransformer can now finish the processing of data before stopping (atomic execution).

• The SequencePlotter plot popup menu now has a Hits submenu, which allows you to perform operations on the sequences that the mouse is hovering over.

• The Flow editor has a new window operation for moving the current flow tab to a new editor window.

• added MATH_EXPRESSION_ROUND enum to the SetVariable and SetManyVariables actors, which rounds the result to a long.

• Any ADAMS panel with axis now allows copy/paste of axis range (in Range dialog), to speed up transfer of settings.

• adams-spreadsheet:

  • The LookUpUpdate parser no longer transfers back local variables (ie ones starting with an underscore).

  • Added "CONNECTION" meta-data type for outputting URL, database, user, password to the DatabaseMetaData source.

  • Added support for supplying a filter (simple string or regexp) per column.

• adams-imaging:

  • GetImageObjectIndices is more flexible now, by using an object finder for locating the objects of interest.

  • The ImageObjectFilter now allows you to apply an object filter to the images that the object finder located.

  • ScaleReportObjects has been deprecated, instead you can use the ImageObjectFilter in conjunction with the Scale object filter.

  • The generators of the SubImages BufferedImage transformer now limit the objects in a sub-image to the ones that fall into that region.

  • ImageObjectInfo transformer now accepts LocatedObject objects as well (eg as output by GetImageObjects).

  • upgraded zxing to 3.3.1 (used for barcode detection/generation)

• adams-rats: The Rat standalone can now finish the processing of data before stopping (atomic execution).

• adams-spectral-2dim-core:

  • The Rebase spectrum filter can change the step size between wave numbers now as well.

  • The plot popup menu of the spectrum panel now has a Hits submenu, which allows you to perform operations on the spectra that the mouse is hovering over.

• adams-weka: Added support for supplying a filter (simple string or regexp) per column to table displaying Instances.

Additions

• Added boolean condition for matching strings with a regular expression: RegExp.

• adams-imaging:

  • added new class hierarchy for filtering objects stored in reports (e.g., annotations).

  • With the ChangeImageObjectPrefix transformer, you can replace the prefix of objects in the image's report. Allows to combine annotations and predictions in one report, using separate overlays when displaying the image.

  • GetImageObjects outputs all the objects located by the specified finder.

  • GetImageObjects outputs objects of type LocatedObject identified by the specified finder scheme, instead of their indices like GetImageObjectIndices does. Allows for querying meta-data etc via the ImageObjectInfo transformer.

  • ImageObjectOverlap computes the overlap of objects between two reports.

  • Added the following operations for the Draw transformer: ObjectCentersFromReport, ObjectLocationsFromReport

  • The ImageObjectIndexOffset transformer changes the offset of the objects in the report by the specified amount. Useful when merging multiple reports.

  • MultiQRCode barcode decoder allows detection of multiple QR codes in a single image.

  • The DownSample BufferedImageTransformer is a simple scheme for downsizing images.

  • Added the erode/dilate morphology operations.

• adams-spectral-2dim-core:

  • The PredictionCheck transformer allows the validaton of the evaluations in an EvaluationContainer, e.g., checking whether they are within a predefined range.

  • spectrum filter for adding the output of a prebuilt Weka filter to the spectrum's report: WekaFilterToReport.

Stay tuned for the release!

Currently, I'm working mainly on a commercial project that involves meta-flows, i.e., flows that use their parametrization to generate flows on the fly that do the actual work, aka worker-flows. Hence there being many tweaks and changes based around this approach, e.g., making it easier to load models into memory and use them directly from there.

Fixes

• The TryCatch control actor now works again as expected within Rat, LocalScopeTransformer and LocalScopeTrigger actors. It also now properly stops its try and catch branches.

• Fixed rendering of arrays in the Preview browser, now each element has its own renderer determined. Necessary for mixed object arrays. The PlainText handler no longer crashes if it cannot read a file, one with binary content.

• The BaseDirectoryChooser dialog now correctly retrieves the currently selected directory for adding bookmarks. Also displaying a text field with the currently selected directory, enabling fast copy/paste.

Changes

• All help screens in the GUI have been centralized into a separate Help frame, which keeps a history of the screens.

• Remote commands can be sent and received now in JSON as well, using the JsonProcessor instead of the DefaultProcessor.

• The Serialize transformer and Deserialize sink now take advantage of the object writer/reader class hierarchies for more flexibility in output/input formats.

• Updated processoutput4j to 0.0.6

• The Command source actor now has a timeOut option, which kills the process once exceeded.

• adams-event: added template support for cron schedules: adams/core/base/CronSchedule.props

• adams-weka:

  • The attribute index is now displayed in the table header of the WekaInstancesDisplay sink and the object renderer at debugging time.

  • Added support for updating properties via variables to: WekaAssociatorSetup, WekaClassifierSetup, WekaClustererSetup, WekaDataGenerator, WekaFilter, WekaStreamFilter

  • simplified model loading, centralizing it in the adams.flow.core.AbstractModelLoader class, now offering loading from file, source actor and internal storage: WekaFilter, WekaClassifying, WekaClustering

  • The evaluations of classifiers in the Weka Investigator no longer require class values to be present on data sets other than the training set. This allows to simply make predictions then.

• adams-moa: simplified model loading, centralizing it in the adams.flow.core.AbstractModelLoader class, now offering loading from file, source actor and internal storage: MOAClassifying, MOAClustering, MOARegressing

• adams-meka: simplified model loading, centralizing it in the adams.flow.core.AbstractModelLoader class, now offering loading from file, source actor and internal storage: MekaClassifying

• adams-spectral-2dim-core:

  • The Cleaner transformer now adds the actual cleaner to the output container now as well.

  • The PostProcessor transformer can output a container now, which contains the input and output data, as well as the actual processor in use.

  • The spetrum's wave-numbers can be used as suffix now instead of the amplitude indices (affecting only the SimpleInstanceGenerator).

  • renamed IntensityImageSpectrumWriter to SpectrumImageWriter, uses new class hierarchy for image generators: adams.data.spectrumimage.AbstractSpectrumImageGenerator

• adams-rsync:

  • updated rsync4j to 3.1.2-5.

  • added maxTime option to RSync and SimpleRSync sources, which kills the rsync process once exceeded.

• adams-cntk-weka: The CNTKSaver converter now turns a nominal class attribute

  into CNTK's 1-hot encoding format (aka unsupervised NominalToBinary).

• adams-imaging: The ImageAnnotator transformer now allows the manual selection of objects via a selection processor.

Additions

• Added reader/writer for objects implementing adams.core.SerializableObject

• The object export in the debug's object tree or the Flow editor's debug view now allows you to export any serializable object as well.

• Added OutlierDetector transformer to directly tap into detection messages returned by detection scheme.

• Added the SetManyVariables standalone and transformer, which allow to update multiple variables at the same time.

• The DeserializeToStorage standalone simplifies loading serialized models into storage.

• adams-cntk:

  • Added handler for CNTK models to the Preview browser

• adams-cntk-weka:

  • Added Weka loader for CNTK text files: CNTKLoader

  • Added Weka saver for CNTK text file format: CNTKSaver

  • Added classifier for building a CNTK using a Brainscript file and using the final model for making predictions: functions.CNTKBrainscriptModel

Have a good weekend!