Updates 2019/11/08

Lots of minor changes and fixes happened over the last few weeks. But the most notable one is the generation of Debian and Redhat installer packages. These make it easier to install ADAMS on servers or in Docker images. Also, with the adams-maven-plugin it is now possible to compile flows into Java code during the build process.

Fixes

  • The use of databases other than MySQL in case the adams-db module is present now uses auto-detection again for determining the backend (using the JDBC URL). This avoids having to explicitly configure a DbBackend.props file.
  • The LocalScopeTransformer and LocalScopeTrigger actors now only clean up local variables and storage data structures if not using shared resources.
  • The forceVariables method in the AbstractActor (ancestor for pretty much all actosr) no longer calls cleanUp of current Variables instance: this caused havoc when dynamically instantiating sub-flows at runtime, linking them into the existing flow to have access to variables and storage, as they could remove the outer flow's variables altogether.
  • adams-net: The Base64ToString conversion now applies the selected decoding type.
  • adams-imaging:
    • The BinaryCrop cropping algorithm now works for objects of any shape, by finding the rectangle encompassing the center object.
    • The NegativeRegions transformer now sets the supplied object type.
    • The MinDimensions negative regions meta-algorithm no longer removes two objects at a time when logging is enabled.
  • adams-json: retrieving an array via $.expr no longer results in the elements being forwarded one-by-one, but as a JSONArray object (JSONArray also implements the List interface, which got interpreted incorrectly; $.expr can return more than one value).
  • adams-addons-all: now has a dependency on adams-tensorflow as well.

Changes

  • Flows that had errors when loading, the string " (incomplete)" is now appended to the filename, to mark them as such. Previously, the filename got lost and a simple "FlowXYZ" was used.
  • The TextFileReader now accepts InputStream objects as well.
  • upgraded java-utils dependency to 0.0.3
  • upgraded commons-compress dependency to 1.19 to address CVE-2019-12402 (https://nvd.nist.gov/vuln/detail/CVE-2019-12402)
  • The PromptUser template now allows configuring the restoration of settings as well.
  • The adams.core.logging.FileHandler logging output handler now makes use of the ADAMS_LOGFILE_PREFIX environment variable to inject a prefix into the log file (eg, "console.log" becomes "testing-console.log" with "ADAMS_LOGFILE_PREFIX=testing-"). This allows multiple log outputs from multiple ADAMS services on the same server.
  • The Preview browser now offers a search panel when viewing spreadsheets and no longer outputs exceptions in the console in case custom preview handlers cannot be instantiated (eg when switching from one ADAMS application to another).
  • The ImageAnnotator transformer now maintains last selected label, window position and size between invocations.
  • adams-json:
    • upgraded jsonpath dependency to 2.4.0
    • The StringToJson conversion how has an output type option for casting the JSON object (any, array, object).
  • adams-ml: The ActualVsPredictedPlot now exposes the overlays for the plot, allowing to add other ones in addition to the diagonal (StraightLineOverlay), like LinearRegressionOverlay.
  • adams-weka: upgraded multisearch-weka-package dependency to 2019.10.4
  • adams-weka and adams-weka-lts: The ActualVsPredictedPlot now exposes the overlays for the plot, allowing to add other ones in addition to the diagonal (StraightLineOverlay), like LinearRegressionOverlay.
  • adams-meta: The FlowFileReader now accepts InputStream and Reader objects as well (must be closed separately).
  • adams-rsync: The RSync and SimpleRSync sources now output the output (stdout/stderr)
    as it occurs rather than all of it after the rsync process finishes.
  • adams-spreadsheet: The SpreadSheetFileReader now accepts InputStream and Reader objects as well (must be closed separately).
  • All ADAMS applications are now available as Debian and RPM packages as well.

Additions

  • Added support for InputStream/Reader instance generation via the InputStreamGenerator/ReaderGenerator sources and the CloseInputStream/CloseReader sink. This allows loading files from the classpath, e.g., files that are packaged within the jars of an application.
  • Added BooleanToString and StringToBoolean conversions.
  • With the ForwardSlashSwitch actor processor, it is now possible to switch all ForwardSlashSupporter objects in one go.
  • adams-rest:
    • added RESTPlugin support for processing text and JSON with a callable transformer template ("pipeline"): CallableJsonPipeline, CallableTextPipeline.
    • added RESTplugin for just calling callable transformer with JSON or text: CallableJsonTransformer and CallableTextTransformer.
  • adams-maven-plugin: a Maven plugin that allows compilation of flows into Java code as part of the Maven build process.
  • adams-spectral-2dim-core: added the Export spectra... plugin to the SpreadSheet and Instances tables to allow export of selected rows as spectral files.
  • adams-weka and adams-weka-lts:
    • added the conversion MapToWekaInstance and WekaInstanceToMap to provide an easy way to convert Weka Instance objects to JSON and vice versa.
    • introduced a class hierarchy for building a final model after a cross-validation run of a classifier in the Weka Investigator.
    • added filter for extracting a range of instances from a dataset in the specified order: KeepRange.

Updates 2019/09/30

Didn't get around to posting it last Friday, as I ran out of time.

Notably are the refactoring underneath the hood, slimming down the adams-core module by moving out support for additional databases to adams-db, math-stuff to adams-math, XML processing to adams-xml, YAML support to adams-yaml and JSON handling to adams-json. Furthermore, the adams-rats module is now a meta-module, with the actual functionality split into adams-rats-core , adams-rats-net, adams-rats-rest and adams-rats-webservice. This allows for slimmer applications, e.g., when a simple REST service is required, there is no need to pull in SOAP related dependencies. These changes were triggered by generating a smaller standalone application for the data exchange server, which is to be used in conjunction with RabbitMQ (DEX is used for storing the actual payloads of the RabbitMQ messages).

Most HTTP-related queries are now handled using the requests4j library (https://github.com/fracpete/requests4j).

Fixes

  • The customColumnHeaders option of the CsvSpreadSheetReader now works also without having to check the noHeader option.
  • adams-net: HttpPostFile now uses correct CRLF in POST rather than just LF, according to specifications.
  • adams-weka and adams-weka-lts:
    • When the WekaFilter transformer initializes the actual filter, the error message no longer gets discarded; it is also possible to update properties with variables that access an array of the underlying filter.
    • The Associate, Attribute selection, Classify and Cluster tab in the Weka Investigator can once again get copied (deserialization of output generators was failing).
    • The SimpleArffLoader now supports attribute weights, fixed instance weights handling.

Changes

  • In the Debug control panel, it is now possible to hide programmatic variables in the Variables tab, by checking the Show user-only vars checkbox.
  • The AddActorHere/After/Beneath keyboard actions of the Flow editor now allow you to specify whether you want to edit this actor in a GOE window and if that is the case whether you want to see the class tree of actors.
  • Added methods contains(str, find) and replaceall(str, find, replace) to the BooleanExpression, MathematicalExpression and StringExpression parsers.
  • Report fields now allow inline editing in the GenericObjectEditor.
  • Large tables now render faster in general, by not enforcing to calulate the optimal width anymore.
  • Errors/warnings encountered during loading a flow are now displayed in the notification area rather than in separate dialogs.
  • Moved markdown support from adams-core to adams-html.
  • Removed JShell support (not really practical).
  • Moved terminal-based application support (using lanterna) into adams-terminal module.
  • Moved math-related libraries and functions into adams-math module.
  • Moved XML, XSLT, XPath related actors and code into adams-xml module.
  • Moved YAML related actors and code into adams-yaml module.
  • adams-rats: split into adams-rats-core, adams-rats-net, adams-rats-rest and adams-rats-webservice to be able to build more lightweight applications.
  • adams-rsync: upgraded rsync4j to 3.1.2-16
  • adams-html:
    • added requests4j library as dependency
    • removed JSoup dependecy
  • adams-net:
  • adams-spectral-2dim-core:
    • The Spectrum Explorer now allows the user to create a new window or to duplicate the current one. It now also allows to stop the execution of the current action via Scripts - Stop execution.
    • The SpectrumPaintlet now has the option paintAll to force drawing all points.
  • adams-spreadsheet:
    • the header and cell popup menu items for the spreadsheet table have been reworked to work more consistently across visible/selected rows.
    • The SpreadSheetColumnIterator can now output 1-based indices instead of names and also output the items as an array rather than one-by-one.
  • adams-weka and adams-weka-lts:
    • the header and cell popup menu items for the instances table have been reworked to work more consistently across visible/selected rows.
    • WekaInstancesInfo now allows outputting the name of the class attribute via CLASS_ATTRIBUTE_NAME
    • Blacklisted the ArffViewer and SqlViewer menu items.
    • Added support for per-fold output in Classify tab of the Weka Investigator
    • The Weka Investigator now allows changing attribute and instance weights.
    • upgraded version of datasets-weights-weka-package to 2019.9.13
    • removed cat swarm optimization support (CSO), since not working properly

Additions

  • The Flow editor received new keyboard actions: ActionQuickSearch (search for available menu items or tree popup actions for current context, invoked by Ctrl+Space), QuickSearchActorHere/After/Beneath (similar to SearchActorHere/After/Beneath, but uses list rather than tree).
  • Added a simpler, faster CSV spreadsheet reader which only supports text and doubles called FastCsvSpreadSheetReader, which is aimed at reading very large CSV files quickly.
  • Created the adams.core.TimeIt class for timing code executions. Its static methods generate timings for Runnable instances.
  • The InformationDialog sink allows displaying a simple message to the user.
  • With the adams.core.option.ApplyActorProducer, it is possible to generate Java code from actors that can be incorporated as methods in other projects.
  • Allowing export of sub-flows with the Export... menu item available from flow tree popup menu (eg generating Java code using the ApplyActorProducer).
  • adams-meta: the SpecifiedActor source outputs the actor identified by actor path in flow, useful when storing an actor setup in a file for reference (variables can be expanded).
  • adams-net:
    • added FTPDelete and SFTPDelete transformers
    • added conversions for base64 operations: Base64ToByteArray, Base64ToString, ByteArrayToBase64, StringToBase64.
  • adams-rabbitmq:
    • added support for a simple REST webservice that allows data exchange via tokens: adams.flow.rest.dex.DataExchange and adams.flow.rest.dex.DataExchangeServer.
    • added standalone for defining a data exchange server connection context: DataExchangeServerConnection.
    • added transformers for performing upload/download/remove operations on a data exchange server: DataExchangeServerDownload, DataExchangeServerRemove, DataExchangeServerUpload.
    • added (send/receive) converter for sending payloads via a data exchange server (which can take advantage of a DataExchangeServerConnection available through the flow context): DataExchangeServerBasedConverter.
  • adams-rest: added conversion JsonToObject and ObjectToJson for easy conversions of Java objects from/to JSON using Jackson's data-binding functionality (https://github.com/FasterXML/jackson-databind).
  • adams-spectral-2dim-core:
    • The ExtractIdAndTypeSpectrumReader spectrum reader allows you to post-process the sample ID and type using the specified ID and group extractors (eg using the filename from the report).
    • The Median multi-spectrum filter computes the median from the spectral points.
  • adams-spectral-3way-core: The ExtractIdAndTypeThreeWayDataReader 3-way data reader allows you to post-process the sample ID and type using the specified ID and group extractors.
  • adams-spreadsheet:
    • The content of individual spreadsheet cells can now be displayed via the View cell menu item or with the new Cell content viewer tab.
    • With the new Column stats tab it is possible to calculate some statistics on the fly from the currently selected column and rows (default is sum).
  • adams-weka and adams-weka-lts:
    • the content of individual instances cells can now be displayed via the View cell menu item.
    • added dependency for the data-dumper-weka-package

Updates 2019/08/02

For once, there were mainly changes and additions and no fixes. In terms of changes, these were mainly to improve usability, but MOA has finally been updated to the latest release. As for additions, the ability to generate word clouds is noteworthy.

Changes

  • The help dialog in the GenericObjectEditor now shows the help for the current object configuration, rather than from the object with default options.
  • The SimplePlot and SequencePlotter sinks accept arrays of SequencePlotContainer objects now as well (for more efficient plotting).
  • The Flow editor now allows the following operations on all selected flows to be executed at once: validate/run/stop/clear graphical output. When applying actor processors to a flow that generate graphical output (eg finding actors, listing variable occurrences), then these no longer show up in a separate dialog but as a tab on the right hand side.
  • The ExecuteJobs transformer has now the ability to override the number of threads used by the incoming JobRunner instance (but only if that one implements ThreadLimiter, of course).
  • The SpreadSheetTable (eg used by the Spreadsheet file viewer) now allows the plotting of multiple rows.
  • Added the envVarOptional flag to the SetVariable standalone: if false it will generate an error if the env var is not present. An empty env var will now generate an error if overrideWithEnvVar is set to true.
  • Reworked the database connection dialog (from the main menu) to allow connecting to multiple databases without having to disconnect active one first. Also added buttons for creating a new connection and removing an existing one.
  • Plots (like Weka's error plots, sequence plots, etc) now allow the user to copy/paste the ranges between windows to align them easily.
  • adams-moa: MOA was upgraded to 2019.05.0 (thanks to Corey)
  • adams-compress: The following transformer can handle byte arrays now as well: Bzip2/UnBzip2, GZIP/GUNZIP, Lzf/UnLzf, Lzma/UnLzma, Xz/UnXz, Zstd/UnZstd.
  • adams-rabbitmq: BasicAuthConnectionFactory and GuestConnectionFactory now have an option for specifying the virtualhost on the RabbitMQ server.
  • adams-spectral-2dim-core: upgraded jcamp-dx dependency to 0.9.6.
  • adams-spreadsheet:
    • The SpreadSheetPlotGenerator transformer can output arrays of plot containers now as well (for more efficient plotting).
    • The row statistics now offer an option to specify the range of columns to operate on.
    • The popup plugins for table rows now allow the user to specify the subset of columns to operate on (eg including only subset of columns in SimplePlot/JFreeChart/Histogram).
  • adams-weka and adams-weka-lts:
    • if the WekaFileWriter sink is configured to use the relation name as file name and the provided filename points to dir, then relation name is simply appended to that directory.
    • The InstancesTable (eg used by the Weka Investigator) now allows plotting of multiple rows.
    • The popup plugins for table rows now allow the user to specify the subset of attributes to operate on (eg including only subset of columns in SimplePlot/JFreeChart/Histogram).

Additions

  • The ArrayNormalizeRange array statistic scales the values to the specified lower/upper bound.
  • The ArrayStandardize array statistic transforms the values to have a mean of 0 and a stddev of 1.
  • The conversion StringToByteArray turns as string into a byte array using the specified encoding and the ByteArrayToString turns it back into a string, also using the specified encoding.
  • adams-applications: added the .-exec* script to the Debian packages, which allows the execution of any class or flow from the command-line.
  • adams-ml: added a handler in the Preview browser that can generate actual vs predicted plots from CSV files.
  • adams-net: The HttpPostFile transformer allows you to upload a file to a specified URL using HTTP POST as "multipart/form-data".
  • adams-nlp: added basic support for generating word clouds using the kumo library https://github.com/kennycason/kumo.
  • adams-spreadsheet:
    • added the SpreadSheetSupporterToSpreadSheet conversion which turns any object implementing SpreadSheetSupporter into a spreadsheet.
    • The ArrayStatistic plugin for the InstancesTable allows you to compute statistics for the selected rows.
  • adams-spectral-2dim-core: added optional cross-hair tracker, can be enabled/disabled through the plot's popup menu.
  • adams-spectral-3way-core: added rudimentary FitsLibsReader for reading LIBS (https://en.wikipedia.org/wiki/Laser-induced_breakdown_spectroscopy) spectra in FITS format (https://fits.gsfc.nasa.gov/).
  • adams-weka and adams-weka-lts:
    • The split generators BinnedNumericClassRandomSplitGenerator and BinnedNumericClassCrossValidationFoldGenerator take the class distribution into account by using a binning algorithm to generate similar distributed train/test splits. Grouped versions are available as well.
    • Added the ResidualsVsPredictor and ResidualsVsFitted output generators for classification to the Weka Investigator.
    • The ArrayStatistic plugin for the InstancesTable allows you to compute statistics for the selected rows.

10th Birthday

Happy birthday!

Today marks 10 years since the first code was committed to the ADAMS repository (back then still stored in an in-house subversion repository)!

Back then, ADAMS wasn't yet a standalone project, but merely the common code base between a project for processing GC-MS data and one for processing NIR data. The GC-MS research project never really took off, but the NIR one (whic started off as an unofficial side-project) was successfully turned into a commercial application. The code-based for this offshoot project was migrated to ADAMS data structures again over the last few years and a new version for processing NIR data has been successfully launched this year at Eurofins Agro in the Netherlands and Ravensdown/ARL here in New Zealand.

Recently, we embarked on better integrating deeplearning frameworks, such as Keras. These frameworks can (in theory) now be used as a regular Weka classifier thanks to the Pyro4 library which we use for exchanging data. You only need to implement the Python side of things for parsing the data and making use of it (predictions and/or training). This opens up doors for models that work better on certain domains, like data fusion, and still being able to integrate them with all the functionality within ADAMS.

Exciting times ahead!

Flow editor

The Flow editor, the heart of ADAMS, goes back further than the initial commit, has seen a constant evolution of features and appearance. But one thing that has never changed: never uses explicit connections between operators (or actors in ADAMS terms). It is more of a graphic programming language than other workflow engines, with flow control, variables and internal data storage. You could say that it is Scratch for Data Scientists. It leaves out the nitty-gritty details of the underlying APIs involved and allows you to concentrate on rapidly developing machine learning and data processing applications, which you can then deploy as Linux daemons or Windows services.

Flow editor from the (internal) 0.2.0 release (June 2011)

This version only allowed a single flow to be edited and only offered the list of actors as side panel.

/galleries/floweditor/0.2.0.thumbnail.png

Flow editor from the 18.12.0 release (Dec 2018)

Much has happened since then, multiple open flows, various context-sensitive tabs, locating of variables and storage items, much improved context-sensitive menus and object editor.

/galleries/floweditor/18.12.0.thumbnail.png

Powerful tools

Over the years, ADAMS has added its own set of tools to better support the users in their tasks. Below are some of them:

Weka Investigator

The Weka Investigator is a much more powerful tool than the vanilla Weka Explorer. It allows you to manage multiple sessions in one interface and each session can manage multiple files and multiple tabs for the same operation (e.g., classification). It allows you to predefine various outputs for evaluations (rather than manually requesting them via popup menus) and you can also compare them side by side.

/galleries/screenshots/weka_investigator.thumbnail.png

Preview browser

The Preview browser allows you to browse a directory and quickly display file content with various handlers. Inspect Weka models? Check! Display annotations for images? Check! Many more file types are supported.

/galleries/screenshots/previewbrowser-default.thumbnail.png

Spreadsheet file viewer

A built-in spreadsheet application that can handle many different types of file formats (depending on modules): CSV, fixed tabular, Gnumeric, ODF, MS Excel, ...

/galleries/screenshots/spreadsheet_file_viewer.thumbnail.png

Spreadsheet processor

You are performing the same operation on several spreadsheets all the time? You might want to check out the Spreadsheet processor, which allows you to extract, transform and load spreadsheets of various types.

/galleries/screenshots/spreadsheet-processor.thumbnail.png

SQL Workbench

Need to work with databases then you can use the SQL Workbench to connect to (theoretically) any database via JDBC and retrieve data from it.

/galleries/screenshots/sqlworkbench.thumbnail.png

File commander

Need a file manager that can handle 10s of thousands of files and doesn't need minutes to display them (like the Windows File Explorer) but mere seconds? Need to copy things from/to remote servers? Then the File commander tool might be for you. It sports a two-panel layout with optional filtering, that was inspired by similar tools (Norton Commander, Midnight Commander, Double Commander).

/galleries/screenshots/filecommander.thumbnail.png

Spectrum Explorer

For exploring spectral like NIR (near infrared), MIR (mid infrared), XRF (x-ray fluorescence), you can use the Spectrum explorer. It allows you to explore and filter the data and also apply PLS or PCA.

/galleries/screenshots/spectrum_explorer.thumbnail.png

3-way Heatmap Viewer

For multi-dimensional spectra, like 3D fluorescence spectra, you can use the 3-way heatmap viewer.

/galleries/screenshots/3way_heatmap_viewer.thumbnail.png

Updates 2019/07/10

It is not about quantity in this update, but more about the importance of the changes.

After several weeks of incremental development and testing, I have pushed out changes regarding random split and cross-validation fold generation. To avoid duplication of code, I implemented a general framework for such operations, which is now utilized across the board (which also fixes some bugs).

Fixes

  • The transformers BaseName and DirName now handle placeholder abbreviations ("${...}") correctly.
  • adams-weka and adams-weka-lts: by moving to the generalized framework for random split and cross-validation fold generation, a number of bugs in aligning additional attributes to predictions have been fixed.

Changes

  • upgraded jshell-scripting to 0.1.0.
  • upgraded pdfbox to 2.0.16 to address https://nvd.nist.gov/vuln/detail/CVE-2019-0228
  • adams-ml: the file-based dataset preparation schemes for train test split and cross-validation now make use of the new generalized framework for such operations.
  • adams-weka and adams-weka-lts: the random split and cross-validation fold generators (both default and grouped) now make use of the new generalized framework for such operations.

Additions

  • adams-ml: added a generalized framework for random split and cross-validation fold generation, to be utilized in other data domains. Makes use of the work implemented for binning algorithms.

Updates 2019/07/03

Been spending a lot of time lately on sorting out Debian package generation for Maven projects. The .deb packages available for download now make it easier to install ADAMS on Debian systems (and derivatives like Ubuntu). This includes installing them inside Docker containers. The MySQL JDBC driver is back to the old version due to a problem introduce with 6.x (my bug report is linked below).

Fixes

  • adams-weka and adams-weka-lts: The number of threads defined in the JobRunner used for cross-validation in the Classify tab of the Weka Investigator is no longer getting ignored.
  • adams-rats: fixed a race condition in the FileLister rat input scheme when used inside a Cron one, which occurred when the Cron was triggering faster than the file listing was moving the files, resulting in multiple outputs.

Changes

  • reverted MySQL driver back to 5.1.42, due to driver listing columns from table from any database that the database user has access to - file a bug report with MySQL regarding this https://bugs.mysql.com/bug.php?id=95717
  • upgraded processoutput4j to 0.0.10
  • upgraded jclasslocator to 0.0.14
  • obsolete provenance support has been completely removed.
  • The new ListSelectionValueDefinition class, to be used with the EnterManyValues source, is targed at pre-defined lists with a pre-selected default value (rather than just using the first list item as the DefaultValueDefinition only offers).
  • The SequentialDirectory, as used by the Sequence control actor (and therefore by a lot of other actors internally) has seen speed improvements for large sequences that contain multiple disabled/skipped actors. Code for storing output from last active actor in such a sequence has been optimized as well. In some cases, this reduced the actor execution overheads by 20 percent. NB: these overheads may be trivial to the other operations executed in the sequence itself. In order for this to work properly, variable support has been disabled for the skip option of actors.
  • introduced lenient flag in AbstractDataContainerDbReader which only outputs warnings instead of generating errors when containers cannot be loaded from DB.
  • Class names (ADAMS and JDK ones) are now better hyperlinked in the GUI help frame.
  • adams-video: updated vljc dependency to 3.12.1 (VLC player works again on Linux Mint 19.1).

Additions

  • The ArrayBinning statistic applies selected binning algorithm/postprocessing.
  • added module adams-scripts-debian for generic scripts to be used in debian packages
  • adams-applications: after forking the abandoned debian-maven project (http://debian-maven.sf.net/ to https://github.com/fracpete/debian-maven-plugin) and reworking the code base extensively, support for generating .deb install packages for Debian was added. and Ubuntu, using deb:package maven target.
  • adams-meta: The CurrentFlow source outputs the flow itself, with optional expansion of variables using their current state. Useful when trying to store the state of a flow at a certain time or when auto-generating flows.
  • adams-spreadsheet:
    • With the MakeJFreeChartDataset transformer, you can generate datasets for JFreeChart plots or file writers.
    • The StorageJFreeChartDataset transformer allows you to add data series to an existing JFreeChart dataset in storage, e.g., when combining plots from multiple spreadsheets.

Updates 2019/05/24

A big push towards support for distributing jobs from anywhere within ADAMS that make use of the JobRunner interface has been happened in the last few weeks. Thanks to the new module offering RabbitMQ functionality (adams-rabbitmq, https://www.rabbitmq.com/), it is now possible to send jobs to a job queue on a RabbitMQ server and slave machines running flows for processing jobs can poll this job queue. Since the RabbitMQ server does not need to know about any slaves, it makes it easy for scaling up, by simply adding more slaves that pull jobs off the queue. Encrypting the communication via SSL/TLS is posible as well.

Furthermore, (experimental) SSL/TLS support is now available in the adams-rest module thanks to the new adams-security module.

And finally, it is now possible to monitor the memory consumption from within a flow and send out notifications (eg emails) in case the available free memory gets low over prolonged periods of time (MemoryMonitor standalone). This is something that background flows should make use of.

Fixes

  • Pasting a flow from the clipboard as new flow now works again. Now handles plain text nested format, command-line and nodes format.
  • Graphical flow components (like interactive actors such as SelectArraySubset) now work again when execute via the active flow functionality, but not being visible.
  • The JVisualVM menu item under Help (displayed in Developer user mode or higher) now checks whether the JVisualVM binary is actually present (no longer bundled with JDK after Java 8).
  • adams-weka and adams-weka-lts: ClassificationViaRegressionD classifuer now sets the class value to missing at prediction time before pushing through the filter, as MakeIndicator is taking the class value into account generating an exception if the value is invalid.

Changes

  • Upgraded jclasslocator dependency to 0.0.13
  • Upgraded java-utils dependency to 0.0.2
  • Upgraded MySQL JDBC driver to 8.0.16
  • Positions and sizes of the main tool frames are now recorded and restored.
  • The FindInFiles tool can search compressed files now as well (as long as the module adams-compress is present).
  • The SetMapValue now offers a built-in conversion for converting the value into the appropriate type, if necessary (eg string into integer).
  • The MemoryUsage source now also outputs init and max for heap and non-heap.
  • JMap and JDeps now support inline variables in the additional options parameter.
  • adams-rest:
    • experimental SSL/TLS support (only tested on self-signed certificates so far).
    • expanded all the missing sections in the manual.
  • adams-weka and adams-weka-lts:
    • The Weka Investigator now allows the definition of a JobRunner instead of #threads to allow for remote execution, eg via RabbitMQ. #threads can be configured within the LocalJobRunner (the default setting).

Additions

  • Javadoc for all classes is now available at: Javadoc
  • The FindInFile transformer allows to search for a text string in files, using the same functionality as the FindInFiles tool.
  • The ColorPerRegExpGroup meta-data color scheme, allows coloring based on a group string extracted from a label using a regular expression (eg parts of a sample ID).
  • Added enclose/pull up menu items in the GenericObjectEditor's popup menu for boolean conditions, conversions, spreadsheet row/column finders.
  • Added simple class hierarchy for sending notifications via SendNotification sink.
  • The MemoryMonitor standalone sends out a notification if the heap usage exceeds the specified threshold, making use of the new notification scheme class hierarchy.
  • Added support for generating heapdumps from within ADAMS, via main menu or through the HeapDump control actor.
  • The CollectionSubset transformer allows you to generate a subset collection, with only the elements from the specified indices.
  • Added a multi-page action to the Flow editor for comparing two flows with each other.
  • adams-meta: added actor for executing the actor passing through: ExecuteActor.
  • adams-security: new module for providing security related support, like SSL context.
  • adams-rabbitmq: new module for making use of the RabbitMQ message broker (https://www.rabbitmq.com/). Also includes a job runner for distributing jobs and actors for executing sub-flows remotely.
  • adams-rabbitmq-rats: new module for adding RabbitMQ consume/publish functionality to the RATS framework.
  • adams-weka and adams-weka-lts: Added enclose/pull up menu items in the GenericObjectEditor's popup menu for row/column finders working on Instances.
  • adams-spreadsheet: The Spreadsheet processor now has a Passthrough processor, if nothing needs to be done to the data and a Database target for storing the processed data back in a database table.

Updates 2019/04/30

Well, I didn't quite manage to get this post out the door before going on holiday over Easter. Collection handling is much improved in the flow, a new tool for handling spreadsheets, the Spreadsheet processor, has been added, a new merge transformer for spreadsheets added (SpreadSheetMethodMerge), and lots of little fixes and improvements.

Fixes

  • Added the -close-connection option to the DatabaseConnection standalone, which closes the connection one the flow wraps up. This is required for SQLite to release the database file properly. Otherwise ADAMS has to be restarted.
  • Errors and warnings that occur when processing the first line in the compact flow format are now being propagated properly, i.e., displayed in the user interface. External actors cano now report these errors again.
  • Pasting actors in the Flow editor now updates the display again to show the modified state.
  • The Debug execution listener in the flow now ensures that it is visible when a breakpoint is triggered.
  • The Find usages action from the Flow editor's popup menu now also finds nested variables and storage names. It also allows jumping to actors now that are in external flows, which don't have a root Flow actor and therefore miss a UI parent component context.
  • adams-imaging: The additionalObject option of the ImageObjectOverlap transformer now works when no label key was provided.
  • adams-spreadsheet: charts used by the Spreadsheet file viewer now plot all the data points with differing X/Y pairs, not just with differing X.
  • adams-weka and adams-weka-lts:
    • The Multi-Experimenter now handles the split generators correctly in the user interface, the train/test split experiment now works again, some minor user interface bugs fixed.
    • Hermione now stops the run if there are no optimization handlers defined, e.g., when loading invalid classnames.
    • In the Preprocess tab of the Weka Investigator, you can now filter the attribute names using the provided filter text field.

Changes

  • added flow_start_timestamp programmatic variable
  • upgraded RSyntaxTextArea dependency to 2.6.1
  • upgraded matrix-algorithms dependency to 0.0.15
  • removed the Quick edit tab in the Flow editor, due to strange lockups in the GenericObjectEditor.
  • The ReportHandler in the Preview browser now allows the report file reader to be configured, to display a wider range of report file formats.
  • adams-spreadsheet:
    • The DefaultXY JFreeChart dataset now allows X to be empty, uses row index for X in this case.
    • The SpreadSheetDisplay now accepts objects implementing the SpreadSheetSupporter interface as well.
  • adams-weka and adams-weka-lts:
    • The Instance tab in the Weka Investigator now allows the selection of an attribute to use for coloring instances (each unique value represents a color).
    • On the Preprocess tab in the Weka Investigator, you can now select the menu item Use selected as class from the split button of the attribute selection panel. Batch-filtering now only works if the datasets are actually compatible. The table listing the attribute names now also lists their types in a separate column.
    • Any GOE panel handling Weka classifiers now has an additional popup menu item called Enclose, which allows you to enclose the current classifier within a SingleClassifierEnhancer scheme (eg FilteredClassifier).
  • adams-heatmap:
    • The Heatmap class now implements SpreadSheetSupporter;
    • The HeatmapToSpreadSheet conversion now simply calls the toSpreadSheet() method.

Additions

  • Added the ProgrammaticTokenListener transformer for hooking into flow with non-flow code, allowing the listening to the data flowing through, e.g., for updating Swing user interface components.
  • Added the ArrayCovariance array statistic.
  • Added jshell-scripting dependency and JShell menu item (you need to switch to Developer mode). In order to execute a script it needs to be saved first.
  • Added ArrayToCollection transformer for turning an array into a collection.
  • Added CollectionAppend transformer to append a collection in storage with the one passing through.
  • Added CollectionProcess control actor for processing collections similar to ArrayProcess.
  • adams-imaging:
    • added the ColorProvider transformer which outputs a Color object whenever a token passes through.
    • added PngIsComplete file in use check scheme (looks for IEND end of file marker).
  • adams-spreadsheet:
    • The SpreadSheetMethodMerge transformer allows merging of spreadsheets via a class hierarchy of merge algorithms, similar to the SpreadSheetMerge transformer.
    • The Spreadsheet processor tool allows you to easily load spreadsheet data (file, database, clipboard), process it (query, flow) and output it (file, chart, clipboard). It also allows you to save configurations to easily re-use processing in certain scenarios.
  • adams-pdf: The PDFGenerate transformer allows PDF generation using specified generator scheme. Similar to PDFCreate, but depending on the generator, is not limited to processing files.
  • adams-heatmap: added conversions for computing correlation and covariance between columns of a spreadsheet: ColumnCorrelationToHeatmap and ColumnCovarianceToHeatmap.
  • adams-weka and adams-weka-lts:
    • The Classify tab of the Weka Investigator now has tree visualization output and GraphML output using the JGraphT library.
    • Added MSLE (mean squared logarithmic error) evaluation metric.
    • Added MSLE to Hermione as measure to be used for evaluation.
    • Added evaluation post-processors to remove worst predictions: RemoveWorst (using percentage), RemoveWorstStdDev (based on mean and stdev of actual class values).
  • adams-spectral-2dim-core:
    • added the DeleteDbSampleDataValue transformer for removing specific fields from the database for a sample ID.
    • added the SampleDataValueDbWriter transformer for writing specific sample data fields to the database.

Updates 2019/03/15

Fixed a bunch of little bugs and also worked on many minor improvements in the user interface.

A new Weka classifier got added: XGBoost, though currently only available for Linux/Mac.

The database backend within ADAMS has been further abstracted and now allows mirroring (ie replicating the queries on another server). Useful when trying to use a backup database server.

Fixes

  • Pasting of actors no longer generates long name suffixes consisting of (x) (y) (z) ...
  • EnterManyValues: date types now get inserted with their string representation into a map.
  • Histoy lists, like used in the Weka Investigator, now prompt the user when selecting Remove all from the popup menu, whether they really want to remove X entries.
  • The generates() method of the StringConstants source now outputs the class that the selected conversion generates (proper compatibility check).
  • The Close/cleanup and Remove graphical output actions in the Flow editor now close any Debug views as well.
  • Row plots in spreadsheet tables (eg Spreadsheet file viewer) no longer sub-sample points if the same number of points was selected.
  • Fixed incorrect path handling when placeholders present: Command, Exec, Java, SimpleStringReplace, StringInsert, StringReplace.
  • Command source now launches the commands again.
  • adams-python: fixed incorrect path handling when placeholders present: PipExec, PythonExec, PythonVenvExec.
  • adams-spreadsheet:
    • fixed incorrect path handling when placeholders present: SpreadSheetInsertColumn, SpreadSheetInsertRow, SpreadSheetReplaceCellValue.
    • entering multiple values through the GenericArrayEditor now works again in the Spreadsheet file viewer (Java decided to place the dialog behind all other windows).
  • adams-rats: fixed incorrect path handling when placeholders present: Exec (rat input and output).
  • adams-weka and adams-weka-lts: Row plots in instances tables (eg Weka Investigator) no longer sub-sample points if the same number of points was selected.

Changes

  • Changed the Compare actors menu item in the Flow editor's tree popup menu into a sub-menu with Just actors (original functionality) and Sub-flows (additional) as menu items.
  • Added support for unordered ranges: adams.core.UnorderedRange
  • External actor files are now being monitored and reloaded/reexpanded if necessary through the GenericObjectEditor.
  • The ArrayProcess control actor now has an optional variable for storing the index of the current element being processed (1-based index).
  • Added FILE_FORWARD_SLASHES variable type, which generates a filename with forward slashes. Affects transformers/standalones: SetVariable, SetManyVariables.
  • The Switch control now has a lenient option, which drops the requirement of the token having a payload (eg the Start source generates a null token, simply to kick off things).
  • Removed module adams-phmm4weka.
  • adams-spreadsheet:
    • Added support for unordered ranges: adams.data.spreadsheet.SpreadSheetUnorderedColumnRange
    • The SpreadSheetReorderColumns transformer now uses SpreadSheetUnorderedColumnRange for its order option.
    • The SpreadSheetJoinColumns conversion now uses SpreadSheetUnorderedColumnRange for its columns option, allowing for out-of-order joins.
    • Disabled Spreadsheet Explorer in the menu for the time being.
    • Added support for overlaying a diagonal in the scatter plot (via a second data series): enable addDiagonalSeries in the DefaultXY dataset. Affects JFreeChartPlot and JFreeChartFileWriter.
    • Added History button to the Query tab.
  • adams-weka and adams-weka-lts:
    • Added support for unordered attribute ranges: adams.data.weka.WekaUnorderedAttributeRange
    • The Weka Investigator now allows moving its tabs (like Preprocessing and Classify) via the Move tab to submenu in the tab's popup menu.
  • adams-spectral-2dim-core:
    • added ability to Update Sample data dialog to work off sample IDs obtained from a spreadsheet.
    • SpectrumT and SampleDataT are no longer directly accessed, but through facades, which allows swapping out of backend (e.g., introduce mirroring).
  • adams-rats:
    • The RatControl actor now has a Console button in the user interface, which allows the display of the last X lines that occurred in the console.
    • The type of log object generated from errors and sent to the log actor (if available) is now defined via the AbstractLogObjectGenerator class hierarchy.
  • adams-cntk-weka and adams-cntk-weka-lts: moved CNTKPyroProxy classifier from into new module adams-python-weka(-lts) and renamed it to PyroProxy, as it is framework-agnostic.
  • adams-pdf: The PDF Viewer switched from multi-tab layout to multi-page.

Additions

  • Added the DeleteVariable and DeleteStorageValue standalones.
  • Added the Configure option handler tool (developer mode) that allows configuring any option handler.
  • adams-weka and adams-weka-lts: i
    • added the GraphSource serialized object preview handler for weka.core.Drawable objects, which outputs the graph source code (dot or XML BIF).
    • added the GraphSource output for the Investigator's Classify/Cluster tabs, which outputs the graph source (dot or XML BIF) for weka.core.Drawable classifiers/clusterers.
    • added WekaDatasetsMerge transformer, an alternative to WekaInstancesMerge for merging multiple Weka Instances objects into one (side-by-side). This transformer can be selected in the Weka Investigator now as well for merging datasets (Preprocess panel, data table actions).
    • The XGBoost classifier got added (using https://github.com/dmlc/xgboost) - currently only for Linux/Mac.
    • The Weka Multi-Experimenter now has a table-based results panel as well.
  • adams-spectral-2dim-core:
    • added List Sample data dialog which lists all sample data for the selected spectra in a single table.
    • added Delete Sample data dialog which allows removal of specified sample data values from the selected spectra.
    • no longer storing Dummy report flag in sample data table.
  • adams-spreadsheet:
    • spreadsheet tables now allow sorting on multiple columns using via the Sort... menu item of the table's popup menu.
    • added query panel for meta-data to the SQL Workbench tool

Updates 2019/01/30

A lot of minor UI improvements and dependency refreshments happened since the release in December last year. New in the flow is EXIF tag support and communication with Python processes across the network using Pyro4 (Python Remote Object, https://github.com/irmen/Pyro4).

Fixes

  • Multi page panes are now showing border again when focused, to make it easier to see which widget has the focus.
  • The Preview browser did not maintain correctly its internal list of content handlers, resulting in apparently random preview handlers getting selected.
  • The SQL Workbench now disables the user interface while a query is being executed.
  • Chooser panels (eg for dates, files or colors) now change the font color to RED if the entered value is not in a valid format and will revert to the default value when opening the dialog via the ... button. The Use default menu item was added to allow the user to revert back to the default value.
  • EnterManyValues now shows the tool tip of the underlying component if there was no explicit help string entered.
  • The sources EnterValue, EnterManyValues, SelectDirectory and SelectFile now honor any information recovered from restoration files during non-interactive and/or headless operation.
  • adams-weka and adams-weka-lts: the transformers WekaFilter, WekaClassifying and WekaClustering are now threadsafe when using the internally used Weka object (previously, when using a shared model in storage in concurrent situations, it could end up in an unsuable state).
  • adams-net: upgraded tika-core to 1.20 due to https://nvd.nist.gov/vuln/detail/CVE-2018-17197
  • adams-ml: The CrossValidation scheme for preparing file-based datasets now generates correct leave-one-out datasets.
  • adams-groovy: Upgraded Groovy to 2.5.5 to make the Groovy Console work with Java versions 9 and later.
  • adams-video: fixed user interface issues in VLCjPlayer and Annotator

Changes

  • Retired the obsolete NamedSetup framework.
  • The Find in files tool now has all its operations in a button with a drop-down list, using a class hierarchy. New actions are: Copy to... and Move to.... The View... action now uses the Preview browser.
  • How graphical actors are being displays is now handled by a class hierarchy, replacing the displayInEditor option. The default scheme (appropriately named Default) just displaying it on top of all windows. With DisplayInEditor, the window gets displayed in Flow editor window. NoDisplay suppresses the display completely and Background displays behind all other windows. The ChangeDisplayType actor processor allows you to change the type(s) to a new one.
  • The menu item MakeInteractive of the Flow editor tree popup allows one to turn one or more SetVariable standalones into an interactive sub-flow (Trigger/EnterManyValues/MapToVariables).
  • The DefaultValueDefinition used by EnterManyValues now offers an option to output paths (file or directory) with forward slashes.
  • All value definitions, as used by EnterManyValues can be now disabled/enabled, which makes it easier to maintain testing/production ones side by side.
  • The LocalScopeTransformer and LocalScopeTrigger control actors now add the programmatic variables to their local scope when using EMPTY for variable propagation: flow_filename_long, flow_filename_short, flow_filename_restore, flow_dir, flow_id, is_headless, has_gui, project_name, project_home, project_modules
  • The JsonFileReader can now enforce output type (OBJECT or ARRAY) if know beforehand, default is ANY (no enforcement)
  • Added ability to flag a flow as active and manage that one through the Active menu and the buttons in the toolbar that have the little A in the bottom right corner. This is useful when editing external flows, as the switching back to the main flow is now no longer necessary.
  • The pause/resume and stop buttons are now also available through a debug flow perspective.
  • Removed the Run/Debug flow notification area actions.
  • adams-spreadsheet: the sinks JFreeChartFileWriter and JFreeChartPlot now allow the selection of shapes for the data point markers, the background color is now white (aligned with other ADAMS plots) and the plot color can be selected as well.
  • adams-weka and adams-weka-lts:
    • The WekaEvaluationSummary transformer now has options for custom titles for summary, confusion matrix and class details.
    • Updated Weka dependencies to 3.9.3-fork-0.0.4 and 3.9.0-fork-0.0.9 respectively (fixes duplicate output when using Logback or log4j; AbstractClassifier.forName no longer casts to AbstractClassifier, but Classifier).
  • adams-jython: upgraded Jython to 2.7.1b3.

Additions

  • adams-imaging:
    • added image reader/writer using the Apache Commons Imaging library, added preview handlers for images and meta-data using this library as well. This will eventually replace the Sanselan-based classes.
    • The ExifTagOperation transformer allows you to work on the EXIF data in files, e.g., read/write/remove fields or check for fields. The HasExifTag boolean condition checks whether the specified tag is present.
  • adams-python:
  • adams-maps: added GPSDistance conversion for computing distance (in meters) between two GPS points.