Updates 2019/07/03

Been spending a lot of time lately on sorting out Debian package generation for Maven projects. The .deb packages available for download now make it easier to install ADAMS on Debian systems (and derivatives like Ubuntu). This includes installing them inside Docker containers. The MySQL JDBC driver is back to the old version due to a problem introduce with 6.x (my bug report is linked below).

Fixes

  • adams-weka and adams-weka-lts: The number of threads defined in the JobRunner used for cross-validation in the Classify tab of the Weka Investigator is no longer getting ignored.

  • adams-rats: fixed a race condition in the FileLister rat input scheme when used inside a Cron one, which occurred when the Cron was triggering faster than the file listing was moving the files, resulting in multiple outputs.

Changes

  • reverted MySQL driver back to 5.1.42, due to driver listing columns from table from any database that the database user has access to - file a bug report with MySQL regarding this https://bugs.mysql.com/bug.php?id=95717

  • upgraded processoutput4j to 0.0.10

  • upgraded jclasslocator to 0.0.14

  • obsolete provenance support has been completely removed.

  • The new ListSelectionValueDefinition class, to be used with the EnterManyValues source, is targed at pre-defined lists with a pre-selected default value (rather than just using the first list item as the DefaultValueDefinition only offers).

  • The SequentialDirectory, as used by the Sequence control actor (and therefore by a lot of other actors internally) has seen speed improvements for large sequences that contain multiple disabled/skipped actors. Code for storing output from last active actor in such a sequence has been optimized as well. In some cases, this reduced the actor execution overheads by 20 percent. NB: these overheads may be trivial to the other operations executed in the sequence itself. In order for this to work properly, variable support has been disabled for the skip option of actors.

  • introduced lenient flag in AbstractDataContainerDbReader which only outputs warnings instead of generating errors when containers cannot be loaded from DB.

  • Class names (ADAMS and JDK ones) are now better hyperlinked in the GUI help frame.

  • adams-video: updated vljc dependency to 3.12.1 (VLC player works again on Linux Mint 19.1).

Additions

  • The ArrayBinning statistic applies selected binning algorithm/postprocessing.

  • added module adams-scripts-debian for generic scripts to be used in debian packages

  • adams-applications: after forking the abandoned debian-maven project (http://debian-maven.sf.net/ to https://github.com/fracpete/debian-maven-plugin) and reworking the code base extensively, support for generating .deb install packages for Debian was added. and Ubuntu, using deb:package maven target.

  • adams-meta: The CurrentFlow source outputs the flow itself, with optional expansion of variables using their current state. Useful when trying to store the state of a flow at a certain time or when auto-generating flows.

  • adams-spreadsheet:

    • With the MakeJFreeChartDataset transformer, you can generate datasets for JFreeChart plots or file writers.

    • The StorageJFreeChartDataset transformer allows you to add data series to an existing JFreeChart dataset in storage, e.g., when combining plots from multiple spreadsheets.

Updates 2019/05/24

A big push towards support for distributing jobs from anywhere within ADAMS that make use of the JobRunner interface has been happened in the last few weeks. Thanks to the new module offering RabbitMQ functionality (adams-rabbitmq, https://www.rabbitmq.com/), it is now possible to send jobs to a job queue on a RabbitMQ server and slave machines running flows for processing jobs can poll this job queue. Since the RabbitMQ server does not need to know about any slaves, it makes it easy for scaling up, by simply adding more slaves that pull jobs off the queue. Encrypting the communication via SSL/TLS is posible as well.

Furthermore, (experimental) SSL/TLS support is now available in the adams-rest module thanks to the new adams-security module.

And finally, it is now possible to monitor the memory consumption from within a flow and send out notifications (eg emails) in case the available free memory gets low over prolonged periods of time (MemoryMonitor standalone). This is something that background flows should make use of.

Fixes

  • Pasting a flow from the clipboard as new flow now works again. Now handles plain text nested format, command-line and nodes format.

  • Graphical flow components (like interactive actors such as SelectArraySubset) now work again when execute via the active flow functionality, but not being visible.

  • The JVisualVM menu item under Help (displayed in Developer user mode or higher) now checks whether the JVisualVM binary is actually present (no longer bundled with JDK after Java 8).

  • adams-weka and adams-weka-lts: ClassificationViaRegressionD classifuer now sets the class value to missing at prediction time before pushing through the filter, as MakeIndicator is taking the class value into account generating an exception if the value is invalid.

Changes

  • Upgraded jclasslocator dependency to 0.0.13

  • Upgraded java-utils dependency to 0.0.2

  • Upgraded MySQL JDBC driver to 8.0.16

  • Positions and sizes of the main tool frames are now recorded and restored.

  • The FindInFiles tool can search compressed files now as well (as long as the module adams-compress is present).

  • The SetMapValue now offers a built-in conversion for converting the value into the appropriate type, if necessary (eg string into integer).

  • The MemoryUsage source now also outputs init and max for heap and non-heap.

  • JMap and JDeps now support inline variables in the additional options parameter.

  • adams-rest:

    • experimental SSL/TLS support (only tested on self-signed certificates so far).

    • expanded all the missing sections in the manual.

  • adams-weka and adams-weka-lts:

    • The Weka Investigator now allows the definition of a JobRunner instead of #threads to allow for remote execution, eg via RabbitMQ. #threads can be configured within the LocalJobRunner (the default setting).

Additions

  • Javadoc for all classes is now available at: Javadoc

  • The FindInFile transformer allows to search for a text string in files, using the same functionality as the FindInFiles tool.

  • The ColorPerRegExpGroup meta-data color scheme, allows coloring based on a group string extracted from a label using a regular expression (eg parts of a sample ID).

  • Added enclose/pull up menu items in the GenericObjectEditor's popup menu for boolean conditions, conversions, spreadsheet row/column finders.

  • Added simple class hierarchy for sending notifications via SendNotification sink.

  • The MemoryMonitor standalone sends out a notification if the heap usage exceeds the specified threshold, making use of the new notification scheme class hierarchy.

  • Added support for generating heapdumps from within ADAMS, via main menu or through the HeapDump control actor.

  • The CollectionSubset transformer allows you to generate a subset collection, with only the elements from the specified indices.

  • Added a multi-page action to the Flow editor for comparing two flows with each other.

  • adams-meta: added actor for executing the actor passing through: ExecuteActor.

  • adams-security: new module for providing security related support, like SSL context.

  • adams-rabbitmq: new module for making use of the RabbitMQ message broker (https://www.rabbitmq.com/). Also includes a job runner for distributing jobs and actors for executing sub-flows remotely.

  • adams-rabbitmq-rats: new module for adding RabbitMQ consume/publish functionality to the RATS framework.

  • adams-weka and adams-weka-lts: Added enclose/pull up menu items in the GenericObjectEditor's popup menu for row/column finders working on Instances.

  • adams-spreadsheet: The Spreadsheet processor now has a Passthrough processor, if nothing needs to be done to the data and a Database target for storing the processed data back in a database table.