Updates 2016/08/26

All lot of effort has gone into making ADAMS work in a headless environment, like on a Linux server in the cloud. Thanks to Lanterna, there is now a simple, terminal-based interface available for such environments. A number of interactive actors can be used in headless environments now as well.

Fixes

  • The ManageInteractiveActors now enables/disables interactive actors correctly.

Changes

  • Added support for interactive actors to be used in a headless environment (e.g., in a terminal on a server), with the following actors supporting this now: EnterValue, SelectCharset, SelectDateTime, SelectDirectory, SelectFile, WekaSelectDataset, ConfirmationDialog, PasteFromClipboard. The following actors can now ask for the password in the console as well: DatabaseConnection, FTPConnection, SMTPConnection, SSHConnection.
  • The adams.flow.FlowRunner class now offers a -non-interactive flag, that ensures that all interactive actors avoid interaction with the user. Useful for flows that can be run with user interaction through GUI or command-line and also as background process.
  • Added support for gzip-compressed Properties files: .props.gz
  • adams-weka: The Weka Investigator now has a Cluster tab as well as a tab in the system-wide Preferences (Program -> Preferences).

Additions

  • Added support for customizable enumerations: adams.core.ConfigurableEnumeration.
  • The adams.terminal.Main class allows you to run flows in a terminal, i.e., without a graphical user interface. Useful when working in a headless environment, like remotely on Linux servers.
  • Introduced a simple file manager called File commander inspired by the Midnight Commander.
  • adams-imaging: added JpegIsComplete file-use check algorithm, which determines whether a JPEG is fully written to disk.
  • adams-spreadsheet:
    • With the SpreadSheetMatrixStatistic you can now calculate statistics for a whole spreadsheet (or a defined subset).
    • The SimpleStreamSpreadSheetReader/Writer pair allows you to store spreadsheet data alongside their types. Sparse format and chunking is supported as well.
    • With the SpreadSheetToCreateTableStatement conversion it is easy to create a SQL CREATE TABLE statement.
  • adams-weka: With WekaClusterAssignments you can output the cluster assignments (i.e., which instance got assigned to what cluster) from the last evaluation as spreadsheet.

Updates 2016/08/15

ADAMS has been in the German news again: JAXenter (Google translation)

Fixes

  • Data loaded in the SpectrumExplorer, InstanceExplorer, etc now keeps its associated color, even when being filtered or transformed.
  • Fixed StackOverflow error caused by AbstractActor.isHeadless() method when using a graphical actor by itself outside a flow.
  • Fixed a headless environment exception adams.gui.application.DatabaseConnections.
  • adams-imaging: the ChangeCanvasSize BufferedImage transformer no longer loses the meta-data.
  • adams-weka: fixed a cast exception in the WekaInstanceViewer sink.

Changes

  • Added support for trainable outlier detectors.
  • DirectoryLister, FileSupplier and FileSystemSearch sources now offer output of files with forward slashes (to avoid \t problems with variables).
  • ReplaceFileExtension and StringToValidFileName conversion can output forwards slashes now as well.
  • The Placeholder management dialog now has two tabs, one for the user-defined placeholders and one listing the system ones (CWD/HOME/TMP).
  • The SetReportFromFile and SetReportFromSource transformers now allow merging as well, instead of just replacing the existing report
  • Centralized headless environment checks in GUIHelper.isHeadless().
  • The run/run.bat files have been trimmed down and the actual logic is now handled by the launcher.sh/launcher.bat scripts in the new adams-scripts module. Reduces duplication to a bare minimum.
  • adams-meta: CopyCallableX actors now replace themselves in the flow with the copy of the callable actor rather than being kept internally. The callable actor can be marked as optional now as well. In case of the callable actor not being available, a dummy acctor is used instead.
  • adams-net: Upgraded tika-core to 1.13.
  • adams-rats:
    • The Rat actor can be started in paused mode now.
    • Fixed a problem with the RatPlague actor that variables were not being propagated correctly; the output queue is now optional as well.
  • adams-spreadsheet: added support for custom type mappers (spreadsheet cell type <-> SQL type).
  • adams-weka: the WekaInstancesDisplay sink now uses the improved table/model from the Weka Investigator, which includes a search panel now.

Additions

  • Added PropertiesToMap and ReportToMap conversions.
  • Added Null dummy standalone and source actors.
  • Added class hierarchy for checking whether files are in use: adams.core.io.fileuse.AbstractFileUseCheck
  • adams-imaging:
    • The JAI transformer ChangeOrientation ensures that images have the specified orientation (landscape or portrait).
    • With the ImageAnnotator, the user can interactively label objects in an image that an object locator determined.
  • adams-net: The Socket sink and source actors allow you to send and receive strings and byte arrays via sockets.
  • adams-rats: The ChangeRatState control actor pauses/resumes the specified Rat actors from within the flow. Allows to initially start up a Rat actor in paused mode and then activate it later on (for instance, after user feedback).
  • adams-spectral-core: added AmplitudeRange outlier detector, which determines its accepted range based on the training spectra. Flags any spectrum that falls outside the ranges.
  • adams-weka: The new Weka Investigator tool is aimed at providing a more versatile alternative to the Weka Explorer. Being able to handle multiple sessions in parallel, multiple datasets per session (e.g., train and test set), using tabs instead of windows, pre-defined outputs to be generated from evaluations, and lots more. At this stage, only classification is available (apart from Data and Preprocessing tabs), with more functionality coming bit by bit.

Critical memory leak fixed!

Well, this week saw some important bug fixes, notably a critical memory leak in logging and deeplearning4j now working on Windows. See below for details.

Fixes

  • Fixed memory leak in the logging framework: Java's logging facility doesn't garbage collected Logger instances and slowly gobbles up memory. Reimplemented the functionality to avoid this issue. Thanks to Sam for uncovering that!
  • Fixed passing on of data for control actors CallableActorScreenshot and ClearCallableDisplay.
  • adams-spreadsheet:
    • NewSpreadSheet now accepts an empty initial column list.
    • SpreadSheetInsertColumn and SpreadSheetInsertRow now ignore the position if the spreadsheet coming through is empty (they simply add a row).
  • adams-dl4j: Fixed the java.lang.UnsatisfiedLinkError: no jnind4j in java.library.path exception on Windows - correct inclusion of native artifacts.

Changes

  • Data container reader/writer classes now state the supported file extensions and the default file extension under "additional information" in the help.
  • Spreadsheet reader/writer classes now state the supported file extensions and the default file extension under "additional information" in the help.
  • Added Open in file browser to directory panel of the Preview Browser.
  • adams-spreadsheet: The SpreadSheetFileViewer now allows the user to paste a spreadsheet from the system clipboard (gets prompted how to read in the data).
  • Added Copy ID(s) to popup menu of container lists.
  • Centralized shortening of strings in adams.core.Shortening and adams.core.ShorteningType.
  • Spectrum Explorer now allows reloading of spectral files, not just from database.
  • The adams.core.management.Launcher class now allows the collapse of the classpath when launching the actual process, i.e., instead of listing every single jar explicitly, only the directory with /* appended is used. This dramatically reduces the length of the commandline string. Should also help with Windows' restriction of only allowing 260 characters in a classpath (Linux filesystems usually handle 4096 characters). The startup scripts, run and run.bat, got updated accordingly.
  • AppendName, DirName and PrependDir can output paths with forward slashes now as well, when processing strings.
  • adams-dl4j: upgraded deeplearning4j/nd4j to 0.4.0 and canova to 0.0.0.17.

Additions

  • Added global switch for anti-aliasing (enable/disable), available from the Visualization menu.
  • Added preview handler to the Preview browser for basic file information: FileInfoHandler
  • MapToVariables transformer turns the key-value pairs into variables and associated values, useful when loading in YAML config files.
  • adams-nlp:
    • added EditDistance transformer for computing the edit distance between strings (supplied base and ones received).
    • added Stemmer transformer for performing word stemming, single words or arrays of words.

Have a good weekend!

Updates 2016/07/15

Many changes and fixes regarding data processing and improvements in the GUI this time round.

Fixes

  • adams-twitter: update the link to the link search API used by the TwitterQuery source actor.
  • Adding large number of objects in the GenericArrayEditor is now much faster.
  • Preview browser: using a split pane now for local/archive files to better utilize the vertical space, also fixed the display of search box for archive files.
  • adams-compress: the listFiles methods of ZipUtils and TarUtils now skip directories in archives when specified.

Changes

  • Debugging is now also possible when using external actors, include external actors and internal actor handlers. An expanded version of the flow is then used in a new tab for debugging. However, this does not work in case of external actors having a variable attached to the actor file.
  • Pushes dimensions of dialogs into GUIHelper.props file, using defaults for normal, small, tiny and large ones (Default[(Small|Tiny|Large)]Dialog.(Width|Height)).
  • ConditionalTee/Trigger now use Expression as the default boolean condition, to align with ConditionalSubProcess (and you simply use Expression more often then Counting)
  • RaiseError's message can contain backquoted characters (\\t and \\n) and variables now.
  • Added ability to add a scope restriction to Breakpoint actors. This is useful when debugging multi-threaded flows (e.g., when using Rats), restricting the scope - and therefore the stepping through the flow - to a sub-flow. With the new automatic suggestions for breakpoints, the scope is automatically set for breakpoints within Rat actors.
  • Added support for custom actions for enclosing actors: adams.gui.flow.tree.enclose.AbstractEncloseActors. Migrated code for DisplayPanelManager and added action for TryCatch.
  • Removed the Open remote flow functionality from Flow editor, as it will get replaced in the future by the more powerful/flexible remote command functionality.
  • The XMLFileWriter sink and the DOMToString conversion now support pretty printing.
  • adams-rats:
    • added proper event-driven support in Rats.
    • AbstractBufferedRatInput uses blocking poll.
    • (most) webservices are now buffered rat inputs to immediately react to data being received, making it unnecessary to wrap them in an InputPolling "meta" rat input
  • adams-ml: it is now possible to restrict the axes of the ActualVsPredictedPlot, as well as placing the plot on the screen like with other graphical ones and use it within the DisplayPanelManager sink.

Additions

  • Added basic YAML support, using java.util.List and java.util.Map objects:
    • actors: YamlFileReader transformer, YamlFileWriter sink
    • conversions: YamlStringToList, YamlStringToMap, ArrayToYamlString, ListToYamlString, MapToYamlString
  • Added more JSON conversions:
    • ListToJson converts a java.util.List.
    • MapToJson converts a java.util.Map.
  • Added support for pretty-printing of JSON: JsonFileWriter sink, JsonToString conversion
  • Added JSON handler to preview browser that displays data as pretty-print text: JsonPrettyPrintHandler
  • Added menu item to the Flow editor's tree popup menu that allows you to compare two selected actors, displauing a diff view (you may have to remove the $HOME/.adams/FlowEditor.props file).
  • Made it possible to swap between SetVariable standalone and transformer (and vice versa).
  • Added notification actions for running/debugging the current flow.
  • New transformer for indentating strings (even multi-line strings): StringIndent
  • GetMapKeys transformer retrieves all the (string) keys from a java.util.Map object.
  • HasQueue flow condition checks whether the specified queue is present.
  • Added ability to list TODOs in flow (search actor name and annotation), using a user-supplied regular expression (ListTODOs processor).
  • Added the Class help menu to the Help menu of the main menu, to allow displaying the help for class.
  • Added dialogs to the main menu for pretty printing XML and JSON (only developer mode).
  • Added adams/flow/control/Flow.props properties file. The boolean AutoRegister property allows system-wide registering of flows with RunningFlowsRegistry (which can be queried by remote commands), default is false.
  • adams-meta: Added actor swap and option transfer for Callable/CopyCallable actor pairs.
  • adams-ml: Added simple spreadsheet reader/writer for ARFF format for ARFF support without adams-weka dependency.
  • adams-webservice: added interceptors for logging of SOAP envelopes, XMLLoggingGenerator and XMLLoggingGenerator.
  • adams-spectral-core: added reader/writer for spectra in JSON format.

Have a good weekend!

Updates 2016/06/25

This week was mostly focused on external actors, fixing a bug of disappearing tokens which has been around for a while.

Fixes

  • Fixed disappearing tokens when using external actors in conjunction with variables.
  • adams-spreadsheet: The TransposeSpreadSheet conversion now keeps the cell types and no longer re-parses the content, making it also more efficient.
  • adams-meta: The include external actors now can be edited in the Flow editor through the "External actor -> Edit" action.

Changes

  • Conversion AnyToString now accepts arrays as well (actually, Unknown which can be arrays).
  • adams-spreadsheet: The SpreadSheetCellFinderToPositions conversion has been deprecated. Instead, use the SpreadSheetCellFinder transformer in conjunction with the SpreadSheetCellLocationToPosition conversion.
  • The find option of the StringIndexOf transformer now accepts backquoted strings, enabling the user to search for tab or new line using t or n.
  • adams-imaging: metadata-extractor library upgraded to 2.9.1.
  • Made the Swap action in the Flow editor more flexible by introducing a class hierarchy for suggesting actors to swap with (adams.gui.flow.tree.actorswap.AbstractActorSwapSuggestion).
  • adams-rats: The rat input FileLister now has an option to skip files that are currently in use, e.g., large files that are still being written.
  • adams-imaging-boofcv: the connect rule is now an option for the BinaryContours and CannyBinary object location algorithms, as well as for the BinaryContours and CannyEdgeDetection image transformers.

Additions

  • The PropertiesToVariables allows you to turn properties, that match a regular expression, into variables (the property value becomes the variable value).
  • adams-spreadsheet:
    • With the SpreadSheetCellFinder transformer you can locate cells (and output the row/column pairs) that match certain criteria defined by the supplied CellFinder. Allows you to locate the row in a spreadsheet with a certain value of a specific column, like finding the row with a certain ID.
    • Conversions for CellLocation objects: SpreadSheetCellLocationToPosition, SpreadSheetCellLocationToCoordinates
    • New spreadsheet reader/writer for Java props files: PropertiesSpreadSheetReader, PropertiesSpreadSheetWriter
  • adams-meta: added option transfer schemes between external and include external actor pairs (and vice versa).

Updates 2016/06/17

Many little fixes and improvements again. The PDF generation got overhauled to make it easier generating PDFs from files.

Fixes

  • adams-weka: The Weka Instance Explorer now correctly shows the options dialog of the loader (modal issue).
  • adams-spreadsheet: The Spreadsheet Explorer now correctly shows the options dialog of the reader (modal issue).
  • The Help -> Memory visualization works again.
  • The default flow format now uses the user-specified encoding now correctly.
  • adams-spectral-core: filtering the spectral table in the Spectrum Explorer now uses the correct colors in the reduced view.

Changes

  • adams-pdf:
    • Upgraded PDFBox dependency to 2.0.1
    • PDFExtractImages transformer now outputs BufferedImageContainer objects instead of just BufferedImage objects.
    • PDF proclets (used by PDFCreate transformer) now can use a regular expression on the filename to further limit the matching (not just the file extension).
    • The following PDF proclets now offer absolute positioning: Image, PlainText, SpreadSheet.
  • Scripting (general/Groovy/Jython): pushed initialization of scripts from setUp() into preExecute() method to allow use of variables.
  • The Swap operation in the Flow editor now transfers variables as well.
  • adams-weka: upgraded multisearch package to 2016.6.6.
  • FileUtils.loadFromFile now removes byte order marks (BOMs) from UTF-16/UTF-32 encoded files.
  • The Flow editor now keeps track of flow reader/writer setups and stores the flow reader setups in "recent files" now as well. This now allows a simple save action for different file encodings instead of requiring a save as action.
  • The OptionUtils.shallowCopy method (when expand=false) now transfers the Variables instance and ensures that current variable values are set. Useful when creating instances inside actors (or further nested classes) that have variables attached.
  • adams-spectral-core: the Evaluator transformer now can obtain the actual evaluator from a callable actor as well, not just from a file with the serialized setup.
  • adams-twitter: upgraded twitter4j library to 4.0.4. I noted that they newer version of the library may not be as stable as the old one. However, there were other problems with the old version. You win some, you lose some.
  • adams-excel: upgraded Apache POI to 3.14.

Additions

  • It is now possible to print the content of a flow's notification area as well, not just copy to clipboard or save it.
  • adams-pdf:
    • Added PDFRenderPages transformer, which renders pages from a PDF as images.
    • added PDF proclets (used by PDFCreate transformer): Circle, Line, NewPage, Rectangle
    • Using PDFNewDocument, PDFAppendDocument and PDFCloseDocument it is now possible to use the PDF proclets in a more targeted fashion, as PDFAppendDocument allows to specify a specific file.
    • Added PDFInfo transformer for outputting information on PDF files.
  • adams-spectral-core:
    • added reader/writer for ASCII XY data format (wavenumber <SEP> amplitude).
    • added more instance evaluators: CrossValidatedNeighborHood, DistanceToClosest, DistanceToCenter, SavitzkyGolayRange
  • actors for handling java.util.Map objects: NewMap, GetMapValue, SetMapValue
  • boolean condition for dealing with java.util.Map objects: HasMapValue
  • adams-spreadsheet: conversions for converting to and fro java.util.Map objects: MapToSpreadSheet, SpreadSheetToMap

Note for developers

If you want to have the fastest possible build, use the following command-line (does not generate any PDF manuals, skips the unit tests and does not generate license files):

mvn clean install -DskipTests=true -Dlicense.skipAddThirdParty=true -Dlatex.dummyBuild=true

Updates 2016/06/03

A major change affecting everyone using snapshots: I incremented the snapshot version to 16.6.0-SNAPSHOT.

Fixes

  • The Locate actor menu item in the flow editor now parses output from the console properly again (also handles the new flow ID).
  • adams-spreadsheet:
    • SpreadSheetDifference with no key columns set computed the difference between the same cell, instead of between the cells from sheet 1 and 2.
    • SpreadSheetDbReader handles timezones now correctly.
    • The conversions SpreadSheetStringColumnToTime and SpreadSheetStringColumnToTimeMsec now have a sensible default format.

Changes

  • The logging output of actors now also includes the (at runtime) unique ID of the flow to make it easier to distinguish output when running multiple flows in parallel.
  • The following spreadsheet readers now implement WindowedSpreadSheetReader as well (offering options firstRow and numRows):
    • FixedTabularSpreadSheetReader
    • ODFSpreadSheetReader
    • GnumericSpreadSheetReader
    • ExcelSpreadSheetReader
    • ExcelStreamingSpreadSheetReader
  • adams-spectral:
    • batch filters are now handled in the spectrum explorer automatically when filtering spectra with a batch filter.
    • the SpectrumPaintlet now always paints the markers, not just when zoomed in.
  • The search text field in the GenericObjectEditor's class tree now gets the focus automatically when being displayed.
  • The EnterManyValues source now allows date/time expressions for the time/date/datetime types, as supported by BaseTime/BaseDate/BaseDateTime. (eg NOW -1 DAY for an initial date value).
  • SequenceToArray and SequenceToCollection can now generate bundled outputs with overlaps. E.g. 1,2,3,4 with overlap=1 will get packaged as (1,2),(2,3),(3,4).
  • adams-weka:
    • upgraded multisearch Weka package to 2016.5.31.
    • WekaInstanceViewer sink now has options for color-provider and paintlet, as well as an updater (to speed up the visualization).
  • The EnterManyValues source actor now offers the ability to enter strings that are constrained by a regular expression (eg for entering an email address). This is done via the RegExpConstrainedStringDefinition.

Additions

  • adams-net: added Send error report menu item (under Program), which allows the user to send an error report to a support email address. Email support needs to be configured and a support email address supplied to make this feature available.
  • adams-spectral-core:
    • SpectrumDisplay sink now allows specifying a paintlet and color provider.
    • New SimpleSpectrumPaintlet comes without any markers, in contrast to SpectrumPaintlet.
  • adams-weka: added the SimpleInstanceLinePaintlet paintlet to be used in conjunction with the WekaInstanceViewer sink.
  • adams-spreadsheet:
    • The Spreadsheet Explorer offers similar functionality for viewing spreadsheets as the Weka Instance Explorer, but without any dependencies on Weka.
    • The SpreadSheetRowViewer sink displays rows from a spreadsheet as line plots.

Have a nice long Queen's Birthday weekend - if it applies to you!

Update 2016/05/20

A major change in versioning of the artifacts has occurred: from now on, artifacts will have YY.MM.x as version, with YY being the year, MM the month and x the release in that month, starting with 0. This makes it a bit easier tracking down from what time period a snapshot is.

Fixes

  • adams-weka: The WekaExtractArray transformer now uses the correct maximum for columns/rows in case of spreadsheets (was accidentally swapped).
  • Default timezone no longer results in using GMT but the system-defined one. Java's timezone uses GMT if a timezone string cannot be parsed.

Changes

  • adams-core:
    • The Injector control actor can replace tokens passing through now, not just adding them before or after. Furthermore, it can also inject data from storage now.
    • The TrainableBatchFilter interface is used by transformers derived from AbstractFilter to train them if necessary with the first batch of data. Such filters can be reset with the monitor variable of the transformer.
  • adams-weka: upgraded multisearch-weka-package to 2016.5.15, which fixes a multithreading bug
  • Moved the following modules from base to addons:
    • gnuplot
    • latex
    • maps
    • moa
    • osm
    • r
    • random
    • twitter

Additions

  • adams-spreadsheet: It is now possible to extract columns or rows from a spreadsheet by using the SpreadSheetExtractArray transformer.
  • Added new addons module adams-dl4j, which makes the deeplearning4j library available within ADAMS.