Updates 2016/07/15

Many changes and fixes regarding data processing and improvements in the GUI this time round.

Fixes

  • adams-twitter: update the link to the link search API used by the TwitterQuery source actor.
  • Adding large number of objects in the GenericArrayEditor is now much faster.
  • Preview browser: using a split pane now for local/archive files to better utilize the vertical space, also fixed the display of search box for archive files.
  • adams-compress: the listFiles methods of ZipUtils and TarUtils now skip directories in archives when specified.

Changes

  • Debugging is now also possible when using external actors, include external actors and internal actor handlers. An expanded version of the flow is then used in a new tab for debugging. However, this does not work in case of external actors having a variable attached to the actor file.
  • Pushes dimensions of dialogs into GUIHelper.props file, using defaults for normal, small, tiny and large ones (Default[(Small|Tiny|Large)]Dialog.(Width|Height)).
  • ConditionalTee/Trigger now use Expression as the default boolean condition, to align with ConditionalSubProcess (and you simply use Expression more often then Counting)
  • RaiseError's message can contain backquoted characters (\\t and \\n) and variables now.
  • Added ability to add a scope restriction to Breakpoint actors. This is useful when debugging multi-threaded flows (e.g., when using Rats), restricting the scope - and therefore the stepping through the flow - to a sub-flow. With the new automatic suggestions for breakpoints, the scope is automatically set for breakpoints within Rat actors.
  • Added support for custom actions for enclosing actors: adams.gui.flow.tree.enclose.AbstractEncloseActors. Migrated code for DisplayPanelManager and added action for TryCatch.
  • Removed the Open remote flow functionality from Flow editor, as it will get replaced in the future by the more powerful/flexible remote command functionality.
  • The XMLFileWriter sink and the DOMToString conversion now support pretty printing.
  • adams-rats:
    • added proper event-driven support in Rats.
    • AbstractBufferedRatInput uses blocking poll.
    • (most) webservices are now buffered rat inputs to immediately react to data being received, making it unnecessary to wrap them in an InputPolling "meta" rat input
  • adams-ml: it is now possible to restrict the axes of the ActualVsPredictedPlot, as well as placing the plot on the screen like with other graphical ones and use it within the DisplayPanelManager sink.

Additions

  • Added basic YAML support, using java.util.List and java.util.Map objects:
    • actors: YamlFileReader transformer, YamlFileWriter sink
    • conversions: YamlStringToList, YamlStringToMap, ArrayToYamlString, ListToYamlString, MapToYamlString
  • Added more JSON conversions:
    • ListToJson converts a java.util.List.
    • MapToJson converts a java.util.Map.
  • Added support for pretty-printing of JSON: JsonFileWriter sink, JsonToString conversion
  • Added JSON handler to preview browser that displays data as pretty-print text: JsonPrettyPrintHandler
  • Added menu item to the Flow editor's tree popup menu that allows you to compare two selected actors, displauing a diff view (you may have to remove the $HOME/.adams/FlowEditor.props file).
  • Made it possible to swap between SetVariable standalone and transformer (and vice versa).
  • Added notification actions for running/debugging the current flow.
  • New transformer for indentating strings (even multi-line strings): StringIndent
  • GetMapKeys transformer retrieves all the (string) keys from a java.util.Map object.
  • HasQueue flow condition checks whether the specified queue is present.
  • Added ability to list TODOs in flow (search actor name and annotation), using a user-supplied regular expression (ListTODOs processor).
  • Added the Class help menu to the Help menu of the main menu, to allow displaying the help for class.
  • Added dialogs to the main menu for pretty printing XML and JSON (only developer mode).
  • Added adams/flow/control/Flow.props properties file. The boolean AutoRegister property allows system-wide registering of flows with RunningFlowsRegistry (which can be queried by remote commands), default is false.
  • adams-meta: Added actor swap and option transfer for Callable/CopyCallable actor pairs.
  • adams-ml: Added simple spreadsheet reader/writer for ARFF format for ARFF support without adams-weka dependency.
  • adams-webservice: added interceptors for logging of SOAP envelopes, XMLLoggingGenerator and XMLLoggingGenerator.
  • adams-spectral-core: added reader/writer for spectra in JSON format.

Have a good weekend!

Updates 2016/06/25

This week was mostly focused on external actors, fixing a bug of disappearing tokens which has been around for a while.

Fixes

  • Fixed disappearing tokens when using external actors in conjunction with variables.
  • adams-spreadsheet: The TransposeSpreadSheet conversion now keeps the cell types and no longer re-parses the content, making it also more efficient.
  • adams-meta: The include external actors now can be edited in the Flow editor through the "External actor -> Edit" action.

Changes

  • Conversion AnyToString now accepts arrays as well (actually, Unknown which can be arrays).
  • adams-spreadsheet: The SpreadSheetCellFinderToPositions conversion has been deprecated. Instead, use the SpreadSheetCellFinder transformer in conjunction with the SpreadSheetCellLocationToPosition conversion.
  • The find option of the StringIndexOf transformer now accepts backquoted strings, enabling the user to search for tab or new line using t or n.
  • adams-imaging: metadata-extractor library upgraded to 2.9.1.
  • Made the Swap action in the Flow editor more flexible by introducing a class hierarchy for suggesting actors to swap with (adams.gui.flow.tree.actorswap.AbstractActorSwapSuggestion).
  • adams-rats: The rat input FileLister now has an option to skip files that are currently in use, e.g., large files that are still being written.
  • adams-imaging-boofcv: the connect rule is now an option for the BinaryContours and CannyBinary object location algorithms, as well as for the BinaryContours and CannyEdgeDetection image transformers.

Additions

  • The PropertiesToVariables allows you to turn properties, that match a regular expression, into variables (the property value becomes the variable value).
  • adams-spreadsheet:
    • With the SpreadSheetCellFinder transformer you can locate cells (and output the row/column pairs) that match certain criteria defined by the supplied CellFinder. Allows you to locate the row in a spreadsheet with a certain value of a specific column, like finding the row with a certain ID.
    • Conversions for CellLocation objects: SpreadSheetCellLocationToPosition, SpreadSheetCellLocationToCoordinates
    • New spreadsheet reader/writer for Java props files: PropertiesSpreadSheetReader, PropertiesSpreadSheetWriter
  • adams-meta: added option transfer schemes between external and include external actor pairs (and vice versa).

Updates 2016/06/17

Many little fixes and improvements again. The PDF generation got overhauled to make it easier generating PDFs from files.

Fixes

  • adams-weka: The Weka Instance Explorer now correctly shows the options dialog of the loader (modal issue).
  • adams-spreadsheet: The Spreadsheet Explorer now correctly shows the options dialog of the reader (modal issue).
  • The Help -> Memory visualization works again.
  • The default flow format now uses the user-specified encoding now correctly.
  • adams-spectral-core: filtering the spectral table in the Spectrum Explorer now uses the correct colors in the reduced view.

Changes

  • adams-pdf:
    • Upgraded PDFBox dependency to 2.0.1
    • PDFExtractImages transformer now outputs BufferedImageContainer objects instead of just BufferedImage objects.
    • PDF proclets (used by PDFCreate transformer) now can use a regular expression on the filename to further limit the matching (not just the file extension).
    • The following PDF proclets now offer absolute positioning: Image, PlainText, SpreadSheet.
  • Scripting (general/Groovy/Jython): pushed initialization of scripts from setUp() into preExecute() method to allow use of variables.
  • The Swap operation in the Flow editor now transfers variables as well.
  • adams-weka: upgraded multisearch package to 2016.6.6.
  • FileUtils.loadFromFile now removes byte order marks (BOMs) from UTF-16/UTF-32 encoded files.
  • The Flow editor now keeps track of flow reader/writer setups and stores the flow reader setups in "recent files" now as well. This now allows a simple save action for different file encodings instead of requiring a save as action.
  • The OptionUtils.shallowCopy method (when expand=false) now transfers the Variables instance and ensures that current variable values are set. Useful when creating instances inside actors (or further nested classes) that have variables attached.
  • adams-spectral-core: the Evaluator transformer now can obtain the actual evaluator from a callable actor as well, not just from a file with the serialized setup.
  • adams-twitter: upgraded twitter4j library to 4.0.4. I noted that they newer version of the library may not be as stable as the old one. However, there were other problems with the old version. You win some, you lose some.
  • adams-excel: upgraded Apache POI to 3.14.

Additions

  • It is now possible to print the content of a flow's notification area as well, not just copy to clipboard or save it.
  • adams-pdf:
    • Added PDFRenderPages transformer, which renders pages from a PDF as images.
    • added PDF proclets (used by PDFCreate transformer): Circle, Line, NewPage, Rectangle
    • Using PDFNewDocument, PDFAppendDocument and PDFCloseDocument it is now possible to use the PDF proclets in a more targeted fashion, as PDFAppendDocument allows to specify a specific file.
    • Added PDFInfo transformer for outputting information on PDF files.
  • adams-spectral-core:
    • added reader/writer for ASCII XY data format (wavenumber <SEP> amplitude).
    • added more instance evaluators: CrossValidatedNeighborHood, DistanceToClosest, DistanceToCenter, SavitzkyGolayRange
  • actors for handling java.util.Map objects: NewMap, GetMapValue, SetMapValue
  • boolean condition for dealing with java.util.Map objects: HasMapValue
  • adams-spreadsheet: conversions for converting to and fro java.util.Map objects: MapToSpreadSheet, SpreadSheetToMap

Note for developers

If you want to have the fastest possible build, use the following command-line (does not generate any PDF manuals, skips the unit tests and does not generate license files):

mvn clean install -DskipTests=true -Dlicense.skipAddThirdParty=true -Dlatex.dummyBuild=true

Updates 2016/06/03

A major change affecting everyone using snapshots: I incremented the snapshot version to 16.6.0-SNAPSHOT.

Fixes

  • The Locate actor menu item in the flow editor now parses output from the console properly again (also handles the new flow ID).
  • adams-spreadsheet:
    • SpreadSheetDifference with no key columns set computed the difference between the same cell, instead of between the cells from sheet 1 and 2.
    • SpreadSheetDbReader handles timezones now correctly.
    • The conversions SpreadSheetStringColumnToTime and SpreadSheetStringColumnToTimeMsec now have a sensible default format.

Changes

  • The logging output of actors now also includes the (at runtime) unique ID of the flow to make it easier to distinguish output when running multiple flows in parallel.
  • The following spreadsheet readers now implement WindowedSpreadSheetReader as well (offering options firstRow and numRows):
    • FixedTabularSpreadSheetReader
    • ODFSpreadSheetReader
    • GnumericSpreadSheetReader
    • ExcelSpreadSheetReader
    • ExcelStreamingSpreadSheetReader
  • adams-spectral:
    • batch filters are now handled in the spectrum explorer automatically when filtering spectra with a batch filter.
    • the SpectrumPaintlet now always paints the markers, not just when zoomed in.
  • The search text field in the GenericObjectEditor's class tree now gets the focus automatically when being displayed.
  • The EnterManyValues source now allows date/time expressions for the time/date/datetime types, as supported by BaseTime/BaseDate/BaseDateTime. (eg NOW -1 DAY for an initial date value).
  • SequenceToArray and SequenceToCollection can now generate bundled outputs with overlaps. E.g. 1,2,3,4 with overlap=1 will get packaged as (1,2),(2,3),(3,4).
  • adams-weka:
    • upgraded multisearch Weka package to 2016.5.31.
    • WekaInstanceViewer sink now has options for color-provider and paintlet, as well as an updater (to speed up the visualization).
  • The EnterManyValues source actor now offers the ability to enter strings that are constrained by a regular expression (eg for entering an email address). This is done via the RegExpConstrainedStringDefinition.

Additions

  • adams-net: added Send error report menu item (under Program), which allows the user to send an error report to a support email address. Email support needs to be configured and a support email address supplied to make this feature available.
  • adams-spectral-core:
    • SpectrumDisplay sink now allows specifying a paintlet and color provider.
    • New SimpleSpectrumPaintlet comes without any markers, in contrast to SpectrumPaintlet.
  • adams-weka: added the SimpleInstanceLinePaintlet paintlet to be used in conjunction with the WekaInstanceViewer sink.
  • adams-spreadsheet:
    • The Spreadsheet Explorer offers similar functionality for viewing spreadsheets as the Weka Instance Explorer, but without any dependencies on Weka.
    • The SpreadSheetRowViewer sink displays rows from a spreadsheet as line plots.

Have a nice long Queen's Birthday weekend - if it applies to you!

Update 2016/05/20

A major change in versioning of the artifacts has occurred: from now on, artifacts will have YY.MM.x as version, with YY being the year, MM the month and x the release in that month, starting with 0. This makes it a bit easier tracking down from what time period a snapshot is.

Fixes

  • adams-weka: The WekaExtractArray transformer now uses the correct maximum for columns/rows in case of spreadsheets (was accidentally swapped).
  • Default timezone no longer results in using GMT but the system-defined one. Java's timezone uses GMT if a timezone string cannot be parsed.

Changes

  • adams-core:
    • The Injector control actor can replace tokens passing through now, not just adding them before or after. Furthermore, it can also inject data from storage now.
    • The TrainableBatchFilter interface is used by transformers derived from AbstractFilter to train them if necessary with the first batch of data. Such filters can be reset with the monitor variable of the transformer.
  • adams-weka: upgraded multisearch-weka-package to 2016.5.15, which fixes a multithreading bug
  • Moved the following modules from base to addons:
    • gnuplot
    • latex
    • maps
    • moa
    • osm
    • r
    • random
    • twitter

Additions

  • adams-spreadsheet: It is now possible to extract columns or rows from a spreadsheet by using the SpreadSheetExtractArray transformer.
  • Added new addons module adams-dl4j, which makes the deeplearning4j library available within ADAMS.