org.canova.api.records.reader.impl.jackson

Class JacksonRecordReader

java.lang.Object

org.canova.api.records.reader.impl.jackson.JacksonRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

public class JacksonRecordReader
extends Object
implements RecordReader

RecordReader using Jackson.
Design for this record reader:
- Support for JSON, XML and YAML: one record per file only, via Jackson ObjectMapper:

• - JSON: new ObjectMapper(new JsonFactory())
• - YAML: new ObjectMapper(new YAMLFactory()) (requires jackson-dataformat-yaml dependency)
• - XML: new ObjectMapper(new XmlFactory()) (requires jackson-dataformat-xml dependency)

- User provides a list of fields to load, using FieldSelection. This complicates configuration for simple structures (user has to specify every field to load), however this allows us to parse files where:

• - The fields in the json/xml/yaml is not in a consistent order (for example, JSON makes no guarantees about order). The order of output fields is provided via the FieldSelection object.
• - Fields may be missing in some files (output will include an (optionally) specified writable for the missing value, defined again in FieldSelection)
• - The fields in the json/yaml/xml files may have arbitrary nested structure: For example, a: b: c: d: someValue

- Optional support for appending a label based on the path of the file, using PathLabelGenerator
- Support for shuffling of records (with an optional RNG seed)

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields inherited from interface org.canova.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
JacksonRecordReader(FieldSelection selection, com.fasterxml.jackson.databind.ObjectMapper mapper)
JacksonRecordReader(FieldSelection selection, com.fasterxml.jackson.databind.ObjectMapper mapper, boolean shuffle)
JacksonRecordReader(FieldSelection selection, com.fasterxml.jackson.databind.ObjectMapper mapper, boolean shuffle, long rngSeed, PathLabelGenerator labelGenerator)
JacksonRecordReader(FieldSelection selection, com.fasterxml.jackson.databind.ObjectMapper mapper, boolean shuffle, long rngSeed, PathLabelGenerator labelGenerator, int labelPosition)

Method Summary

Modifier and Type	Method and Description
void	close()
Configuration	getConf() Return the configuration used by this object.
List<String>	getLabels() List of label strings
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
void	initialize(InputSplit split) Called once at initialization.
Collection<Writable>	next() Get the next record
Collection<Writable>	record(URI uri, DataInputStream dataInputStream) Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	reset() Reset record reader iterator
void	setConf(Configuration conf) Set the configuration to be used by this object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JacksonRecordReader

public JacksonRecordReader(FieldSelection selection,
                           com.fasterxml.jackson.databind.ObjectMapper mapper)

JacksonRecordReader

public JacksonRecordReader(FieldSelection selection,
                           com.fasterxml.jackson.databind.ObjectMapper mapper,
                           boolean shuffle)

JacksonRecordReader

public JacksonRecordReader(FieldSelection selection,
                           com.fasterxml.jackson.databind.ObjectMapper mapper,
                           boolean shuffle,
                           long rngSeed,
                           PathLabelGenerator labelGenerator)

JacksonRecordReader

public JacksonRecordReader(FieldSelection selection,
                           com.fasterxml.jackson.databind.ObjectMapper mapper,
                           boolean shuffle,
                           long rngSeed,
                           PathLabelGenerator labelGenerator,
                           int labelPosition)

Method Detail

initialize

public void initialize(InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Parameters:

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

next

public Collection<Writable> next()

Description copied from interface: RecordReader

Get the next record

Specified by:

next in interface RecordReader

Returns:

hasNext

public boolean hasNext()

Description copied from interface: RecordReader

Whether there are anymore records

Specified by:

hasNext in interface RecordReader

Returns:

getLabels

public List<String> getLabels()

Description copied from interface: RecordReader

List of label strings

Specified by:

getLabels in interface RecordReader

Returns:

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

Specified by:

reset in interface RecordReader

record

public Collection<Writable> record(URI uri,
                                   DataInputStream dataInputStream)
                            throws IOException

Description copied from interface: RecordReader

Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

record in interface RecordReader

Throws:

IOException - if error occurs during reading from the input stream

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

setConf

public void setConf(Configuration conf)

Description copied from interface: Configurable

Set the configuration to be used by this object.

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Description copied from interface: Configurable

Return the configuration used by this object.

Specified by:

getConf in interface Configurable