Package com.astrolabsoftware.FinkBrowser.Avro

Class AvroImporter

java.lang.Object

com.Lomikel.Januser.JanusClient

com.astrolabsoftware.FinkBrowser.Avro.AvroImporter

All Implemented Interfaces:

ModifyingGremlinClient

Direct Known Subclasses:

HDFSAvroImporter

public class AvroImporter
extends JanusClient

AvroImporter imports Avro files into JanusGraph.

Author:

J.Hrivnac

Field Summary

Fields

Modifier and Type	Field	Description
private int	_commitLimit
private boolean	_create
private java.lang.String	_dataType
private java.lang.String	_date
private boolean	_drop
private java.lang.String	_fitsDir
private GremlinRecipies	_gr
private java.lang.String	_hbaseUrl
private java.lang.String	_jd
private int	_n
private int	_nPrvCandidates
private boolean	_replace
private int	_reportLimit
private boolean	_reuse
private boolean	_skip
private boolean	_top
private java.lang.String	_topFn
private static java.util.List<java.lang.String>	COLUMNS_ALERT
private static java.util.List<java.lang.String>	COLUMNS_CANDIDATE
private static Logger	log	Logging .
private static java.lang.String[]	PCAS
private static java.lang.String	VERSION

Constructor Summary

Constructors

Constructor	Description
AvroImporter(java.lang.String properties, int reportLimit, int commitLimit, java.lang.String strategy, java.lang.String fitsDir, java.lang.String hbaseUrl, java.lang.String dataType)	Create with JanusGraph properties file.

Method Summary

Modifier and Type	Method	Description
void	close()	Close graph.
protected java.lang.String	fitsDir()	The directory for FITS files.
private java.util.List<java.lang.String>	getSimpleFields(GenericRecord record, java.util.List<java.lang.String> keeps, java.lang.String[] avoids)	Get Schema.Fields corresponding to simple types and having non-null values.
private java.util.Map<java.lang.String,java.lang.String>	getSimpleValues(GenericRecord record, java.util.List<java.lang.String> fields)	Register part of GenericRecord in HBase.
protected java.lang.String	hbaseUrl()	The data HBase table url.
static void	main(java.lang.String[] args)	Import Avro files or directory.
protected int	n()	Give number of created alerts/pcas.
protected void	now()	Set new Date.
void	process(java.lang.String fn)	Process Avro alert file or directory with files (recursive).
Vertex	processAlert(GenericRecord record)	Process Avro alert.
private void	processCutout(GenericRecord record, Vertex mother, java.lang.String objectId, java.lang.String jd)	Process Avro cutout.
void	processDir(java.lang.String dirFN, java.lang.String fileExt)	Process directory with Avro alert files (recursive).
void	processFile(java.io.File file)	Process Avro alert file .
private Vertex	processGenericRecord(GenericRecord record, java.lang.String name, java.lang.String idName, boolean tryDirection, Vertex mother, java.lang.String edgeName, java.util.List<java.lang.String> fields, java.lang.String objectId)	Process Avro GenericRecord.
Vertex	processPCA(GenericRecord record)	Process Avro PCA.
void	processRecord(GenericRecord record)	Process GenericRecord of the requested type.
void	register(java.lang.String fn)	Register Import Vertex.
protected boolean	skip()	Tell, whether import shpuld be skipped.
private Vertex	vertex(GenericRecord record, java.lang.String label, java.lang.String property, java.lang.String strategy)	Create or drop a Vertex according to chosen strategy.
protected void	writeFits(java.lang.String fn, byte[] data)	Write FITS file.

Methods inherited from class com.Lomikel.Januser.JanusClient

commit, g, open, open, reopen, timer, timerStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COLUMNS_ALERT

private static java.util.List<java.lang.String> COLUMNS_ALERT

COLUMNS_CANDIDATE

private static java.util.List<java.lang.String> COLUMNS_CANDIDATE

_gr

private GremlinRecipies _gr

_jd

private java.lang.String _jd

_reportLimit

private int _reportLimit

_commitLimit

private int _commitLimit

_fitsDir

private java.lang.String _fitsDir

_hbaseUrl

private java.lang.String _hbaseUrl

_dataType

private java.lang.String _dataType

_create

private boolean _create

_reuse

private boolean _reuse

_replace

private boolean _replace

_drop

private boolean _drop

_skip

private boolean _skip

_n

private int _n

_nPrvCandidates

private int _nPrvCandidates

_date

private java.lang.String _date

_top

private boolean _top

_topFn

private java.lang.String _topFn

PCAS

private static java.lang.String[] PCAS

VERSION

private static java.lang.String VERSION

log

private static Logger log

Logging .

Constructor Detail

AvroImporter

public AvroImporter(java.lang.String properties,
                    int reportLimit,
                    int commitLimit,
                    java.lang.String strategy,
                    java.lang.String fitsDir,
                    java.lang.String hbaseUrl,
                    java.lang.String dataType)

Create with JanusGraph properties file.

Parameters:

properties - The file with the complete Janusgraph properties.

reportLimit - The number of events to use for progress report (-1 means no report untill the end).

commitLimit - The number of events to commit in one step (-1 means commit only at the end).

strategy - The creation strategy. drop,replace,getOrCreate.

fitsDir - The directory for FITS files. If null or empty, FITS are included in the Graph. Ignored if HBase url set.

hbaseUrl - The url for HBase table with full data as ip:port:table:schema. May be null or empty.

dataType - The data type, alert|pca. If null, then considering as alert.

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Import Avro files or directory.

Parameters:

args - [0] The Janusgraph properties file.

args - [1] The Avro file or directory with Avro files.

args - [2] The directory for FITS files. If null or empty, FITS are included in the Graph. Ignored if HBase url set.

args - [3] The url for HBase table with full data as ip:port:table:schema. May be null or empty.

args - [4] The number of events to use for progress report (-1 means no report untill the end).

args - [5] The number of events to commit in one step (-1 means commit only at the end).

args - [6] The creation strategy. create,drop,replace,skip.

args - [7] The data type, alert|pca. If null, then considering as alert.

Throws:

LomikelException - If anything goes wrong.

java.io.IOException

processDir

public void processDir(java.lang.String dirFN,
                       java.lang.String fileExt)
                throws java.io.IOException

Process directory with Avro alert files (recursive).

Parameters:

dirFN - The dirname of directiory with data file.

fileExt - The file extention.

Throws:

java.io.IOException - If problem with file reading.

process

public void process(java.lang.String fn)
             throws java.io.IOException,
                    LomikelException

Process Avro alert file or directory with files (recursive).

Parameters:

fn - The filename of the data file or directory with files.

Throws:

java.io.IOException - If problem with file reading.

LomikelException - If anything wrong.

register

public void register(java.lang.String fn)

Register Import Vertex.

Parameters:

fn - The filename of the data file or directory with files.

processFile

public void processFile(java.io.File file)
                 throws java.io.IOException

Process Avro alert file .

Parameters:

file - The data file.

Throws:

java.io.IOException - If problem with file reading.

processRecord

public void processRecord(GenericRecord record)

Process GenericRecord of the requested type.

Parameters:

record - The GenericRecord to be rocessed.

processAlert

public Vertex processAlert(GenericRecord record)

Process Avro alert.

Parameters:

record - The full alert GenericRecord.

Returns:

The created Vertex.

processPCA

public Vertex processPCA(GenericRecord record)

Process Avro PCA. Only create PCAs for already existing sources. Do not verify if the PCA already exists (so multiple PCAs fvor one source may be created.

Parameters:

record - The full PCA GenericRecord.

Returns:

The created Vertex.

processGenericRecord

private Vertex processGenericRecord(GenericRecord record,
                                    java.lang.String name,
                                    java.lang.String idName,
                                    boolean tryDirection,
                                    Vertex mother,
                                    java.lang.String edgeName,
                                    java.util.List<java.lang.String> fields,
                                    java.lang.String objectId)

Process Avro GenericRecord.

Parameters:

record - The GenericRecord to process.

name - The name of new Vertex.

idName - The name of the unique identifying field.

tryDirection - Whether try created Direction property from ra,dec fields.

mother - The mother Vertex.

edgerName - The name of the edge to the mother Vertex.

fields - The list of fields to fill. All fields are filled if null

objectId - The objectId of the containing source.

Returns:

The created Vertex.

processCutout

private void processCutout(GenericRecord record,
                           Vertex mother,
                           java.lang.String objectId,
                           java.lang.String jd)

Process Avro cutout.

Parameters:

record - The GenericRecord to process.

mother - The Vertex to attach to.

jd - The jd of the corresponding candidate.

getSimpleValues

private java.util.Map<java.lang.String,java.lang.String> getSimpleValues(GenericRecord record,
                                                                               java.util.List<java.lang.String> fields)

Register part of GenericRecord in HBase.

Parameters:

record - The GenericRecord to be registered in HBase.

fields - The fields to be mapped.

getSimpleFields

private java.util.List<java.lang.String> getSimpleFields(GenericRecord record,
                                                         java.util.List<java.lang.String> keeps,
                                                         java.lang.String[] avoids)

Get Schema.Fields corresponding to simple types and having non-null values.

Parameters:

record - The GenericRecord to use.

keeps - The GenericRecord to report. Report all if null.

avoids - The array of fields names not to report. Cannot cancel keeps argument.

Returns:

The list of coressponding fields.

vertex

private Vertex vertex(GenericRecord record,
                      java.lang.String label,
                      java.lang.String property,
                      java.lang.String strategy)

Create or drop a Vertex according to chosen strategy.

Parameters:

record - The full GenericRecord.

label - The Vertex label.

property - The name of Vertex property. If null strategy is ignored and Vertex is created.

strategy - The creation strategy: drop, replace, reuse, skip, create. If anything else, the global strategy b is used.

Returns:

The created Vertex or null.

writeFits

protected void writeFits(java.lang.String fn,
                         byte[] data)

Write FITS file.

Parameters:

fn - The FITS file name.

data - The FITS file content.

fitsDir

protected java.lang.String fitsDir()

The directory for FITS files.

Returns:

The FITS file directory.

hbaseUrl

protected java.lang.String hbaseUrl()

The data HBase table url.

Returns:

The data HBase table url.

n

protected int n()

Give number of created alerts/pcas.

Returns:

The number of created alerts/pcas.

skip

protected boolean skip()

Tell, whether import shpuld be skipped.

Returns:

Whether import shpuld be skipped.

close

public void close()

Description copied from interface: ModifyingGremlinClient

Close graph.

Specified by:

close in interface ModifyingGremlinClient

Overrides:

close in class JanusClient

now

protected void now()

Set new Date.