Package com.astrolabsoftware.FinkBrowser.Avro

Class HDFSAvroImporter

java.lang.Object

com.Lomikel.Januser.JanusClient

com.astrolabsoftware.FinkBrowser.Avro.AvroImporter

com.astrolabsoftware.FinkBrowser.Avro.HDFSAvroImporter

All Implemented Interfaces:

ModifyingGremlinClient

public class HDFSAvroImporter
extends AvroImporter

AvroImporter imports Avro files from HDFS into JanusGraph.

Author:

J.Hrivnac

Field Summary

Fields

Modifier and Type	Field	Description
private org.apache.hadoop.conf.Configuration	_conf
private org.apache.hadoop.fs.FileSystem	_fs
private static Logger	log	Logging .

Constructor Summary

Constructors

Constructor	Description
HDFSAvroImporter(java.lang.String properties, int reportLimit, int commitLimit, java.lang.String strategy, java.lang.String fitsDir, java.lang.String hbaseUrl, java.lang.String dataType)	Create with JanusGraph properties file.

Method Summary

Modifier and Type	Method	Description
static void	main(java.lang.String[] args)	Import Avro files or directory.
void	process(java.lang.String fn)	Process Avro alert file or directory with files (recursive).
void	processDir(java.lang.String dirFN, java.lang.String fileExt)	Process directory with Avro alert files (recursive).
void	processFile(org.apache.hadoop.fs.Path path)	Process Avro alert file .
protected void	writeFits(java.lang.String fn, byte[] data)	Write FITS file.

Methods inherited from class com.astrolabsoftware.FinkBrowser.Avro.AvroImporter

close, fitsDir, hbaseUrl, n, now, processAlert, processFile, processPCA, processRecord, register, skip

Methods inherited from class com.Lomikel.Januser.JanusClient

commit, g, open, open, reopen, timer, timerStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_conf

private org.apache.hadoop.conf.Configuration _conf

_fs

private org.apache.hadoop.fs.FileSystem _fs

log

private static Logger log

Logging .

Constructor Detail

HDFSAvroImporter

public HDFSAvroImporter(java.lang.String properties,
                        int reportLimit,
                        int commitLimit,
                        java.lang.String strategy,
                        java.lang.String fitsDir,
                        java.lang.String hbaseUrl,
                        java.lang.String dataType)

Create with JanusGraph properties file.

Parameters:

properties - The file with the complete Janusgraph properties.

reportLimit - The number of events to use for progress report (-1 means no report untill the end).

commitLimit - The number of events to commit in one step (-1 means commit only at the end).

strategy - The creation strategy. dro If null, then considering as alert.p,replace,getOrCreate.

fitsDir - The directory for FITS files. If nullor empty, FITS are included in the Graph. Ignored if HBase url set.

hbaseUrl - The url for HBase table with full data as ip:port:table:schema. May be null or empty.

dataType - The data type, alert|pca. If null, then considering as alert.

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Import Avro files or directory.

Parameters:

args - [0] The Janusgraph properties file.

args - [1] The Avro file or directory with Avro files.

args - [2] The directory for FITS files. If null or empty, FITS are included in the Graph. Ignored if HBase url set.

args - [3] The url for HBase table with full data as ip:port:table:schema. May be null or empty.

args - [4] The number of events to use for progress report (-1 means no report untill the end).

args - [5] The number of events to commit in one step (-1 means commit only at the end).

args - [6] The creation strategy. create,drop,replace,skip.

args - [7] The data type, alert|pca. If null, then considering as alert.

Throws:

LomikelException - If anything goes wrong.

java.io.IOException

processDir

public void processDir(java.lang.String dirFN,
                       java.lang.String fileExt)
                throws java.io.IOException

Description copied from class: AvroImporter

Process directory with Avro alert files (recursive).

Overrides:

processDir in class AvroImporter

Parameters:

dirFN - The dirname of directiory with data file.

fileExt - The file extention.

Throws:

java.io.IOException - If problem with file reading.

process

public void process(java.lang.String fn)
             throws java.io.IOException,
                    LomikelException

Description copied from class: AvroImporter

Process Avro alert file or directory with files (recursive).

Overrides:

process in class AvroImporter

Parameters:

fn - The filename of the data file or directory with files.

Throws:

java.io.IOException - If problem with file reading.

LomikelException - If anything wrong.

processFile

public void processFile(org.apache.hadoop.fs.Path path)
                 throws java.io.IOException

Process Avro alert file .

Parameters:

path - The data file.

Throws:

java.io.IOException - If problem with file reading.

writeFits

protected void writeFits(java.lang.String fn,
                         byte[] data)

Description copied from class: AvroImporter

Write FITS file.

Overrides:

writeFits in class AvroImporter

Parameters:

fn - The FITS file name.

data - The FITS file content.