Package com.astrolabsoftware.FinkBrowser.Parquet

Class ParquetImporter

java.lang.Object

com.Lomikel.Januser.JanusClient

com.astrolabsoftware.FinkBrowser.Parquet.ParquetImporter

All Implemented Interfaces:

ModifyingGremlinClient

public class ParquetImporter
extends JanusClient

ParquetImporter imports Parquet files into JanusGraph.

Author:

J.Hrivnac

Field Summary

Fields

Modifier and Type	Field	Description
private int	_commitLimit
private org.apache.hadoop.conf.Configuration	_conf
private boolean	_create
private boolean	_drop
private org.apache.hadoop.fs.FileSystem	_fs
private GremlinRecipies	_gr
private int	_n
private boolean	_replace
private int	_reportLimit
private boolean	_reuse
private static Map<String,String>	IDS
private static org.apache.logging.log4j.Logger	log	Logging .
private static Map<String,String>	RELATIONS
private static Map<String,String>	VERTEXES

Constructor Summary

Constructors

Constructor	Description
ParquetImporter(String properties, int reportLimit, int commitLimit, String strategy)	Create with JanusGraph properties file.

Method Summary

Modifier and Type	Method	Description
private LocalDateTime	int96toTimestamp(byte[] bytes)	Transform timestamp from int96 to LocalDateTime.
static void	main(String[] args)	Import Parquet files or directory.
void	process(String fn)	Process Parquet alert file or directory with files (recursive).
void	processDir(String dirFN, String fileExt)	Process directory with Parquet alert files.
private List<Vertex>	processGroup(org.apache.parquet.example.data.Group g, String lbl)	Process Group and create contained Vertexes with the specified label.
private void	registerVertices(Map<String,List<Vertex>> map, String key, List<Vertex> vertices)	Register Vertexes in a Map according their label.
private String	reLabel(String label)	Change label.
private Vertex	vertex(Map<String,String> props, String label, String property)	Create or drop a Vertex according to chosen strategy.

Methods inherited from class com.Lomikel.Januser.JanusClient

close, commit, g, open, open, reopen, timer, timerStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

VERTEXES

private static Map<String,String> VERTEXES

RELATIONS

private static Map<String,String> RELATIONS

IDS

private static Map<String,String> IDS

_conf

private org.apache.hadoop.conf.Configuration _conf

_fs

private org.apache.hadoop.fs.FileSystem _fs

_gr

private GremlinRecipies _gr

_n

private int _n

_reportLimit

private int _reportLimit

_commitLimit

private int _commitLimit

_create

private boolean _create

_reuse

private boolean _reuse

_replace

private boolean _replace

_drop

private boolean _drop

log

private static org.apache.logging.log4j.Logger log

Logging .

Constructor Details

ParquetImporter

public ParquetImporter(String properties,
 int reportLimit,
 int commitLimit,
 String strategy)

Create with JanusGraph properties file.

Parameters:

properties - The file with the complete Janusgraph properties.

reportLimit - The number of events to use for progress report (-1 means no report until the end).

commitLimit - The number of events to commit in one step (-1 means commit only at the end).

strategy - The creation strategy. drop,replace,getOrCreate.

Method Details

main

public static void main(String[] args)
                 throws IOException

Import Parquet files or directory.

Parameters:

args - [0] The Janusgraph properties file.

args - [1] The Parquet file or directory with Parquet files.

args - [2] The number of events to use for progress report (-1 means no report untill the end).

args - [3] The number of events to commit in one step (-1 means commit only at the end).

args - [4] The creation strategy. drop,replace,getOrCreate.

Throws:

LomikelException - If anything goes wrong.

IOException

processDir

public void processDir(String dirFN,
 String fileExt)
                throws IOException,
FileNotFoundException

Process directory with Parquet alert files.

Parameters:

dirFN - The dirname of directiory with data file.

fileExt - The file extention.

Throws:

IOException - If problem with file reading.

FileNotFoundException - If problem with file founding.

process

public void process(String fn)
             throws IOException,
LomikelException

Process Parquet alert file or directory with files (recursive).

Parameters:

fn - The filename of the data file or directory with files.

Throws:

IOException - If problem with file reading.

LomikelException - If anything wrong.

processGroup

private List<Vertex> processGroup(org.apache.parquet.example.data.Group g,
 String lbl)

Process Group and create contained Vertexes with the specified label. Runs recursively.

Parameters:

g - The Group to process.

lbl - The label for created Vertexes.

Returns:

The created Vertexes.

vertex

private Vertex vertex(Map<String,String> props,
 String label,
 String property)

Create or drop a Vertex according to chosen strategy.

Parameters:

props - The Map of values.

label - The Vertex label.

property - The name of Vertex property. If null strategy is ignored and Vertex is created.

Returns:

The created Vertex or null.

registerVertices

private void registerVertices(Map<String,List<Vertex>> map,
 String key,
 List<Vertex> vertices)

Register Vertexes in a Map according their label.

Parameters:

map - The Map of all registered Vertexes, organised according their label.

key - The label of new Vertexes/

vertices - The List of new Vertexes to be added.

reLabel

private String reLabel(String label)

Change label.

Parameters:

label - The original label.

Returns:

The changed label.

int96toTimestamp

private LocalDateTime int96toTimestamp(byte[] bytes)

Transform timestamp from int96 to LocalDateTime.

Parameters:

bytes - The timestamp of int96 bytes.

Returns:

The timestamp converted to LocalDateTime.