Class ParquetImporter
- java.lang.Object
-
- com.Lomikel.Januser.JanusClient
-
- com.astrolabsoftware.FinkBrowser.Parquet.ParquetImporter
-
- All Implemented Interfaces:
ModifyingGremlinClient
public class ParquetImporter extends JanusClient
ParquetImporter
imports Parquet files into JanusGraph.- Author:
- J.Hrivnac
-
-
Field Summary
Fields Modifier and Type Field Description private int
_commitLimit
private org.apache.hadoop.conf.Configuration
_conf
private boolean
_create
private boolean
_drop
private org.apache.hadoop.fs.FileSystem
_fs
private GremlinRecipies
_gr
private int
_n
private boolean
_replace
private int
_reportLimit
private boolean
_reuse
private static java.util.Map<java.lang.String,java.lang.String>
IDS
private static org.apache.logging.log4j.Logger
log
Logging .private static java.util.Map<java.lang.String,java.lang.String>
RELATIONS
private static java.util.Map<java.lang.String,java.lang.String>
VERTEXES
-
Constructor Summary
Constructors Constructor Description ParquetImporter(java.lang.String properties, int reportLimit, int commitLimit, java.lang.String strategy)
Create with JanusGraph properties file.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private java.time.LocalDateTime
int96toTimestamp(byte[] bytes)
Transform timestamp from int96 toLocalDateTime
.static void
main(java.lang.String[] args)
Import Parquet files or directory.void
process(java.lang.String fn)
Process Parquet alert file or directory with files (recursive).void
processDir(java.lang.String dirFN, java.lang.String fileExt)
Process directory with Parquet alert files.private java.util.List<Vertex>
processGroup(org.apache.parquet.example.data.Group g, java.lang.String lbl)
ProcessGroup
and create containedVertex
es with the specified label.private void
registerVertices(java.util.Map<java.lang.String,java.util.List<Vertex>> map, java.lang.String key, java.util.List<Vertex> vertices)
RegisterVertex
es in aMap
according their label.private java.lang.String
reLabel(java.lang.String label)
Change label.private Vertex
vertex(java.util.Map<java.lang.String,java.lang.String> props, java.lang.String label, java.lang.String property)
Create or drop aVertex
according to chosen strategy.-
Methods inherited from class com.Lomikel.Januser.JanusClient
close, commit, g, open, open, reopen, timer, timerStart
-
-
-
-
Field Detail
-
VERTEXES
private static java.util.Map<java.lang.String,java.lang.String> VERTEXES
-
RELATIONS
private static java.util.Map<java.lang.String,java.lang.String> RELATIONS
-
IDS
private static java.util.Map<java.lang.String,java.lang.String> IDS
-
_conf
private org.apache.hadoop.conf.Configuration _conf
-
_fs
private org.apache.hadoop.fs.FileSystem _fs
-
_gr
private GremlinRecipies _gr
-
_n
private int _n
-
_reportLimit
private int _reportLimit
-
_commitLimit
private int _commitLimit
-
_create
private boolean _create
-
_reuse
private boolean _reuse
-
_replace
private boolean _replace
-
_drop
private boolean _drop
-
log
private static org.apache.logging.log4j.Logger log
Logging .
-
-
Constructor Detail
-
ParquetImporter
public ParquetImporter(java.lang.String properties, int reportLimit, int commitLimit, java.lang.String strategy)
Create with JanusGraph properties file.- Parameters:
properties
- The file with the complete Janusgraph properties.reportLimit
- The number of events to use for progress report (-1 means no report until the end).commitLimit
- The number of events to commit in one step (-1 means commit only at the end).strategy
- The creation strategy. drop,replace,getOrCreate.
-
-
Method Detail
-
main
public static void main(java.lang.String[] args) throws java.io.IOException
Import Parquet files or directory.- Parameters:
args
- [0] The Janusgraph properties file.args
- [1] The Parquet file or directory with Parquet files.args
- [2] The number of events to use for progress report (-1 means no report untill the end).args
- [3] The number of events to commit in one step (-1 means commit only at the end).args
- [4] The creation strategy. drop,replace,getOrCreate.- Throws:
LomikelException
- If anything goes wrong.java.io.IOException
-
processDir
public void processDir(java.lang.String dirFN, java.lang.String fileExt) throws java.io.IOException, java.io.FileNotFoundException
Process directory with Parquet alert files.- Parameters:
dirFN
- The dirname of directiory with data file.fileExt
- The file extention.- Throws:
java.io.IOException
- If problem with file reading.java.io.FileNotFoundException
- If problem with file founding.
-
process
public void process(java.lang.String fn) throws java.io.IOException, LomikelException
Process Parquet alert file or directory with files (recursive).- Parameters:
fn
- The filename of the data file or directory with files.- Throws:
java.io.IOException
- If problem with file reading.LomikelException
- If anything wrong.
-
processGroup
private java.util.List<Vertex> processGroup(org.apache.parquet.example.data.Group g, java.lang.String lbl)
ProcessGroup
and create containedVertex
es with the specified label. Runs recursively.
-
vertex
private Vertex vertex(java.util.Map<java.lang.String,java.lang.String> props, java.lang.String label, java.lang.String property)
Create or drop aVertex
according to chosen strategy.
-
registerVertices
private void registerVertices(java.util.Map<java.lang.String,java.util.List<Vertex>> map, java.lang.String key, java.util.List<Vertex> vertices)
RegisterVertex
es in aMap
according their label.
-
reLabel
private java.lang.String reLabel(java.lang.String label)
Change label.- Parameters:
label
- The original label.- Returns:
- The changed label.
-
int96toTimestamp
private java.time.LocalDateTime int96toTimestamp(byte[] bytes)
Transform timestamp from int96 toLocalDateTime
.- Parameters:
bytes
- The timestamp of int96 bytes.- Returns:
- The timestamp converted to
LocalDateTime
.
-
-