Package com.astrolabsoftware.FinkBrowser.Parquet

Class ParquetImporter

    • Field Summary

      Fields 
      Modifier and Type Field Description
      private int _commitLimit  
      private org.apache.hadoop.conf.Configuration _conf  
      private boolean _create  
      private boolean _drop  
      private org.apache.hadoop.fs.FileSystem _fs  
      private GremlinRecipies _gr  
      private int _n  
      private boolean _replace  
      private int _reportLimit  
      private boolean _reuse  
      private static java.util.Map<java.lang.String,​java.lang.String> IDS  
      private static org.apache.logging.log4j.Logger log
      Logging .
      private static java.util.Map<java.lang.String,​java.lang.String> RELATIONS  
      private static java.util.Map<java.lang.String,​java.lang.String> VERTEXES  

    • Constructor Summary

      Constructors 
      Constructor Description
      ParquetImporter​(java.lang.String properties, int reportLimit, int commitLimit, java.lang.String strategy)
      Create with JanusGraph properties file.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      private java.time.LocalDateTime int96toTimestamp​(byte[] bytes)
      Transform timestamp from int96 to LocalDateTime.
      static void main​(java.lang.String[] args)
      Import Parquet files or directory.
      void process​(java.lang.String fn)
      Process Parquet alert file or directory with files (recursive).
      void processDir​(java.lang.String dirFN, java.lang.String fileExt)
      Process directory with Parquet alert files.
      private java.util.List<Vertex> processGroup​(org.apache.parquet.example.data.Group g, java.lang.String lbl)
      Process Group and create contained Vertexes with the specified label.
      private void registerVertices​(java.util.Map<java.lang.String,​java.util.List<Vertex>> map, java.lang.String key, java.util.List<Vertex> vertices)
      Register Vertexes in a Map according their label.
      private java.lang.String reLabel​(java.lang.String label)
      Change label.
      private Vertex vertex​(java.util.Map<java.lang.String,​java.lang.String> props, java.lang.String label, java.lang.String property)
      Create or drop a Vertex according to chosen strategy.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • VERTEXES

        private static java.util.Map<java.lang.String,​java.lang.String> VERTEXES

      • RELATIONS

        private static java.util.Map<java.lang.String,​java.lang.String> RELATIONS

      • IDS

        private static java.util.Map<java.lang.String,​java.lang.String> IDS

      • _conf

        private org.apache.hadoop.conf.Configuration _conf

      • _fs

        private org.apache.hadoop.fs.FileSystem _fs

      • _n

        private int _n

      • _reuse

        private boolean _reuse

      • _drop

        private boolean _drop

      • log

        private static org.apache.logging.log4j.Logger log
        Logging .

    • Constructor Detail

      • ParquetImporter

        public ParquetImporter​(java.lang.String properties,
                       int reportLimit,
                       int commitLimit,
                       java.lang.String strategy)
        Create with JanusGraph properties file.
        Parameters:
        properties - The file with the complete Janusgraph properties.
        reportLimit - The number of events to use for progress report (-1 means no report until the end).
        commitLimit - The number of events to commit in one step (-1 means commit only at the end).
        strategy - The creation strategy. drop,replace,getOrCreate.

    • Method Detail

      • main

        public static void main​(java.lang.String[] args)
                 throws java.io.IOException
        Import Parquet files or directory.
        Parameters:
        args - [0] The Janusgraph properties file.
        args - [1] The Parquet file or directory with Parquet files.
        args - [2] The number of events to use for progress report (-1 means no report untill the end).
        args - [3] The number of events to commit in one step (-1 means commit only at the end).
        args - [4] The creation strategy. drop,replace,getOrCreate.
        Throws:
        LomikelException - If anything goes wrong.
        java.io.IOException

      • processDir

        public void processDir​(java.lang.String dirFN,
                       java.lang.String fileExt)
                throws java.io.IOException,
                       java.io.FileNotFoundException
        Process directory with Parquet alert files.
        Parameters:
        dirFN - The dirname of directiory with data file.
        fileExt - The file extention.
        Throws:
        java.io.IOException - If problem with file reading.
        java.io.FileNotFoundException - If problem with file founding.

      • process

        public void process​(java.lang.String fn)
             throws java.io.IOException,
                    LomikelException
        Process Parquet alert file or directory with files (recursive).
        Parameters:
        fn - The filename of the data file or directory with files.
        Throws:
        java.io.IOException - If problem with file reading.
        LomikelException - If anything wrong.

      • processGroup

        private java.util.List<Vertex> processGroup​(org.apache.parquet.example.data.Group g,
                                            java.lang.String lbl)
        Process Group and create contained Vertexes with the specified label. Runs recursively.
        Parameters:
        g - The Group to process.
        lbl - The label for created Vertexes.
        Returns:
        The created Vertexes.

      • vertex

        private Vertex vertex​(java.util.Map<java.lang.String,​java.lang.String> props,
                      java.lang.String label,
                      java.lang.String property)
        Create or drop a Vertex according to chosen strategy.
        Parameters:
        props - The Map of values.
        label - The Vertex label.
        property - The name of Vertex property. If null strategy is ignored and Vertex is created.
        Returns:
        The created Vertex or null.

      • registerVertices

        private void registerVertices​(java.util.Map<java.lang.String,​java.util.List<Vertex>> map,
                              java.lang.String key,
                              java.util.List<Vertex> vertices)
        Register Vertexes in a Map according their label.
        Parameters:
        map - The Map of all registered Vertexes, organised according their label.
        key - The label of new Vertexes/
        vertices - The List of new Vertexes to be added.

      • reLabel

        private java.lang.String reLabel​(java.lang.String label)
        Change label.
        Parameters:
        label - The original label.
        Returns:
        The changed label.

      • int96toTimestamp

        private java.time.LocalDateTime int96toTimestamp​(byte[] bytes)
        Transform timestamp from int96 to LocalDateTime.
        Parameters:
        bytes - The timestamp of int96 bytes.
        Returns:
        The timestamp converted to LocalDateTime.