Package com.astrolabsoftware.FinkBrowser.Avro

Class HDFSAvroImporter

java.lang.Object
com.Lomikel.Januser.JanusClient
com.astrolabsoftware.FinkBrowser.Avro.AvroImporter
com.astrolabsoftware.FinkBrowser.Avro.HDFSAvroImporter
All Implemented Interfaces:
ModifyingGremlinClient
public class HDFSAvroImporter extends AvroImporter
AvroImporter imports Avro files from HDFS into JanusGraph.
Author:
J.Hrivnac

  • Field Details

    • _conf

      private org.apache.hadoop.conf.Configuration _conf

    • _fs

      private org.apache.hadoop.fs.FileSystem _fs

    • log

      private static Logger log
      Logging .

  • Constructor Details

    • HDFSAvroImporter

      public HDFSAvroImporter(String properties, int reportLimit, int commitLimit, String strategy, String fitsDir, String hbaseUrl, String dataType)
      Create with JanusGraph properties file.
      Parameters:
      properties - The file with the complete Janusgraph properties.
      reportLimit - The number of events to use for progress report (-1 means no report untill the end).
      commitLimit - The number of events to commit in one step (-1 means commit only at the end).
      strategy - The creation strategy. dro If null, then considering as alert.p,replace,getOrCreate.
      fitsDir - The directory for FITS files. If nullor empty, FITS are included in the Graph. Ignored if HBase url set.
      hbaseUrl - The url for HBase table with full data as ip:port:table:schema. May be null or empty.
      dataType - The data type, alert|pca. If null, then considering as alert.

  • Method Details

    • main

      public static void main(String[] args) throws IOException
      Import Avro files or directory.
      Parameters:
      args - [0] The Janusgraph properties file.
      args - [1] The Avro file or directory with Avro files.
      args - [2] The directory for FITS files. If null or empty, FITS are included in the Graph. Ignored if HBase url set.
      args - [3] The url for HBase table with full data as ip:port:table:schema. May be null or empty.
      args - [4] The number of events to use for progress report (-1 means no report untill the end).
      args - [5] The number of events to commit in one step (-1 means commit only at the end).
      args - [6] The creation strategy. create,drop,replace,skip.
      args - [7] The data type, alert|pca. If null, then considering as alert.
      Throws:
      LomikelException - If anything goes wrong.
      IOException

    • processDir

      public void processDir(String dirFN, String fileExt) throws IOException
      Description copied from class: AvroImporter
      Process directory with Avro alert files (recursive).
      Overrides:
      processDir in class AvroImporter
      Parameters:
      dirFN - The dirname of directiory with data file.
      fileExt - The file extention.
      Throws:
      IOException - If problem with file reading.

    • process

      public void process(String fn) throws IOException, LomikelException
      Description copied from class: AvroImporter
      Process Avro alert file or directory with files (recursive).
      Overrides:
      process in class AvroImporter
      Parameters:
      fn - The filename of the data file or directory with files.
      Throws:
      IOException - If problem with file reading.
      LomikelException - If anything wrong.

    • processFile

      public void processFile(org.apache.hadoop.fs.Path path) throws IOException
      Process Avro alert file .
      Parameters:
      path - The data file.
      Throws:
      IOException - If problem with file reading.

    • writeFits

      protected void writeFits(String fn, byte[] data)
      Description copied from class: AvroImporter
      Write FITS file.
      Overrides:
      writeFits in class AvroImporter
      Parameters:
      fn - The FITS file name.
      data - The FITS file content.