lundi 25 février 2019

How to consume and process AVRO data in GPU efficiently?

Requirement :

  1. I have an application producing AVRO data (around 100 GB per hour).
  2. I want to create another application that would read the generated AVRO data (hourly) and transform the data in GPU

Example class generated by AVRO (not accurate but just to present my case) :

struct XXX {
    std:string s1;
    std:string s2;
    int i1;
    float f1; 
}

struct YYY { 
    typedef XXX x1;
    typedef XXX x2;
    std:string s1;
    int i1;
}

Problem :

  1. After de-serializing an AVRO record, it contains std::string in nested structure
  2. GPU does not support std::string

Question :

Is there a way, to send the deserialized AVRO record to GPU for further processing as it is?

Please bear with me, if the question seems very naive, as I am new to both c++ and GPU programming.

Aucun commentaire:

Enregistrer un commentaire