mercredi 2 novembre 2016

Embedding big data file into executable binary

I am working on a C++11 application that is supposed to ship as a single executable binary file. Optionally, users can provide their own CSV data files to be used by the application. To simplify things, assume each element is in format key,value\n. I have created a structure such as:

typedef struct Data {
    std::string key;
    std::string value;

    Data(std::string key, std::string value) : key(key), value(value) {}
} Data;

By default, the application should use data defined in a single header file. I've made a simple Python script to parse default CSV file and put it into header file like:

#ifndef MYPROJECT_DEFAULTDATA
#define MYPROJECT_DEFAULTDATA

#include "../database/DefaultData.h"

namespace defaults {
    std::vector<Data> default_data = {
        Data("SomeKeyA","SomeValueA"),
        Data("SomeKeyB","SomeValueB"),
        Data("SomeKeyC","SomeValueC"),

        /* and on, and on, and on... */

        Data("SomeKeyASFHOIEGEWG","SomeValueASFHOIEGEWG")
    }
}

#endif //MYPROJECT_DEFAULTDATA

The only problem is, that file is big. I'm talking 116'087 (12M) lines big, and it will probably be replaced with even bigger file in the future. When I include it, my IDE is trying to parse it and update indices. It slows everything down to the point where I can hardly write anything.

I'm looking for a way to either:

  1. prevent my IDE (CLion) from parsing it or
  2. make a switch in cmake that would use this file only with release executables or
  3. somehow inject data directly into executable

Aucun commentaire:

Enregistrer un commentaire