dimanche 9 octobre 2022

Multithreaded concurrent file reading/writing, managing container of processes

Wholly new to multithreading.

I am writing a program which takes as input a vector of objects and an integer for the number of threads to dedicate. The nature of the objects isn't important, only that each has several members that are file paths to large text files. Here's a simplified version:

// Not very important. Reads file, writes new version omitting
// some lines
void proc_file(OBJ obj) {
  std::string inFileStr(obj.get_path().c_str());
  std::string outFileStr(std::string(obj.get_path().replace_extension("new.txt").c_str()));

  std::ifstream inFile(inFileStr);
  std::ofstream outFile(outFileStr);

  std::string currLine;
  while (getline(inFile, currLine)) {
    if (currLine.size() == 1 ||
        currLine.compare(currLine.length()-5, 5, "thing") != 0) {
      outFile << currLine << '\n';
    }
    else {
      for (int i = 0; i < 3; i++) {
        getline(inFile, currLine);
      }
    }
  }
  inFile.close();
  outFile.close();
}

// Processes n file concurrently, working way through 
// all OBJ in objs
void multi_file_proc(std::vector<OBJ> objs, int n) {
  std::vector<std::thread> procVec;
  for (int i = 0; i < objs.size(); i++) {
    /*
    Ensure that n files are always being processed.
    Upon completion of one, initiate another, until
    all OBJ in objs have had their text files changed.
    */
  }
}

I want to loop through each OBJ and write altered versions of their text files in concurrence, the limitation on simultaneous file read/writes being the thread value (n). Ultimately, all the objects' text files must be changed, but in such a way that there are always n files being processed, to maximize efficiency in concurrence.

Note the vector of threads, procVec. I originally approached this by managing a vector of threads, with a file being processed for each thread in procVec. From my reading, it seems a vector for managing these tasks is logical. But how do I always ensure there are n files open until all have been processed, without exiting with an open thread?

Aucun commentaire:

Enregistrer un commentaire