lundi 11 janvier 2021

Read big files in C++ but also small files as well in C++?

I want to make a C++ program to read huge files (like 50Gb each) while you have only 4 or 8Gb of RAM. I want this algorithm to be faster and work with small files as well.

This is the code I have until now:

#include<iostream>
#include<fstream>
#include<string>

using namespace std;
//Making a buffer to store the chuncks of the file read:
// Buffer size 1 Megabyte (or any number you like)
size_t buffer_size = 1<<20;
char *buffer = new char[buffer_size];



int main(){
string filename="stats.txt";

//compute file size
size_t iFileSize = 0;
std::ifstream ifstr(filename.c_str(), std::ios::binary); // create the file stream    - this is scoped for destruction

if(!ifstr.good()){
    cout<<"File is not valid!"<<endl;
    exit(EXIT_FAILURE);
}
//get the file size
iFileSize = ifstr.tellg();
ifstr.seekg( 0, std::ios::end ); // open file at the end to get the size
iFileSize = (int) ifstr.tellg() - iFileSize;
cout<<"File size is: "<<iFileSize<<endl;
//close the file and reopen it for reading:
ifstr.close();
cout<<"Buffer size before check is:"<<buffer_size<<endl;
if(buffer_size>iFileSize){
        buffer_size=iFileSize;
}
cout<<"Buffer size after check is:"<<buffer_size<<endl;


ifstream myFile;

myFile.open(filename);

if(myFile.fail()){
        cerr<<"Error opening file!"<<endl;
        exit(EXIT_FAILURE);

}

if(!myFile.good()){
        cout<<"File is not valid!"<<endl;
        exit(EXIT_FAILURE);

}

if(!myFile.is_open()){
        cout<<"File is NOT opened anymore!"<<endl;
        return 1;
}

while(myFile.is_open()&&myFile){
    // Try to read next chunk of data
    myFile.read(buffer, buffer_size);
    // Get the number of bytes actually read
    size_t count = myFile.gcount();
     // If nothing has been read, break
    if (!count){
        break;
    }
    // Do whatever you need with first count bytes in the buffer:
    string line;
    while(getline(myFile, line)){
             if(!line.empty()){
                cout <<"Line: '" << line << "'" <<endl;
             }


    }

}
delete[] buffer;
buffer = NULL;
myFile.close();


return 0;

}

My files could have blank lines between the text line, also even the first lines could be blank lines. So, I tested the program on a small file size (128kb in size) named to see how it works. But it doesn't work. It doesn't display any line on the screen even the file is so small.

What is wrong? Also, if I change the buffer size to a very small number, it reads just first one or two lines but why it doesn't loop to the end of the file to read and display all of the lines from that small file? Any help, please?

Thank you in advance!

This is the test file: (It starts with a few blank lines also.)

Population UK: 97876876723


Population France: 898989













This is the test end of the file: Yay!

This is the result: And no line from file is displayed. enter image description here

Aucun commentaire:

Enregistrer un commentaire