dimanche 6 décembre 2020

Sorting huge text file - how to find out the chuck size fro a cross-platform code

I need to sort huge (10, 20, 30 GB...) text files. So obviously I'm gonna split the file into smaller files and sort them separately and then merge them. But how am I supposed to choose the size of that chunks I want to split the main file into, so it would be the most efficient? If I want to write a cross platform code, I can't get the RAM size. Is it a way to check can I read at once, say, 1 Gb (or better start with larger number?), if not check 800 Mb and so on? Or there is some other way to handle this?

Aucun commentaire:

Enregistrer un commentaire