Binary filestream to byte std::vector skips spaces

When I use this code

std::string filename = "tmp.bin";
std::ifstream fileStream;
std::vector<unsigned char> fileBuffer;

fileStream = std::ifstream(filename.c_str(), std::ios::binary | std::ios::ate);     
fileBuffer.reserve(fileStream.tellg());
fileStream.seekg(0, std::ios::beg);
fileBuffer.insert(fileBuffer.begin(), std::istream_iterator<BYTE>(fileStream), std::istream_iterator<BYTE>());

all original spaces in my binary file are skipped -> fileBuffer contains no spaces, but need all tokens for Base64 encoding.

What is wrong here?