Saving and loading std::chrono::time_point to file

I am trying to simply save and reload the current time in a file. For example:

std::ifstream ifs(solar_system_cache_file,
        std::ios::in | std::ios::binary);

if (!ifs.is_open()) {
    return false;
}

std::chrono::system_clock::time_point cache_valid_time;
ifs >> cache_valid_time;

if (std::chrono::system_clock::now() < cache_valid_time) {
    std::cout << "Cache is VALID." << std::endl;
    return true;
}
ifs.close();

and

std::ofstream ofs(solar_system_cache_file,
        std::ios::out | std::ios::binary);

if (!ofs.is_open())
    return;

ofs << std::chrono::system_clock::now() + 12h;
ofs.close();

This sort of thing isn't complicated, but I've been looking around for hours and I cannot find relevant info. There are some example on how to cast using duration_cast<long, std::milli>, but std::chrono is extremely convoluted and hard to navigate (and digest).

In a nutshell, I believe I need to cast the current time to long (or some similar large type) and save that. When deserializing the time, I just have to cast it back to a time_point. It sounds easy enough, but I am unable to do so.

Finally, simply piping the time in fstream gives the usual invalid operands to binary expression error.

Any help is appreciated, or links to good tutorials/guides. Thank you