Delete all not found i.e. delete all key/values in map not found in set

I tried but failed to get the following working with std::algorithms: I have a a std::map<key_t,value_t> cache and a std::set<key_t> selected_items and I want to delete all key/value pairs in the map cache, which keys are not found in the set new. Like this I wrote without the algorithms:

//This could really be wrote better with std::algorithms but time...
//Delete old
for (auto pair = cache.begin(); pair != cache.end();){
    if (selected_items.find(pair->first) == selected_items.end()){
        pair = cache.erase(pair);
        if (pair == cache.end()){
            break;
        }
        else{
            continue;
        }
    }
    ++pair;
}

So I figured I have to use std::set_difference with a compare function and either std::remove or std::map::erase. But I can't connect the pieces, failed at:

1. What's the correct compare function
2. Do I have to generate a temporary set with the keys that should be delete or can I use the output iterator directly for the remove/erase?

Maybe someone could give me the whole example, please.