Memory efficient hash maps in C++

I'm using boost::unordered_map for a requirement for a hash table mapping a std::string to a pointer to a class - ie - boost::unordered_map <string, ClassA*> . I see that after there are around 100000+ entries in the map, the memory grows quite a bit. During the course, I would also be continuously removing and re-adding entries to the map. Is there a good hash map implementation I can use which has excellent memory management(not use too much memory for book-keeping) but give me decent speed for lookups and decent speeds during insertions?

Thanks!