Maintaining a object level lock in a map

I have a class A and I am creating multiple objects eg a1, a2, a3 for it. I am also maintaining a map to cache these objects. It looks like this:

Example class:
class A
{
public:
    A() {}
};

std::map<std::string, std::weak_ptr<A>> cache_;
std::mutex m;

std::shared_ptr<A> get_A(std::string a)
{
   std::lock_guard<std::mutex> lock(m);
   // If map has an entry for a, then return the object from map else,
   // create a shared_ptr of object of A with a custom deleter
   // and add it to the map
   std::shared_ptr<A> obj = cache_[a].lock();
   if (!obj) {
       auto temp_obj = std::make_unique<A>();
       auto deleter = [a, cache_, m](A *ptr) {
                 std::lock_guard<std::mutex> lock(m);
                 auto it = cache_.find(a);
                 if (it != cache_.end()) {
                      cache_.erase(it);
                 }
         delete ptr;
       }
       obj = std::unique_ptr<A, decltype(deleter)>(temp_obj.release(), deleter);
       cache_[a] = obj;
   }
   return obj;
}

This code is executed in a multi threaded environment and construction/destruction of an object of A can take time. Other threads can potentially block on the lock if one thread is taking time to create the object. I am trying to have a object level lock.

My solution looks like this:

std::map<std::string, std::mutex> lock_map;

std::shared_ptr<A> get_A(std::string a)
{
   std::unique_lock<std::mutex> lock(m);
   // This will get a lock for a
   std::lock_guard<std::mutex> obj_lock(lock_map[a]);
   lock.unlock();

   // If map has an entry for a, then return the object from map else,
   // create a shared_ptr of object of A with a custom deleter
   // and add it to the map
   std::shared_ptr<A> obj = cache_[a].lock();
   if (!obj) {
       auto temp_obj = std::make_unique<A>();
       auto deleter = [a, cache_, m](A *ptr) {
                 std::lock_guard<std::mutex> lock(m);
       std::unique_lock<std::mutex> lock(m);
       // This will get a lock for a
       std::lock_guard<std::mutex> obj_lock(lock_map[a]);
       lock.unlock();
                 auto it = cache_.find(a);
                 if (it != cache_.end()) {
                      cache_.erase(it);
                 }
         delete ptr;
       }
       obj = std::unique_ptr<A, decltype(deleter)>(temp_obj.release(), deleter);
       cache_[a] = obj;
   }
   return obj;
}

This works well but my only concern is when to remove the entry from lock_map. Is there any better approach for this problem?