Confused on what implementation to use for an AssetManager that loads files into objects

The main job of this entity is provide a Asset& Load(string path) polymorphic interface. At present I only have TextureAsset : Asset used by SpriteRenderer, but there will be more, for example, AudioAsset : Asset. The entity also maintains a cache of already loaded assets. It is currently a singleton, as I feel this class justifies the usage of the usually-avoided pattern. But I'm worried that I might be being overzealous/overconfident, and that this is exactly what the elders mean when they say "overuse of the pattern", etc.

Should it be a static class instead, without a destructor, as the singleton's destructor won't be called till the end of the application anyway?

Should it not even be a class and be a procedural function in a namespace?

Should it be a regular class, a reference to whose object is owned and provided by the Engine object? But it's not really related to the Engine; you could theoretically have two Engine objects use the same AssetManager (there is no practical reason to have two Engines but my code structure doesn't prohibit it). It's also a pain to keep passing such an "orphaned" object around.

class AssetManager final {
public:
    static AssetManager& Instance() {
        static AssetManager instance;
        return instance;
    }

    template<typename T>
    std::shared_ptr<T> LoadAsset(const std::string& path) {
        static_assert(std::is_base_of<Asset, T>::value, "T needs to derive from Asset");
        // This assert works, but the error is generated on this line,
        // and VS2017 doesn't give any more info.
        // Is there a way for me to point to the illegal call instead?

        auto search = loaded.find(path);
        if (search != loaded.end()) {
            std::shared_ptr<Asset> asset = search->second;
            return std::dynamic_pointer_cast<T>(asset);
        }

        std::shared_ptr<T> t_ptr = std::make_shared<T>(path);
        loaded.insert(std::pair<std::string, std::shared_ptr<T> >(path, t_ptr));
        return t_ptr;
    }

    void Clear() {
        loaded.clear();
    }

private:
    AssetManager() = default;
    ~AssetManager() {
        loaded.clear();
    }
    AssetManager(const AssetManager&) = delete;
    AssetManager(AssetManager&&) = delete;
    AssetManager& operator=(const AssetManager&) = delete;
    AssetManager& operator=(const AssetManager&&) = delete;
    std::map<std::string, std::shared_ptr<Asset> > loaded;
};