Why does std::is_same always give me an error about conversion

In my project, there are some config files. I need to extract them either as integers or as strings. To do so, I write a function as below:

template<typename T>
void loadTxtConfig(const std::string& filename, std::set<T>& ids) {
    std::ifstream infile(filename);
    if (!infile) {
        return;
    }
    std::string line;
    if (std::is_integral<T>::value) {
        while (std::getline(infile, line)) {
            ids.emplace(static_cast<T>(std::stoll(line))); // noexcept
        }
    } else if (std::is_same<T, std::string>::value) {
        while (std::getline(infile, line)) {
            ids.emplace(line);
        }
    }
}

I try to cal it as below:

std::set<int> intCfg;
std::set<std::string> strCfg;
loadTxtConfig("./conf/int.txt", intCfg);
loadTxtConfig("./conf/str.txt", strCfg);

However, when I compile it, I always get an error about template:

/usr/include/c++/5/ext/new_allocator.h:120:4: error: cannot convert ‘std::__cxx11::basic_string<char>’ to ‘int’ in initialization
  { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }

What did I wrongly do?