Intersection of two `unordered_set`s in C++ "not working" when in a class method but works with `set`?

What I have here is two "maps", xs and ys, which I use to store "points" in a grid. I also have an array std::pair<int,int> arr, such that xs[x] returns all the indices in arr with x-coordinate x, and ys[y] returns all indices with y-coordinate y.

Let us have the following:

std::unordered_map<int, std::unordered_set<size_t>> xs {
    {3, {2, 0}},
    {11, {1}}
};
std::unordered_map<int, std::unordered_set<size_t>> ys {
    {2, {2 ,1}},
    {10, {0}}
};

std::unordered_set<size_t> intersection;
std::set_intersection(xs[3].begin(), xs[3].end(),
    ys[2].begin(), ys[2].end(),
    std::inserter(intersection, intersection.begin()));
std::cout << intersection.size() << std::endl; // should return 1

intersection.clear();
std::set_intersection(xs[11].begin(), xs[11].end(),
    ys[2].begin(), ys[2].end(),
    std::inserter(intersection, intersection.begin()));
std::cout << intersection.size() << std::endl; // should return 1

They do return 1 and 1. No surprises there.

Let us now isolate the intersection counter in a function:

size_t count_intersection(const std::pair<int,int>& pt,
        const std::unordered_map<int, std::unordered_set<size_t>>& xs,
        const std::unordered_map<int, std::unordered_set<size_t>>& ys) {
    
    const int x = pt.first, y = pt.second;

    // Omitted some error checking

    std::unordered_set<size_t> intersection;
    std::set_intersection(xs.at(x).begin(), xs.at(x).end(),
        ys.at(y).begin(), ys.at(y).end(),
        std::inserter(intersection, intersection.begin()));

    return intersection.size();
}

std::cout << count_intersection({3, 2}, xs, ys) << std::endl;
std::cout << count_intersection({11, 2}, xs, ys) << std::endl;

So far, so good.

Now, let's isolate all of this in a class:

class Counter {

    std::vector<std::pair<int,int>> _pts;
    std::unordered_map<int, std::unordered_set<size_t>> _xs;
    std::unordered_map<int, std::unordered_set<size_t>> _ys;

public:
    void add(const std::pair<int,int>& pt) {
        const int x = pt.first, y = pt.second;
        const size_t ind = _pts.size();

        _pts.push_back(pt);
        _xs[x].insert(ind);
        _ys[y].insert(ind);
    }

    size_t count(const std::pair<int,int>& pt) {
        std::unordered_set<size_t> intersection;
        const int x = pt.first, y = pt.second;

        std::set_intersection(_xs[x].begin(), _xs[x].end(),
            _ys[y].begin(), _ys[y].end(),
            std::inserter(intersection, intersection.begin()));
        return intersection.size();
    }
};

int main() {
    Counter c;
    c.add({3,10}); // ind == 0
    c.add({11,2}); // ind == 1
    c.add({3,2});  // ind == 2

    // By this point,
    // _xs == {3: {0, 2}, 11: {1}}
    // _ys == {10: {0}, 2: {1,2}}
    std::cout << c.count({3,2}) << std::endl;  // Should return 1
    std::cout << c.count({11,2}) << std::endl; // Should return 1
}

Instead, what I got is

1
0

However, when I replaced std::unordered_set with std::set, the result becomes expected.

What's up with std::unordered_set?

By the way, my command in compiling is

g++ -Wall -Wextra -Werror -O3 -std=c++17 -pedantic -fsanitize=address -o main.out main.cpp && ./main.out

and my g++ --version is g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0.