invalid user-defined conversion to rvalue reference

What is reasoning behind the fact that most versions gcc, except 4.9.0-4.9.4 and 9.1.0, consider this C++11 code ill-formed unless -pedantic and -fpermissive options are used at same time? clang compiles it.

struct A {
    int a;
    operator int() &&  { return a; }
    operator int&() &  { return a; }
};

void b(int &&) {}

int main()
{
    b(A{});
}

Output is similar to:

prog.cc: In function 'int main()':
prog.cc:11:10: error: invalid user-defined conversion from 'A' to 'int&&' [-fpermissive]
     b(A{});
          ^
prog.cc:4:5: note: candidate is: A::operator int&() & <near match>
     operator int&() &  { return a;