Differentiate conversion operator template for reference and non-reference types

I have a struct InferValueType which shall be implicitly convertible to any other type including reference types. I accomplish this using the following code.

struct InferValueType {
    // ...

    template <typename Type>
    operator Type() const && {
        // ...
    }

    // We need this special version to allow conversions to reference types
    template <typename Type>
    operator Type&() const & {
        // ...
    }
};

InferValueType is never instantiated by the the user, instead I provide it as the return value of a function.

const InferValueType read(State* state, int index);

The combination of InferValueType and read allows the user to write lazy code like this:

int i = read(state, 1);
MyType t1 = read(state, 2);
MyType& t2 = read(state, 2);
const MyType& t3 = read(state, 2);

I have tested this against GCC 4.8, 4.9, 5, 6 and Clang 3.7 and 3.8. Only GCC 4.8 complains about an ambiguity.

conversion from ‘const InferValueType’ to ‘int’ is ambiguous
conversion from ‘const InferValueType’ to ‘MyType’ is ambiguous

I assume this is because int and MyType can be constructed using const int& and const MyType& respectively.

Is this a peculiarity of GCC before 4.9 or is it actually an ambiguity under C++11?