const-reference qualified member function

The stock example of a reference-qualified member function seems to be something like this:

#include <stdio.h>
#include <stdexcept>
#include <string>

/**
 *  Easy access to literals
 */
using namespace std::literals;

/**
 *  File wrapper
 */
class File
{
private:
    /**
     *  The wrapped file
     */
    FILE *_file;
public:
    /**
     *  Constructor
     *
     *  @param  name    The file to open
     *  @throws std::runtime_error
     */
    File(const char *name) : 
        _file(fopen(name, "r"))
    { 
        // unable to open the file?
        if (!_file) throw std::runtime_error{ "Unable to open file: "s + name };
    } 

    /**
     *  Destructor
     */
    ~File()
    { 
        // close the file
        fclose(_file);
    } 

    /**
     *  Convert to the underlying wrapped file
     */
    operator FILE *() & 
    { 
        return _file;
    } 

    // TODO: Member functions for working with the file
};

This works well. It is not possible to retrieve the underlying FILE pointer from an unnamed temporary directly. However, if we make the casting operator also const-qualified, this no longer seems to work.

Different compilers simply swallow it without complaint even though it's a terribly useful idea. Take, for example the std::string::c_str() member function. You feel it should be reference-qualified (because otherwise you have an invalid pointer) yet it isn't.

Is this a hole in the C++11 standard? Am I missing something here?