C++ const_cast operator for multi dimensional array

In general I'm searching for a way in C++11 to encapsulate an overloaded const_cast operator for a multi dimensional array in a structure / class which defines operations on this array. I've searched a lot in this group, but I can't really find the answer to my problem described below.

More specific I'm dealing with a 4x4 matrix which is defined in a 3rd Party C API as typedef on the double[4][4] and functions using this matrix are provided by the API.

The API looks like the following code:

typedef double ApiMatrix[4][4];

bool ApiMatrixFunction(ApiMatrix matrix)
{
    // .. some code
    return true;
}

I have implemented the structure MyMatrix to encapsulate operations on this data type:

struct MyMatrix
{
private:
//public:
    double m[4][4];

public:

    MyMatrix() { ... }

    // lots of operations and initialization members
    ....

    // overloading typical calculation operators +,-,*, ....
    ....

    // cast operator to the API data type
    operator ApiMatrix& ()
    {
        return m;
    }
};

This works well when using MyMatrix as a reference (MyCodeWithRef), but it makes trouble to use it as a constant reference (MyCodeWithConstRef). Possible workarounds are to duplicate the variable in the function or to give access to the private data and cast it in place by const_cast(matrix.m).

// Using MyMatrix reference
void MyCodeWithRef(MyMatrix& matrix)
{
    ApiMatrixFunction(matrix);
}

// Using MyMatrix constant reference
void MyCodeWithConstRef(const MyMatrix& matrix)
{

    // Unfortunately this fails
    ApiMatrixFunction(matrix);

    // workaround 1 copy the matrix
    MyMatrix m = matrix;

    // workaround 2 use a cast operator in this function
    // requires access to the private m.
    ApiMatrixFunction(const_cast<double(*)[4]>(matrix.m));
}

Both workaround have obviously disadvantages and I'm searching for a way to define the const_cast operator in the MyMatrix structure so I could use the same call for the const reference as for the reference.