Worse performance with constexpr?

I am trying to evaluate performance difference by using constexpr. I am using the following code:

#include<iostream>
using namespace std;

constexpr double factorial(int n) {

    return n==0?1:n*factorial(n-1);
}

main() {
    double a;
    for(int i=0;i<10000000;i++) {
        a=factorial(100);

    }
    cout<<a<<endl;
}

I tried out two versions of the above program, one with the factorial function as constexpr, and one without. I expected to see the constexpr version perform better during runtime, but it in fact, runs slower. Here are the measurements (in seconds) from 4 trials each:

Without constexpr:

2.491, 2.597, 2.547, 2.554

With constexpr:

2.999, 3.045, 3.035, 3.034

Could someone explain the reason behind this? Am I using constexpr wrong? I am using g++ 4.9.1, and I used the O3 optimization flag.