clang osx march=native approximately 1/3rd speed of no optimisation

I have a project where speed is paramount, so was experimenting with compiler flags to try and get some free performance. I have two builds that are identical except for the additional flag march=native in build 2.

For completeness the flags are:

A) -std=c++14 -g -fno-omit-frame-pointer -O3

B) -std=c++14 -g -fno-omit-frame-pointer -O3 -march-native

Running benchmarks on these builds yields a confusing result:

A) 61s

B) 160s

What can possibly going on here?