Using `extern template` with third-party header-only library

I am using the glm library, which is a header-only collection of math utilities intended for 3D graphics. By using -ftime-trace on Clang and ClangBuildAnalyzer, I've noticed that a lot of time is being spent instantiating glm types:

**** Templates that took longest to instantiate:
 16872 ms: glm::vec<4, signed char, glm::packed_highp> (78 times, avg 216 ms)
 15675 ms: glm::vec<4, unsigned char, glm::packed_highp> (78 times, avg 200 ms)
 15578 ms: glm::vec<4, float, glm::packed_highp> (78 times, avg 199 ms)

...

---

So, I decided to create a wrapper header/source pair for glm, and use extern template to avoid unnecessary instantiations:

// glmwrapper.h

#pragma once

#include <glm.hpp>

extern template struct glm::vec<4, signed char, glm::packed_highp>;
extern template struct glm::vec<4, unsigned char, glm::packed_highp>;
extern template struct glm::vec<4, float, glm::packed_highp>;

// glmwrapper.cpp

template struct glm::vec<4, signed char, glm::packed_highp>;
template struct glm::vec<4, unsigned char, glm::packed_highp>;
template struct glm::vec<4, float, glm::packed_highp>;

Now, in my project, instead of including <glm.hpp>, I include "glmwrapper.h" instead. Unfortunately, that did not change anything. Using -ftime-trace and ClangBuildAnalyzer again reports the same number of instantiations.

I suspect that this is because #include <glm.hpp> does actually end up including the template definition, and at that point the subsequent extern template declarations are just redundant.

Is there a way to achieve what I want without modifying the glm library?

---

In pseudocode, I kinda want something like this:

// glmwrapper.h (psuedocode)

#pragma once

#include <glm.hpp>

// Make definition of the templates unavailable:
undefine template struct glm::vec<4, signed char, glm::packed_highp>;
undefine template struct glm::vec<4, unsigned char, glm::packed_highp>;
undefine template struct glm::vec<4, float, glm::packed_highp>;

// Make declaration of the templates available:
extern template struct glm::vec<4, signed char, glm::packed_highp>;
extern template struct glm::vec<4, unsigned char, glm::packed_highp>;
extern template struct glm::vec<4, float, glm::packed_highp>;

// glmwrapper.cpp (psuedocode)

// Define templates only in the `.cpp`, not in the header:
template struct glm::vec<4, signed char, glm::packed_highp>;
template struct glm::vec<4, unsigned char, glm::packed_highp>;
template struct glm::vec<4, float, glm::packed_highp>;