How to implement your own image class differetiating rgb and grayscale images in dlib?

I am new to dlib but have gone through a few examples. I'm trying to implement/extend a class, really a POD, to make it seen as an image type in dlib. This POD is a generic image represetation, with a data pointer, and a depth identifier, telling you the pixel type, how the bytes are ordered. The underlying data is row major order, so I believe it can work.

I read generic_image.h.html and How to assign memory images to a DLIB array2d or image?, and implemented all the global functions. But the question is on the image_traits. What's the best way to differentiate the rgb and grayscale images?

Should I define a templated class that takes rgb_pixel or unsigned char, like how array2d<> does it? Or should I create two classes that inherits from the POD? Or would I be able to combine these two cases in some way? If I could expose a single class, that would be great, and I can convert a vector of these PODS into a vector of the image objects, and do routines on them.

Is it something like:

namespace dlib
{
    template <>
    struct image_traits<CustomRGBImage>
    {
        typedef rgb_pixel pixel_type;
    };

    template <>
    struct image_traits<CustomGrayscaleImage>
    {
        typedef unsigned char pixel_type;
    };
}