Getting offset and back with aligned pointer in Visual Studio

I'm porting an application to Visual Studio 2012 (in C++11), which currently compiles and runs correctly on gcc on several platforms.

When I tried to run it, something odd (at least to me) happened. After some moments I isolated the problem, and came up to this:

// This one succeeds
assert(jumper.fst == (left_prog->buffer - program->buffer));

// This one fails!
assert(left_prog->buffer == (jumper.fst + program->buffer));

The program->buffer and left_prog->buffer are of type Instruction *, and jumper.fst has type ptrdiff_t.

It wouldn't return to the original value; upon some testing, I realized that this had something to do with the aligment. My Program type is aligned (I've written my own allocator), and this was causing the problem. After I multiplied the aligment by sizeof(Instruction), it worked as expected.

Could someone please explain to me why this is happening? Is this behaviour documented somewhere? Am I using undefined behaviour here?

(As I said, I already fixed the code by changing the aligment value, making it a multiple of sizeof(Instruction), I just want to know why this happened.)