You forget one crucial thing, namely that your program will have to interact with the operating system to do anything interesting.
The conventions are different between Linux and OS X so the same binary cannot run as-is without essentially having a chunk of operating system dependent code to be able to interact with it. Many of these things are hidden away inprovided through libraries, which you then need to link in, and that means your program needs to be linkable, and linking is also different between the two systems.
And so it goes on and on. What on the surface sounds like doing the same thing is very different in the actual details.