I would say it's not impossible but really hard to manage. Since a so 32bit OS is usually packaged with (and accepts) 32bits only binaries and libraries, you'd need to heavily tweak the system to make it work with 64bits ones.
The main problem you'd be facing with a RPI3 is the lack of 64bits kernel (at least with raspbian).
Long story short : use 32bits binaries and you'll be fine.
EDIT :
If you want to use a 64bits kernel, you'll need to install a distro supporting the ARM64 architecture. You should take a look at ArchLinux ARM (here), but it's not fully supported.
The information you're looking for is at the bottom of the installation tab.
You also could take a look at an official debian port , however there's still big issues with the RPI3 port, so it's up to you to decide if it's worth the trouble