The normal way to access a filesystem at an offset on a disk is with a partition. However, since you don't have room for a partition table at the beginning of the disk, you'll have to go through a more complex method.
You can use a loop device. Loop devices are mostly used to make a file appear as a block device, but you can make the block device start at an offset (and end at an offset, too), and the underlying file can itself be a block device, so you get a block device that corresponds to a segment of another block device. The command to manipulate loop devices is losetup.
losetup -o 1600g /dev/loop0 /dev/sda
Make sure that your filesystem indeed ends before the start of the loop device. Then you can copy it wholesale. I don't see why you would need to make a different filesystem¹: you shouldn't duplicate a btrfs filesystem on the same machine, but moving one (i.e. you won't ever access both at the same time) is ok.
head -c 1100g /dev/sda >/dev/loop0
mount -r /dev/loop0 /mnt
# Check that the filesystem looks fine
umount /mnt
losetup -d /dev/loop0
You could also do that by a suitable invocation of dd, but that's error-prone. It's easier to proceed step by step and to verify sizes before destroying data if you first set up block devices and then shuffle data around.
Now that you've moved the filesystem, create partitions, so that you won't run into this problem again.
fdisk /dev/sda
…
Create a partition that starts at the same offset that you passed to losetup above. Write the changes and verify that you can mount that partition.
Now create a partition for the LUKS volume, create the LUKS volume (cryptsetup luksFormat), and move the btrfs filesystem again onto the LUKS volume. Verify that the filesystem is still fine. Delete the partition at the end of the disk and extend the partition at the beginning of the disk. Extend the LUKS volume (cryptsetup resize), then extend the filesystem.
¹ It's straightforward to do it — mkfs.btrfs /dev/loop0 — but this is not the best solution to your problem.