EDIT 2: change of enclosure was the culprit
The comment of @frostschutz made me realize:
The question remains "how did this drive shrink in the first place". Did you change enclosures or controllers? Or enable raid mode in a controller bios or something?
I had the drive connected through SATA on an old laptop. Deleted the partitions, made the LUKS container, copied everything I needed and then took it out. I was trying to use it with a USB/SATA cable, and that was the source of the problem. I just installed it again on the old laptop and works with no problem.
Now I'll add a partition table and build the LUKS container from 0. My question now is, the partition table will avoid this issue or should I still be careful about the block size?