A very simple solution:

1. I create the swap file on a btrfs filesystem. Btrfs has transparent compressed file support.
2. There is no way to just swapon it; because btrfs has also CoW (copy-on-write) support enabled by default, which is required for transparent compression. But, to have swapfile in btrfs, turned off CoW is needed (Cow can be turned on or off by the chattr -C / +C command).
3. However, our schweizer knife helps this time: the loopback block devices. So losetup /dev/loop0 /swap and we have what we wanted.
4. After that, I could freely swapon /dev/loop0.

At the moment I do not know, maybe it might cause deadlocks or similar, but I think it is very unlikely. It might have some speed degradation, because swapped out pages have to walk twice in the block cache. Time will say that.

You can have a loop block device on anything, then you can swap on that loop device.

In details, this very simple solution:

1. I create the swap file on a btrfs filesystem. Btrfs has transparent compressed file support.
2. There is no way to just swapon it; because btrfs has also CoW (copy-on-write) support enabled by default, which is required for transparent compression. But, to have swapfile in btrfs, turned off CoW is needed (Cow can be turned on or off by the chattr -C / +C command).
3. However, our schweizer knife helps this time: the loopback block devices. So losetup /dev/loop0 /swap and we have what we wanted.
4. After that, I could freely swapon /dev/loop0.

At the moment I do not know, maybe it might cause deadlocks or similar, but I think it is very unlikely. It might have some speed degradation, because swapped out pages have to walk twice in the block cache. Time will say that.