I know that similar questions have been asked about permissions, but which form of compression or archiving keeps the permissions and file owners of each file and directory?
I was considering the tar.gz format, but would this be right?
I need to move 37GB of files and directories to another server and need everything to be exactly the same when uncompressed.
Both traditional archiving tools tar and cpio preserve ownership and Unix permissions (user/group/other) as well as timestamps (with cpio, be sure to pass -m when extracting). If you don't like their arcane syntax¹, you can use their POSIX replacement pax (pax -w -pe) All of these output an uncompressed archive; pipe the archive into a tool like gzip or xz to compress it (GNU tar has options to do the compression). Users and groups are identified by their name; GNU tar has an option .
None of these tools preserve modern features such as ACL, capabilities, security contexts or other extended attributes.
Some versions of tar can store ACL. See What to use to backup files, preserving ACLs? With GNU tar, pass --acls both when creating and extracting the archive.
A squashfs filesystem, as suggested by mikeserv, stores capabilities and extended attributes including SELinux context, but not ACL. (You need versions that aren't too antique.) If you have both ACL and security contexts, you can use a squashfs filesystem, and save the ACL by running getfacl -R at the root of the original filesystem and restore them after extracting the files with setfacl --restore.
If you want to save absolutely everything including ACL, subsecond timestamps, extended attributes and filesystem-specific attributes, you can clone the filesystem. The downside of this approach is that you can't conveniently directly write a compressed copy. The ultimate way to clone the filesystem is to copy the block device; of course this is wasteful in that it copies the empty space. Alternatively, create a filesystem that's large enough to store all the files and use cp -a with the cp command from GNU coreutils to copy the files; GNU cp is pretty good at copying everything including non-traditional features such as extended attribute and ACLs.
¹ Though this one is really overblown.
If you're talking about linux systems then another option for you is squashfs. It can often achieve very high compression ratios - and the compression process itself is multi-threaded - which means that you can apply all processor cores to the compression task.
A squashfs archive differs from most other kinds in that it is a file-system. If you've ever booted a linux live disc then you've very likely already seen this in action - very nearly all of these work by mounting a squashfs archive as their root file-system. squashfs is supported in vanilla linux kernels since version 2.6.34. And so it is fairly universal to any modern linux system.
Squash supports any of gzip, lzma, lzo, xz, or (since kernel 3.19) lz4 compression methods. To mount and access the contents of a squash archive you shouldn't need any tools at all and can just do:
mount ./img.sfs /mnt; cd /mnt
...to get at it. The contents of the archive will then be provided you by the kernel's native vfs as a read-only mount, and you can read all files within in the same way you would any other kind of file - and without having to decompress the archive (which is handled on the fly into disk-cache on an as-needed basis by the linux kernel itself). In fact, with the latest kernels' lz4 support, you can likely do this at native speed (and perhaps even faster in some cases) as well - though there will be increased cpu-usage for squashed file access.
To create or decompress a squashfs archive you will need the squashfs-tools package installed. It is not generally installed by default by any distribution of which I am aware, but neither am I aware of any distribution which does not provide the package via package-manager. Once installed you can create an archive like:
echo 'this is my new file' >~/Downloads/newfile.txt
mksquashfs ~/Downloads ./mysqsh.sfs -comp xz
Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on ./mysqsh.sfs, block size 131072.
[===================================================-] 1018/1018 100%
Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
compressed data, compressed metadata, compressed fragments, compressed xattrs
duplicates are removed
Filesystem size 12592.01 Kbytes (12.30 Mbytes)
57.19% of uncompressed filesystem size (22019.04 Kbytes)
Inode table size 8482 bytes (8.28 Kbytes)
23.91% of uncompressed inode table size (35477 bytes)
Directory table size 10210 bytes (9.97 Kbytes)
42.90% of uncompressed directory table size (23802 bytes)
Xattr table size 3976 bytes (3.88 Kbytes)
48.67% of uncompressed xattr table size (8170 bytes)
Number of duplicate files found 61
Number of inodes 1064
Number of files 926
Number of fragments 68
Number of symbolic links 6
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 132
Number of ids (unique uids + gids) 1
Number of uids 1
mikeserv (1000)
Number of gids 1
mikeserv (1000)
mksquashfs ... 7.08s user 0.35s system 462% cpu 1.607 total
As you can see - it clearly respects file permissions - and even preserves and respects extended file-system file attributes (xattrs) in most cases. And the compression ratio you see noted there is in addition to my regular file-system's default lzo compression - (my root fs is btrfs and all files are compressed w/ lzo already) - which is not to mention that much of ~/Downloads is occupied by downloaded compressed archives in the first place.
It is immediately mountable as a proper file-system in its own right:
sudo mount ./mysqsh.sfs /mnt; \
cd /mnt; cat newfile.txt; cd -; \
sudo umount /mnt
this is my new file
As always, root permissions are generally required for arbitrary mounts, but a squashfs mount can be named like any other in /etc/fstab if you desire it.
And last, you don't need any elevated privileges to decompress a squashfs archive, only the unsquashfs tool (which is also provided in the squashfs-tools package):
unsquashfs -d /tmp/mysqsh ./mysqsh.sfs; \
cd /tmp/mysqsh; cat newfile.txt
Parallel unsquashfs: Using 6 processors
933 inodes (1025 blocks) to write
[=================================================|] 1025/1025 100%
created 927 files
created 132 directories
created 6 symlinks
created 0 devices
created 0 fifos
this is my new file
Yes. tar retains the owner and permissions.
Wikipedia: tar
The archive data sets created by tar contain various file system parameters, such as time stamps, ownership, file access permissions, and directory organization.
man tarexplains all this...you do have to actually read it tho (see e.g., "Handling of Attributes").