How to store a large folder in a single file without compression

Question

I want to take a 78gb folder and store it in a single file (for upload into a cloud service), as if I am compressing it in an archive, but I don't want any compression (I don't have that much CPU time available). Is there anyway that I can accomplish this, perhaps a terminal command I don't know about?

Answer 1

Use tar: tar -cf my_big_folder.tar /my/big/folder

Restore the archive with tar -xf my_big_folder.tar -C /

-C will change to the root directory to restore your archive since the archive created above contains absolute paths.

EDIT: Due to the relatively big size of the archive, it'd be best to send it [directly] to its final location, using SSH or a mount point of the cloud resource/folder. For example, as Cole Johnson suggests :

tar -cf /network/mount/point/my_big_folder.tar /my/big/folder

or

tar -c /my/big/folder | ssh example.com "cat > my_big_folder.tar"

EDIT: As Blacklight Shining also suggests, If you want to avoid absolute paths, you can change to the big folder's parent and tar from there:

tar -cf /network/mount/point/my_big_folder.tar \
    -C /my/big/folder/location the_big_folder

or

tar -cC /my/big/folder/location the_big_folder | \
ssh example.com "cat > my_big_folder.tar"

---

Personal reflexions

Whether to include relative or absolute paths is a matter of personal preference.

There are cases absolute paths are obvious, e.g. for a restore in a disaster recovery situation. For local projects or collections it's common to archive a directory tree from the desired folder's parent so as to avoid cluttering the current directory, in case the archive is accidentally unpacked in-place. If big_folder lies somewhere deep in a standard *NIX hierarchy, it may make some sense to start archiving the first non-standard folder where big_folder deviates from and its directory tree from there.

Finally — going pedantic here — tar archive members are always relative since a) they may be restored in any directory and b) tar removes the leading / when creating an archive. I personally tend to always use -C when unpacking an archive.

Answer 2

Alternative: cpio

(cd /my/big/folder && find . -depth -print0 | cpio -0o > myfolder.cpio)

Unpacking to current directory:

cpio -id < myfolder.cpio

Caveats:

• If use find /my/big/folder instead of cd, the archive will contain full paths and extraction will try to follow them;
• Big files (> 2GB) may be a problem;

Answer 3

To store a large folder into a single file without compression and with a progress bar, one can use the following one-line command:

tar -cf - ./Downloads/ | (pv -p --timer --rate --bytes > backup.tar)

The command will keep the original folder intact.

Requirements:

sudo apt-get install -y pv

Demo (in my case, my folder is ./Downloads/ and my output file is adb_int.tar):

Credits: the command combines user86969's answer with Jakuje's answer.