13

To backup a snapshot of my work, I run a command like tar -czf work.tgz work to create a gzipped tar file, which I can then drop in cloud storage. However, I have just noticed that gzip has a 4 GB size limit, and my work.tgz file is more than 4 GB.

Despite that, if I create a gzip tar file on my current computer (running Mac OS X 10.15.4, gzip version is called Apple gzip 287.100.2) I can successfully retrieve it. So gunzip works on a >4GB in my particular case. But I want to be able to create and read these large gzip files on either Mac OS X or Linux, and possibly other systems in the future.

My question is: will I be able to untar/gunzip large files anywhere? In other words, how portable is a gzip file which is more than 4 GB in size? Does it matter if I create it on Mac OS, Linux, or something else?

A bit of online reading suggests gzip will successfully gzip/gunzip a larger file, but will not correctly record the uncompressed size, because the size is stored as a 32 bit integer. Is that all the limit is?

Improve this question
2
  • 1
    Related: Is it possible to compress a very large file (~30 GB) using gzip? Commented Oct 5, 2020 at 1:14
  • 3
    Just for the record (since it’s not your use case), apart from gzip potential limitations (I don’t know about them), remember that some file systems have a maximum file size limit too. I’m thinking of FAT32, for instance, which has a 4 GiB file size limit. Commented Oct 5, 2020 at 9:40

2 Answers 2

Reset to default
27

I have just noticed that gzip has a 4 GB size limit

More accurately, the gzip format can’t correctly store uncompressed file sizes over 4GiB; it stores the lower 32 bits of the uncompressed size, and gzip -l misleadingly presents that as the size of the original data. The result is that, up to gzip 1.11 included, gzip -l won’t show the right size for any compressed file whose original size is over 4GiB.

Apart from that, there is no limit due to gzip itself, and gzipped files over 4GiB are portable. The format is specified by RFC 1952 and support for it is widely available.

The confusion over the information presented by gzip -l has been fixed in gzip 1.12; gzip -l now decompresses the data to determine the real size of the original data, instead of showing the stored size.

Will I be able to untar/gunzip large files anywhere?

Anywhere that can handle large files, and where spec-compliant implementations of tar and gunzip are available.

In other words, how portable is a gzip file which is more than 4 GB in size?

The gzip format itself is portable, and gzip files are also portable, regardless of the size of the data they contain.

Does it matter if I create it on Mac OS, Linux, or something else?

No, a gzip file created on any platform can be uncompressed on any other platform with the required capabilities (in particular, the ability to store large files, in the context of this question).

See also Compression Utility Max Files Size Limit | Unix/Linux.

Improve this answer
7
  • Does it actually store something when used in a pipeline? Commented Oct 5, 2020 at 15:28
  • 2
    @SimonRichter good question, and it does: echo "This is a pipe test" | gzip -9 | cat > file.gz (extra cat to ensure gzip is writing to a pipe) produces a file.gz which shows accurate info with gzip -l. gzip -l doesn’t show sizes when given its input from a pipe however; compare gzip -l file.gz, gzip -l < file.gz and cat file.gz | gzip -l. Commented Oct 5, 2020 at 15:38
  • It can (could) store/know the size if the compressed output is only a single block. Once it had to stream the first compressed block out, it can no longer modify the header. Commented Oct 5, 2020 at 20:59
  • 1
    @eckes, that linked RFC seems to indicate that the size field is in the tail part of the file, and repeating Stephen's command with some bigger input still gives me the right size from gzip -l. Commented Oct 5, 2020 at 21:41
  • @StephenKitt, also, gzip -l seeks back from the end of the file to get to the size from there, and it can't do that from a pipe. Commented Oct 5, 2020 at 21:42
1

Besides limits of filesystems and storage systems and encryption containers, it might also be better to have smaller chunks for file transfers — especially if you use tools which do not support parallel upload, and restart of aborted transfers.

With smaller chunks you can work around all that, so I typically aim for 2 GB chunks everywhere. A checksum error is better to handle if you need to process 1 x 2 GB chunk again instead of a 100 GB file. Tar can normally split into multiple “tapes”.

If you can control your whole chain including USB sticks, storage accounts and transmission software, you don’t need to care (but especially for frictionless restores in panic moments having more flexibility is a plus anyway).

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.