Contiguous copy – How do I copy files contiguously?

Question

I want to copy a number of files onto a flash drive such that each individual file is contiguous. They do not need to be contiguous with each other, and can be in any order. The files will be very large – hundreds of megabytes, to gigabytes. No other files on the drive should be defragmented; doing so would waste time and cause unnecessary wear on the flash media.

I want to be able to do this for at least FAT32, but methods for *nix filesystems will also be appreciated.

Essentially, there are 2 approaches to this:

• Copy then defragment the files.
• Defrag enough contiguous free space for each file, then copy each file into its place.

The 2nd option would generally be far quicker than the 1st, and would avoid causing unnecessary wear on the flash media, so the 2nd option would be very much preferable.

I don't mind solutions that work offline, but obviously online is preferable.

Answer 1

If you need to do this on linux:

rsync --preallocate /path/to/source/file /path/to/destination/

rysnc preallocates a contiguous block of storage and copies the file into it. Works for FAT and NTFS too.

Just make sure the file does not already exist at the destination, or rsync won't reallocate and re-copy it. If it is, delete it, Empty Trash to make sure it's really gone, then run this command.

Verify if it copied contiguously:

filefrag /path/to/destination/file

"1 extent found" means the file is contiguous. More than one means it's fragmented.

If it's still fragmented, use filefrag to check if the source file is fragmented or not. If it is, use rsync --preallocate to copy the source file in place with a new name, defragmenting the source. Then rsync it again to the desired destination. Sometimes multiple rounds of this can reduce both the source and destination files down to 1 extent.

If it's still fragemented after that, then you probably don't have enough contiguous space on the destination drive to hold the full file. Defragment the full destination drive, or use filefrag to check large files on that drive for fragmentation and defrag them in place with rsync --preallocate, then try again.

Finally, if you're rsync'ing to an NTFS drive, NTFS reserves a small section at the 3GB point for directory files, which may prevent a file from being contiguous if it overlaps that 3GB point. See the Easy2Boot discussion on that here:

https://easy2boot.xyz/create-your-website-with-blocks/make-using-linux/#Making_files_contiguous

(Easy2Boot also includes the defragfs and udefrag utilities for defragmenting FAT and NTFS drives from Linux, which may be useful if you need to defrag a full drive)

Answer 2

I have to admit that I don't really understand what you're trying to do but I don't think that matters. I found this tool called defragfs which purports to do what you're asking (if I understand what you're trying to do).

http://defragfs.sourceforge.net/index.html

I think the thing with this tool though, is that it defrags after, where I think you're looking to essentially guarantee that as you write the file to disk, that it's done so in a sequential manner. According to this StackOverflow Q&A titled: how to store a file in continuous disk block in linux, it doesn't sound like's is possible given the architecture of how the writes are done.

Just to round out this Q&A I did find the following resources which sound like they might be useful.

• filefrag man page - tool for reporting a file's fragmentation
• tutorial on using filefrag
• defragfs website
• defragfs theory behind file fragmentation
• forum thread from reboot.pro regarding contiguous files in linux

Answer 3

It is possible at least for FAT32, however I don't recall the name of the tool.

If I remember right is based on a first call for a full length hole and assign it to the new stream, then the file is read and write on that stream.

It is also possible with NTFS via pre-allocate copy if I remember right.

But, for NTFS compressed files there is no way... since Windows first write the file (non-compressed) and after that it compress the file in chunks... to tell the truth it does that in a pipeline... that is the reason the file get so fragmented (a file of 10GiB can create more than one hundred thousand fragments).

I do not know any tool / command able to avoid that when NTFS compression is enabled. I used a tool a long time ago that had a nice GUI to copy without causing fragmentation in Windows, but I can't seem to recall the name.

So the best I can tell you is to use on windows the command line utility XCOPY with the /J flag.

It tries to not fragment, ideal for big files if NTFS compression is disabled or you are using FAT32.

Explanation for NTFS compression pipeline fragmentation:

1. Block N will be written on Cluster N
2. Clusters arround N+# will be compressed and stored somewhere else, so clusters between N and N+# will be free, aka, file will be fragmented, very fragmented. 10GiB = 100000 fragments, assuming a compression of 50%, that is why NTFS compression is very BAD. If the file would be compressed first on RAM then sent to disk no fragmentation would occur or at least it can be avoided.

Another side effect of this way of doing things is assume we have 5Gib of free space and I want to write a file of 6GiB that after compressing will take only 3GiB. This is not possible, but if you first move a 2GiB (non-compressible) to another place, then free space will be 7GiB, write 6GiB, compressing makes it only 3GiB, free space will be 4GiB, then put back the 2GiB of original data that we moved and voila all is there and 2GiB free. Point being that if there is not enough space for uncompressed file, it will not be possible to be copied on NTFS and it does not matter if after it would be NTFS compressed there is enough space. It needs all because it first write it without compression, then it applies the compression. Lastly NTFS drivers do that in pipeline so in theory it would still be possible, but they did not change the check free size part.

Windows does not write the file "after" compressing it, windows "compress" the file "after" saving it uncompressed, so for each cluster the hard disk will see two write attempts, first with non compressed data, second with compressed data. Modern pipeline NTFS controllers avoid HDD to see both writes but on first NTFS version the HDD writes the whole file uncompressed, then it compresses the file. It was very noticeable with very large and well compressible files. Writing a file of 10GiB that NTFS compressed down to only a few MiB (A file filled with zeros) took longer to write than writing the non-compressed version of the file. Nowadays pipeline has break that, and now it took a very little amount of time but the prior enough free size is still there.

Hopefully one day the NTFS compression method will do it to a row, so we could get non-fragmented NTFS compressed files without the need to de-fragment them after we write them. Untill then the best option is XCOPY with the /J flag and CONTIG or that GUI tool I can't remember the name of. It was while Windows was only up to XP, and had options to pause, to copy in parallel from HDD1 to HDD2 and HDD3 to HDD4, and of course the one wished, to pre-allocate. Hopefully this helps!

Answer 4

just found this for FAT32 only:

• CFSpace