Here is the scenario, I have 1000 files in a folder each ~3 MB. I want to zip all files into 50MB each zip and remove the original files.
NOTE: 50MB may contain 20 files or 10 files but that zip package should be <=50MB.
I want to transfer those files without data loss. File must be of format (tar/gzip/bzip). Please provide suggestions to me if we have any other way to overcome the data loss.
Need to create a shell script.
One solution is to use the split command.
The split command will split an archive in to multiple files, doing all the hard work for you.
Here is an example:
tar -cvf - file1 file2 file3 | split --bytes=50m --suffix-length=4 --numeric-suffix - myarchive.tar.
And to untar:
cat myarchive.tar.* | tar xvf -
The zip package supports bzip2 compression and --split.
Will bzip2 meet your requirements?
-s splitsize --split-size splitsize Enable creating a split archive and set the split size. A split archive is an archive that could be split over many files. As the archive is created, if the size of the archive reaches the specified split size, that split is closed and the next split opened.
...
Split size is a number optionally followed by a multiplier. Currently the number must be an integer. The multiplier can currently be one of k (kilobytes), m (megabytes), g (gigabytes), or t (terabytes). As 64k is the minimum split size, numbers without multipliers default to megabytes. For example, to create a split archive called foo with the contents of the bar directory with splits of 670 MB that might be useful for burning on CDs, the command:
zip -s 670m -r foo bar
could be used.
To ensure data integrity, rsync with checksum verification. It's much slower, but will calculate checksum on both sides of the transfer.
-c, --checksum
This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a lqquick checkrq that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.
untested
cd /the/directory
files=(*)
i=0
z=0
create_zip=true
for ((i=0; i<${#files[@]}; i++)); do
if $create_zip; then
((z++))
zip_file=prefix.$z.zip
create_zip=false
fi
# add the file
zip $zip_file "${files[i]}"
# check the size
if (( $(stat -c %s $zip_file) >= 50000000 )); then
# remove the previous file
zip -d $zip_file "${files[i]}"
create_zip=true
# decrement the file index so this file gets added to the next zip
((i--))
else
echo rm "${files[i]}" ### remove "echo" if it's OK
fi
done
