0

Suppose, I have some file with path %p which I need to gzip on the fly and send to a remote server and I'm not allowed to use rsync and similar mirroring tools. I do the following:

gzip -c -9 %p | ssh user@server  "cat > backupPath"

In the basic normal case it works good, but I'm wondering what happens when the connection to the remote server fails during the file sending because I'd like to be sure the file is fully sent and saved. Will just the part of the file be written to "backupPath" or will it follow the "all or nothing" strategy - i.e. error happens, file with "backupPath" address is not created on the remote host (which is more suitable for me)?

Improve this question

2 Answers 2

Reset to default
2

From the point of view of the cat process it is just copying data from stdin to stdout, and it knows nothing about if the data at all, and in particular if it is complete or not. So the answer to your question is no it is not an "all or nothing" strategy.

You can do things in 2 steps which will make it more robust.

 #/bin/bash
 set -e
 set -o pipefail
 gzip -c -9 %p | ssh user@server  "cat > backupPath.tmp"
 ssh user@server  "mv backupPath.tmp backupPath"

Note the %p was copied from the original post, it is not shell syntax.

Improve this answer
2
  • Yes, cat doesn't know about data, but probably it has to know about the end of the stream because it should close the file handle. So, are you sure with your answer in the case of cat? Commented Sep 4, 2019 at 15:53
  • In general programs do not close stdin and stdout, and expect the operating system to close them instead. Nothing to say they couldn't, but it is unusual. Commented Sep 4, 2019 at 19:30
0

It looks like renaming file after download completion is the option since the renaming is an atomic operation. As far as I understand checking the zip integrity doesn't guarantee the file is downloaded completely - there is a very small probability that the part of the file is also correct zip file. But, anyway, I like the idea of drewbenn to check that integrity for an additional safety. So, thanks to icarus and drewbenn answers I've come to the final decision:

gzip -c -9 %p | ssh user@server 'set -e; cat > /var/tmp/file.txt.part; gzip -t /var/tmp/file.txt.part; sync /var/tmp/file.txt.part; mv /var/tmp/file.txt.part /var/tmp/file.txt'
Improve this answer
8
  • You could throw in a sync before the gzip -t to ensure the file has reached stable storage. You need to be sure that /var/tmp is not wiped on reboots or periodically purged of old files. Commented Sep 4, 2019 at 19:36
  • icarus, sorry, what do you mean on "throw in a sync"? Commented Sep 5, 2019 at 3:51
  • Change .part; gzip to .part; sync; gzip so that the system writes all outstanding data to the disk, rather than leave it in RAM. The interaction between data (the stuff in files) and metadata (things like filenames) is interesting and complicated. You don't want data to be still buffered in RAM, so it will be eventually written to /var/tmp/file.txt.part but it isn't there yet, when /var/tmp/file.txt.part is renamed to /var/tmp/file.txt. The rename is atomic, but you may or may not have the data actually written to the disk. (You probably will, but why take chances?). Commented Sep 5, 2019 at 5:55
  • icarus, makes sense, but what about renaming then? Is there a chance the renaming itself might not be flushed to disk, so we have to make sync after the renaming as well? Commented Sep 5, 2019 at 6:40
  • I've made a change according to icarus answer. I don't think checking the integrity needs the data flushed - it may be done with reading from the cache buffers, so, I do sync after the integrity check is done, and if synced - rename the file. But I don't do the sync after the file rename. If the system will fail before the rename metadata flushed we will just need to resend the file and that's probably Okay. Commented Sep 5, 2019 at 16:30

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.