Prevent removal of not yet created files

Question

I'm trying to fix my YARN problem. When task is submitted to YARN it's creates new directory with all needed settings and scripts. After finishing this task directory is removed.

My task is failing after 0-2 seconds so files are removed so fast I cannot save them. I also don't know exact name of file before running task (there is autoincementing counter), but I know parent directory and I could try to guess exact path.

I would like to protect or copy this files (whole directory) in some way. I guess I could set up cron running every second and copying parent directory (I don't know if size of files wouln't exceed cp capabilities). I probably could also change (every second) every file in directory into read-only mode (but it could interfere with YARN processes and create new problems). I can't change permissions to directory - YARN wouldn't be able to create them in first place.

Is there better solution? (YARN detail is not important, but maybe someone knows how to use some unknown to me YARN features).

Answer 1

Several of your ideas won't work.

• Cron works only at the minute intervall.
• Making files read only doesn't prevent deletion.
• Making the directory read only prevents deletion, but also creation.
• On the other hand, no size of a file exceeds cp capabilities.

You best solution is to find a way to disable the deletion of the temporary files.

If that doesn't work, the best way is not to copy, but to link the files. If you know the directory where the files will be created, use this in a second shell:

while true; do
    ln sourcedir/* targetdir &> /dev/null
done

You have to terminate this after you are done. sourcedir and targetdir must be on the same file system. this will create a hard link of every file. When it runs again, the target exists, so it would display an error message, therefor the redirect to /dev/null. Unless the files in the source are removed very quickly, you should have all your files in the target directory.

Edit

For a limited number of nested directories, use

ln sourcedir/* sourcedir/*/* sourcedir/*/*/* targetdir &> /dev/null

For an arbitrary level of nesting, use find

find sourcedir -type f -exec ln --target-directory targetdir {} +