Compare 2 files and run the script when there is a difference

Question

I have a shell script for working with text conversions, but it is cumbersome to use and relies on manual judgment and execution. It works as follows.

Step 1: download raw data text.

wget http://example.com/raw.txt

Step 2: Manually compare the contents of the last downloaded file for differences.

The content format of the text is relatively simple, one string per line. If the content has increased, you need to run script processing.

# Last downloaded file content:
wtKpjv
uJlQm1
iS86aE
Hk6ycS
hAYj7k

# Now downloaded file content:
wtKpjv
uJlQm1
RiU8iM
iS86aE
Hk6ycS
qyDNaZ
hAYj7k

# Compare the increased content:
RiU8iM
qyDNaZ

Step 3: If there is a difference in the comparison results, you need to run script processing.

./text_processing.sh raw.txt > new.text

The above is my current usage steps. How do I set up an automated task to complete this process?

I want to check it once a day. If the text content is updated, I need to run the script immediately. If there are no changes, the script is run every 3 days.

I would really appreciate if anyone can help me..

Answer 1

Your question is a bit vague in what you require.

If you only want to check for any differences, use cmp -s oldfile newfile. That will exit with a true status if the files are the same:

if cmp -s oldfile newfile
then echo files are the same
else echo files are different
     ./text_processing.sh ...
fi

If you want to check that the file size has increased, then you can use wc -c (for character count), wc -l (for line count), or stat --format=%s which directly uses the metadata about the file to give the total size in bytes.

if [ $(wc -c oldfile) -lt $(wc -c newfile) ]
then echo oldfile is smaller
     ./text_processing.sh ...
fi

$( ... ) runs the enclosed command and substitutes its output into the command line. [ ... -lt ... ] tests whether the first argument is less than the second argument.

If you specifically want to check that a line has been added, then the best strategy is probably to first sort both files, and then use comm to filter out common lines:

sort -o oldfile.sorted oldfile
sort -o newfile.sorted newfile
if [ $(comm -13 oldfile.sorted newfile.sorted | wc -l) -gt 0 ]
then echo 'line(s) only found in newfile'
     ./text_processing.sh ...
fi

comm -13 matches lines from both files. Normally every line is output, with an indent to indiciate whether the line only occurs in the first file, or only in the second file, or in both. With -13 the lines from only the first file and the common lines are suppressed, so only those lines occuring in the second file is output. That is piped into wc -l which counts the lines, and that is tested to be greater than 0.

Note that changed lines will be represented by one line only in the first file, and another line only in the second file.