Compare script using awk and bash

Question

I try to create compare script using awk and another command and running successful.

But I think the script I create it's to long.

Is there anyone can shorten my script below ?

After plan A shorten the code than, plan B is :

 1. I want eliminated a lot temp file (.txt), only need `lengkap.txt`
 2. Put command in variable if can

It's my pleasure if anyone can help me

Below is the code that I create based searching and trying.

#!/bin/bash

### Path Folder who will be compare ###
path1=/home/rio/apps1
path2=/home/rio/apps2

### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5

### Compare to find different folder ###
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums2.md5 checksums.md5 > hasil2.txt

### Merge result of compare ###
awk '{print $0}' hasil1.txt hasil2.txt > perbedaan.txt

### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print $2}' > hasilperbedaan.txt

### File about result compare (just filename) ###
cekhasil=/home/rio/hasilperbedaan.txt

### Check if File result compare empty or not ###
if [ -s "$cekhasil" ]
then 
   echo " file exists and is not empty "
    ### Find All filename and date, after that put as we want ###
    find $path1 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate1.txt
    find $path2 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate2.txt

    ### Compare to get the date of filename ###
    awk 'A[$1]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
    awk 'A[$1]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
    ### Merge result of compare with date ###
    awk '{print $0}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
   echo " file does not exist, or is empty "
fi

Answer 1

Well, it took time to figure out what your program is doing. So I did shorten it in two phases.

First phase: I removed all duplicate temporary files, and used a pipe when a temp file was used once.

STEP1: you just make a checksum of all files in $path1 and $path2

### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5

You don't need 2 temp files, as path is included in filename. So you can replace this with a single find. I use sort to be able to use uniq afterwards:

find "$path1" "$path2" -type f | xargs md5sum  | sort > cksum.md5

STEP2: you find unique checksums in both files, and get the corresponding filenames.

### Compare to find different folder ###
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[$1]++;next};c[$1] == 0' checksums2.md5 checksums.md5 > hasil2.txt

### Merge result of compare ###
awk '{print $0}' hasil1.txt hasil2.txt > perbedaan.txt

### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print $2}' > hasilperbedaan.txt

As we have a checksum-sorted file, we just filter with uniq, and get filenames.

Note: As we use a MD5 checksum (128 bits), length is 128bits / 8bits * 2hex=32

uniq -u -w32 cksum.md5 | awk '{print $2}' > "$cekhasil"

If you prefer awk to uniq, just compare previous record's $1 with current's.

STEP 3: you match found entries with a new search (find) in the two source paths.

if [ -s "$cekhasil" ]
then
    echo " file exists and is not empty "
    ### Find All filename and date, after that put as we want ###
    find $path1 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate1.txt
    find $path2 -type f -ls | awk '{print $11" "$8" "$9" "$10 }' > filedate2.txt

    ### Compare to get the date of filename ###
    awk 'A[$1]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
    awk 'A[$1]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
    ### Merge result of compare with date ###
    awk '{print $0}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
    echo " file does not exist, or is empty "
fi

Here, it is better to avoid new full find, and just loop over filenames.

if [ -s "$cekhasil" ]
then
    echo " file exists and is not empty "
    while read -r fn ; do
        ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
    done < "$cekhasil" > lengkap2.txt

else
    echo " file does not exist, or is empty "
fi

Alltogether, the script becomes:

#!/bin/bash

path1=/home/br/dev/tools/bash
path2=/home/br/dev/tools/bash2
outfile=hasilperbedaan.txt

find "$path1" "$path2" -type f | xargs md5sum  | sort > cksum.md5

uniq -u -w32 cksum.md5 | awk '{print $2}' > "$outfile"

if [ -s "$outfile" ]; then
    echo " file exists and is not empty "
    while read -r fn ; do
        ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
    done < "$output" > lengkap2.txt

else
    echo " file does not exist, or is empty "
fi

At this point, we can notice the 2 remaining temp files are used only once. We just get rid of them, and just pipe all together to get the output file.

    #!/bin/bash
    
    path1=/home/br/dev/tools/bash
    path2=/home/br/dev/tools/bash2
    outputfile="lengkap-v2.txt"
    
    find "$path1" "$path2" -type f | xargs md5sum  | sort |
        uniq -u -w32 | awk '{print $2}' |
        while read -r fn ; do
            ls -dils "$fn" | awk '{print $11" "$8" "$9" "$10 }'
        done > "$outputfile"
    [[ -s "$outputfile" ]] && echo " file exists and is not empty " ||
            echo " file does not exist, or is empty "

Notes:

• I don't understand your second question: "Put command in variable if can", so I skip that one.
• You should never use the output of find (except with -print0 options) or ls as input of any command. But it is out of scope for this answer.