Shell script to compare

Question

I have a shell script that reads two files which is basically an "ls" output in the following format

File 1 (Server1.txt):
386030    8 -rw-r--r--   1 bfdeploy wasgroup     4809 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/createFamilyAssessment.htm
386044    8 -rw-r--r--   1 bfdeploy wasgroup     6041 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/disabilityBenefits.htm
386179    8 -rw-r--r--   1 bfdeploy wasgroup     6780 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/staffSummaryAlerts.htm

File 2 (Server2.txt):
386030    8 -rw-r--r--   1 bfdeploy wasgroup     4809 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/createFamilyAssessment.htm
386044    8 -rw-r--r--   1 bfdeploy wasgroup     6041 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/disabilityBenefits.htm
386179    8 -rw-r--r--   1 bfdeploy wasgroup     6780 Jul 15 15:50 /apps/ibm/httpserver/htdocs/wp/en_robohelp/staffSummaryAlerts.htm

Using basic awk statements, I am trying to compare the filepermission (column 3); size (column 7); and filename (column 11) as follows, but it is printing even lines that are similar

while read line
    do
            filename1=$(echo "$line"|awk '{print $11}')
            filesize1=$(echo "$line"|awk '{print $7}')
            filepermission1=$(echo "$line"|awk '{print $3}')
            lineinserver2=$(grep "$filename1" "$SERVER2.txt")
            if [ $? -eq 1 ]
            then
                    echo "$filename1 is in $SERVER1 $COMPDIR but not present in $SERVER2 $COMPDIR" >> $DIFFSUMMARYFILE
            else
                    filesize2=$(echo "$lineinserver2"|awk '{print $7}')
            #       echo $lineinserver2
            #       echo $filesize2
                    filepermission2=$(echo "$lineinserver2"|awk '{print $3}')
            #       echo $filepermission2
                    if [ $filesize1 != $filesize2 ]
                    then
                            echo "$filename1 on $SERVER1 has a size of $filesize1 and on $SERVER2 has a size of $filesize2" >> $DIFFSUMMARYFILE
                    fi
                    if [ "$filepermission1" != "$filepermission2" ]
                    then
                            echo "$filename1 on $SERVER1 has permission of $filepermission1 and on $SERVER2 has permission of $filepermission2" >> $DIFFSUMMARYFILE
                    fi

            fi
    done < "$SERVER1.txt"

Based on Janos, comment I have updated the script as follows

 while read filepermission1 fileseize1 filename1; do
            read filepermission2 filesize2 filename2 < <(grep "$filename1" "$SERVER2.txt" | awk '{print $3, $7, $11}')
            if [ $? -eq 1 ]; then
                    echo "$filename1 is in $SERVER1 $COMPDIR but not present in $SERVER2 $COMPDIR" >> $DIFFSUMMARYFILE
            else
                    if [ $filesize1 != $filesize2 ]; then
                    echo "$filename1 on $SERVER1 has a size of $filesize1 and $filename2 on $SERVER2 has a size of $filesize2" >> $DIFFSUMMARYFILE
                    fi
                    if [ "$filepermission1" != "$filepermission2" ]; then
                    echo "$filename1 on $SERVER1 has a permission of $filepermission1 and $filename2 on $SERVER2 has a permission of $filepermission2" >> $DIFFSUMMARYFILE
                    fi
            fi
    done < <(grep -xvf $SERVER2.txt $SERVER1.txt|awk '{print $3, $7, $11}')

In your sample, Server1.txt and Server2.txt are identical, so the script outputs nothing. I made modifications to trigger the summary lines and it worked correctly for all 3 cases: different permission or size or missing file. So what is the problem here? — janos
– janos, Commented Dec 27, 2013 at 21:12
@janos, All i changed was : if [ ! "$lineserver2" ] . Then i create two files called test.txt in the two separate servers with different sizes. The script said this file was present on SERVER1 but not SERVER2. Hence my confusion. — Fat Owl
– Fat Owl, Commented Dec 27, 2013 at 21:30
Show some actually sample input (maybe 6 to 10 lines) and expected output otherwise we're just guessing. All we can tell so far for sure is that the approach you are taking is 100% wrong. — Ed Morton
– Ed Morton, Commented Dec 28, 2013 at 1:44
@FatOwl I rolled back your recent edits removing the script and input files. Questions on SO are a community resource; removing this information makes the question useless to anyone running across it in the future. If you want to remove the information from your ls output, replace it with dummy data that interacts with your script in the same way. — Chris Hayes
– Chris Hayes, Commented Dec 28, 2013 at 3:52

janos · Accepted Answer · 2013-12-27 23:08:47Z

Your script seems to work for me as it is:

If the sample files are identical, it outputs nothing, correctly
If a filesize is different, it reports it correctly
If a permission is different, it reports it correctly
If a file is missing in the second file, it reports it correctly

To see more clearly, I rewrote it more efficiently, using modern bash constructs:

#!/bin/bash
SERVER1=server1
SERVER2=server2
while IFS=' ' read filepermission1 filesize1 filename1; do
    IFS=' ' read filepermission2 filesize2 filename2 < <(grep "$filename1" "$SERVER2.txt" | awk '{print $3, $7, $11}')
    if [ $? -eq 1 ]; then
        echo "$filename1 is in $SERVER1 $COMPDIR but not present in $SERVER2 $COMPDIR"
    else
        if [ $filesize1 != $filesize2 ]; then
            echo "$filename1 on $SERVER1 has a size of $filesize1 and on $SERVER2 has a size of $filesize2"
        fi
        if [ "$filepermission1" != "$filepermission2" ]; then
            echo "$filename1 on $SERVER1 has permission of $filepermission1 and on $SERVER2 has permission of $filepermission2"
        fi
    fi
done < <(grep -vxf $SERVER2.txt $SERVER1.txt | awk '{print $3, $7, $11}')

UPDATE

From your comment, it seems the columns are not where you expect them to be. To see clearer, run with just echo lines, like this:

#!/bin/bash
SERVER1=server1
SERVER2=server2
while read filepermission1 filesize1 filename1; do
    read filepermission2 filesize2 filename2 < <(grep "$filename1" "$SERVER2.txt" | awk '{print $3, $7, $11}')
    echo filepermission1=$filepermission1
    echo filesize1=$filesize1
    echo filename1=$filename1
    echo filepermission2=$filepermission2
    echo filesize2=$filesize2
    echo filename2=$filename2
done < <(grep -vxf $SERVER2.txt $SERVER1.txt | awk '{print $3, $7, $11}')

I hope that based on the output you should see clearer.

CONCLUSION

It seems the problem was due to IFS=... manipulation in the original script or in a calling script. As a workaround I inserted IFS=' ' in front of the read commands, where it was important. Ideally the calling scripts should be changed to use IFS='...' cmd, so that the IFS manipulation will only be effective for cmd only. Then the IFS=' ' in the above script can be safely removed, as the script should work fine with the default value.

Thank you very much. The requirement was to see only what is on server 1 but not on server 2 and not to check the other way around. Hence I did not do the second pass. I will make the above changes and let you know how it goes.
with the change you suggested, even though the file exists in server2, it is not finding it.
I don't believe your $? assignment statement is accurate. See gist.github.com/anonymous/8152563 for example.
@janos I rewrote the script to mimic exactly what you have but am getting the results with no values eg: on <server1> has a permission of -rw-r--r-- 52428800 /tmp/javasharedresources/C240D2A64P_webspherev70_wasgroup_G08 and on <server1>has a permission of -rwxr-xr-x 512 /tmp/IBM_LaunchPad_19802/callback.sh

Collectives™ on Stack Overflow

Shell script to compare

1 Answer 1

13 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

13 Comments

Related