String manipulation in shell

Question

I was writing a shell script to generate a string in a particular format so that it can be used as a input in one of the XML I am working with.

Given a input file in the format <attribute field>,<data_type>,<size>

instanceid,varchar,256
sysdate,date
status,number
notes,varchar,4000
created_on,date

I want to store in a variable the "check sum" like md5( INSTANCEID || STATUS || NOTES). That is I want all the attribute fields except the field having date as it's type Or'd.

The script I had written is this

IFS=$'\n'
file=$(cat source.txt)
line_number=$(cat source.txt | wc -l)
checksum="md5( "
for line in $file
do
let line_number=line_number-1
data_field=$(echo $line | cut -f1 -d','| tr "a-z" "A-Z")
data_type=$(echo $line | cut -f2 -d',' | tr "a-z" "A-Z")
if [ $data_type != "DATE" ]  && [ $line_number -gt 0 ]
  then checksum+="$data_field || "
elif [ $data_type != "DATE" ] && [ $line_number -eq 0 ]
  then checksum+=" $data_field "
fi
done
checksum+=")"
echo $checksum

This script works fine with all the input scenarios except when the last line has a attribute with date as it's type.

In which case the variable has a value likemd5( INSTANCEID || STATUS || NOTES || )

I tried to check if the last line was a date using tail command, but this again would fail if the last few lines had it's type as date.

How can I do away with the || which appears in the end?

Answer 1

The quick answer is checksum="${checksum% || })" instead of checksum+=")". Just unconditionally add the || string in each step and then strip off the last unnecessary one at the very end (so the line_number computation is no longer needed).

A better way to do this is

awk -F, 'BEGIN { printf "md5( " } 
         toupper($2) != "DATE" { printf "%s%s", sep, toupper($1); sep = " || " }
         END { print ")" }' source.txt

Answer 2

• It’s remarkable how rarely cat is useful in a shell script.  $(cat source.txt | wc -l) is a classic useless use of cat; if you needed to count the lines in a file, $(wc -l < source.txt) is a much cleaner ways of doing it.
• But you don’t need to count the lines in source.txt.
• file=$(cat source.txt) is an ugly way to read a file;

  while read …
  do
      ︙
  done < filename

  is better.  read has the benefit that it can split lines into fields for you.
• It’s silly to run tr twice for each line of the file when you need only to run it once on the entire file.  In some situations,

  tr … < filename | while read …
  do
      ︙
  done

  works nicely.  But there’s a problem with this: the while loop runs in a subshell, so changes that you make to shell variables (e.g., checksum) won’t be visible after the loop ends.  Terdon shows one way to work around that problem; here’s another one:

  tr … < filename | { while read …
  do
          ︙
      commands that potentially change checksum.
          ︙
  done
      ︙
  commands that use $checksum.
      ︙
  }

• As you’ve discovered, identifying the last occurrence of something can be difficult.  It’s often easier to identify the first:

  checksum="md5("
  first=1
  tr "a-z" "A-Z" < source.txt | { while IFS=, read data_field data_type size
  do
      if [ "$data_type" != "DATE" ]
      then
          if [ "$first" ]
          then
              first=
          else
              checksum+=" || "
          fi
          checksum+="$data_field"
      fi
  done
  checksum+=")"
  echo "$checksum"
  }
  

  Note that you really don’t need to test if [ "$data_type" != "DATE" ] twice.
  Also note that you should always quote references to shell variables (e.g., "$data_type") unless you have a good reason not to and you’re sure you know what you’re doing.

• As a further optimization, you can eliminate the first variable and simply use checksum itself to identify your first iteration through the loop:

  checksum=
  tr "a-z" "A-Z" < source.txt | { while IFS=, read data_field data_type size
  do
      if [ "$data_type" != "DATE" ]
      then
          if [ "$checksum" != "" ]
          then
              checksum+=" || "
          fi
          checksum+="$data_field"
      fi
  done
  checksum="md5($checksum)"
  echo "$checksum"
  }
  

Answer 3

You don't need anything half as complicated as what you've written. You could just do:

#!/usr/bin/env bash

checksum="md5("
## Read each line into the fields array (read -a fields), with fields
## separated by commas (IFS=,)
while IFS=, read -a fields
do
    ## If the 2nd element of the array is not "DATE"
    if [ ${fields[1]} != "DATE" ]
    then
        ## Add this to $checksum
        checksum+="${fields[0]} || "
    fi
## The tr is making everything upper case and then feeds
## directly into the while loop.
done < <(tr "a-z" "A-Z" < "$1")
## Get rid of the last || and add the closing ")"
checksum="${checksum% || })"
printf "OUT is: %s\n" "$checksum"

You then run the script with your file as input:

$ foo.sh file
OUT is: md5(INSTANCEID || STATUS || NOTES)