Shell script if statement variable comparison issues [duplicate]

Question

I have the following shell script to append a header line to a batch of .csv files. I need to apply a different header txt file to each .csv based on the number of columns in the input csv. The script is working in that it is applying the header files and outputting the new csv, but every file is getting the 'else' header--including the csv's that evaluate to '91' with the csvgrep -n $f | wc -l command.

#!/bin/sh

dir="/Users/XXX/xlsxtocsv"

#move to the csvraw output directory
cd csvraw/

#append the header row needed for DB import
for f in *.csv; do 
    if [[ "csvgrep -n $f | wc -l" == 91 ]]; then
       cat $dir/svcheader3col.txt $f >> "$dir/csvhead/${f%}header.csv"
    else
       cat $dir/svcheader.txt $f >> "$dir/csvhead/${f%}header.csv"
    fi

done

And when I run the csvgrep command manually on the csv files I get:

username:csvraw username$ csvgrep -n file4col.csv | wc -l
      92
username:csvraw username$ csvgrep -n file3col.csv | wc -l
      91

Why is the "file3col.csv" file not evaluating to 91 in the if statement, but is when I run it manually?

It's comparing the string csvgrep -n $f | wc -l, not the output of the command csvgrep -n $f | wc -l. — Charles Duffy
– Charles Duffy, Commented Oct 2, 2018 at 16:36
BTW, consider using bash -x yourscript to run a script logging each command it invokes -- if you actually see what the test is doing in practice, that makes this kind of issue much easier to debug. — Charles Duffy
– Charles Duffy, Commented Oct 2, 2018 at 16:42

2 revs, 2 users 96% · Accepted Answer · 2018-10-02 17:06:40Z

1

You aren't actually running the csvgrep command. Use a command substitution such as $(...) to do so.

#!/bin/bash
: "${dir?dir must be defined}"
for f in *.csv; do 
    if [[ "$(csvgrep -n "$f" | wc -l | tr -d '[:space:]')" = 91 ]]; then
       cat -- "$dir/svcheader3col.txt" "$f" 
    else
       cat -- "$dir/svcheader.txt" "$f"
    fi >> "$dir/csvhead/${f%.csv}header.csv"
done

edited Oct 2, 2018 at 17:06

community wiki

2 revs, 2 users 96%
Charles Duffy

Sign up to request clarification or add additional context in comments.

8 Comments

jamierob Over a year ago

This is working much better and I understand what I was doing wrong. 'bash -x' outputs the following, which seems like it's not getting the 'wc -l' part correct. Is the whitespace in front of '92' and '91' messing things up as well? '+ for f in '.csv' '++ csvgrep -n file4col.csv '++ wc -l '+ [[ 92 == 91 ]] '+ cat -- /Users/xlsxtocsv/svcheader.txt file4col.csv '+ for f in '.csv' '++ csvgrep -n file3col.csv '++ wc -l '+ [[ 91 == 91 ]] '+ cat -- /Users/xlsxtocsv/svcheader.txt file3col.csv

Charles Duffy Over a year ago

Could you backtick-escape that? Formatted as it is there, the *s are being treated as formatting. (That said, a longer log is probably easier to post as a gist so multiline formatting sticks). That said, it looks to me like it's doing what it should from that log. (Note that the %.csv means this code is writing output to CNY04-3colheader.csv, not CNY04-3col.csvheader.csv as the original would have done).

Charles Duffy Over a year ago

What whitespace? [[ 92 == 91 ]] is exactly what it should look like. Whitespace is mandatory in test operations -- keep in mind that [ is a command, not shell syntax, and commands are separated from their arguments with whitespace. ([[ ]] is actually syntax, but still, the whitespace is mandatory). If there were whitespace in your data, then the bash -x logs would show it as something like [[ ' 91' == 92 ]].

Charles Duffy Over a year ago

(BTW, = rather than == in this answer was deliberate: == is a nonportable extension, whereas = as a string comparison operator works even in baseline POSIX test/[; see pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html)

jamierob Over a year ago

Here's the output form bash -x with the whitespace. I'm still unsure how to get '91 to evaluate correctly. gist.github.com/jamierob/d18f3d751893a416b6acae919cf9b3ea

|

Collectives™ on Stack Overflow

Shell script if statement variable comparison issues [duplicate]

1 Answer 1

8 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Linked

Related