Shell script - blanks are not always recognized as such?

Question

I'm confused on how bash interprets blanks while executing a script. In the end, I want a script that is fed by a csv file containing software used by an organisation and it should return the CVEs for these programs, by calling a python script to fetch this information.

My file would look this, but with over 500 lines:

"CRYSTAL REPORTS 2008";"SAP";"reporting software";;"Mr. Smith"

And is fed to my script like

$ getcve.sh < test.csv

When I run this small script I get strange results concerning the wordcount (which I wanted to use to loop through the python script's output to store in another file):

Read from file
Source: CVE-2010-2590 CVE-2010-3032 
Words in variable: 2
CVE-2010-2590 CVE-2010-3032 
Words processed: 1

However, when I hardcode "SAP CRYSTAL REPORTS 2008" in the script, the count changes to what I would expect:

 Hardcoded query
 Query: "SAP CRYSTAL REPORTS 2008"
 Source: CVE-2010-2590 CVE-2010-3032
 Words in variable: 2
 CVE-2010-2590
 CVE-2010-3032
 Words processed: 2

The script itself looks like this:

#!/bin/bash
clear
echo "Hardcoded query"
query='"SAP CRYSTAL REPORTS 2008"'
echo "Query: "$query
var2=$(python3 $HOME/cve-search/bin/search_fulltext.py -q "$query"  | tr '\n' ' ')
echo "Source: "$var2
i=0
echo "Words in variable: "$(echo "$var2"|wc -w)
for cve in $var2
do
    echo $cve
    i=$[ $i+1 ]
done
echo "Words processed: "$i

echo
echo "Read from file"
IFS_OLD=$IFS
IFS=";"
while read title firm desc version manager
do
    query='"'$(echo $firm $title $version | tr -d '"')'"'
    var3=$(python3 $HOME/cve-search/bin/search_fulltext.py -q "$query" | tr '\n' ' ')
    echo "Source: "$var3
    i=0
    echo "Words in variable: "$(echo "$var3"|wc -w)
    for cve in $var3
    do
        echo $cve
        i=$[ $i+1 ]
    done
    echo "Words processed: "$i
done
IFS=$IFS_OLD

Is there a trick or method to get the same results as the hard coded query when reading from a file?

I stumbled onto this by toying a little bit around (shell scripting is new for me) and this weird result is bothering me ^^"

Thank you in advance for your help :)

Answer 1

Your problem comes from IFS=";" I think: this modification will have an impact on the for loop.

Try:

IFS_OLD=$IFS
IFS=";"
while read title firm desc version manager
do
    query='"'$(echo $firm $title $version | tr -d '"')'"'
    var3=$(python3 $HOME/cve-search/bin/search_fulltext.py -q "$query" | tr '\n' ' ')
    echo "Source: "$var3
    i=0
    echo "Words in variable: "$(echo "$var3"|wc -w)
    IFS=" "
    for cve in $var3
    do
        echo $cve
        i=$[ $i+1 ]
    done
    IFS=";"
    echo "Words processed: "$i
done
IFS=$IFS_OLD

Answer 2

If you place a string with spaces in a variable, then that will expand to one value when used like for i in $var3. However, when you use echo $var3 the output will contain the spaces and wc -w will count those.

You need to modify your loop to

for cve in $(echo $var3)

so that the value is re-evaluated and the spaces cause the value to be split up into discrete words.