read line by line and execute cat with grep

Question

Im building a script to load from a file content and then use to cat and grep in another file. The problem is that is not showing any values while executing the grep. If i do the command manually it will retrieve the data.

i already use the while but the issue is the same. the output6.txt is always empty. the echo seems to work, so it seems to be the grep or cat issue.

Sample from BlockspamCLIs.txt i have the values:

4412345 
441236 
2367890

Sample All_events.csv file

1,441236,20220909120909,test
2,441237,20220909120909,test
3,441232,20220909120909,test
4,44136,20220909120909,test
5,2367890,20220909120909,test

As an output im expecting to retrieve and store and record inside of the CSV file that contains the number for example:

1,441236,20220909120909,test
5,2367890,20220909120909,test

my script:

for i in `cat BlockspamCLIs.txt`
do
   grep $i *_All_events.csv >> output6.txt
   
   echo $i
done

thank you in advance.

Answer 1

First of all, please don't use a shell script for this sort of task, it is slow and error prone. In any case, grep can do what you want already:

grep -f BlockspamCLIs.txt *_All_events.csv >> output6.txt

From man grep:

-f FILE, --file=FILE

Obtain patterns from FILE, one per line. If this option is used multiple times or is combined with the -e (--regexp) option, search for all patterns given. The empty file contains zero patterns, and therefore matches nothing.

Next, if you do use a shell loop to iterate over the contents of a file, never do it with for i in $(cat file). And, you must always quote your variables to avoid various problems. For example, your command won't work if any of the lines in BlockspamCLIs.txt contain whitespace.

That said, if your lines don't have spaces or shell globbing characters, the command you show would work. It would be better written like this:

while IFS= read -r i; do
    grep -e "$i" -- *_All_events.csv >> output6.txt
    printf '%s\n' "$i"
done < BlockspamCLIs.txt

Answer 2

Using a tool specifically built for doing these sorts of things:

$ csvgrep -H -f BlockspamCLIs.txt -c 2 All_events.csv | tail -n +2
1,441236,20220909120909,test
5,2367890,20220909120909,test

The csvgrep utility is part of csvkit, and I use it here to extract all records from the header-less (-H) CSV file All_events.csv in which the second field (-c 2) matches any string in another file (-f BlockspamCLIs.txt).

Since the input file is a header-less CSV file, csvgrep will add a dummy header to the resulting CSV output, which we remove with tail (you could also use sed 1d or any other command that removes the first line and passes the rest on unmodified).

Since we use a CSV-aware tool, the input is not restricted to "simple" CSV data but would also work with input that contains fields with embedded commas, quotes and newlines.

This works even if the files are in DOS text format.