Replace strings in a big text file based on fields from a CSV file

Question

I have a big text file (roughly 2GB). I have a csv file that has the following fields:

rowID,pattern,other
1,abc_1z1,90
2,abc_1z2,90
3,abc_1z10,80
4,abc_3p1,77
...

My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
do 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv

Note that map.csv contains over 500000 rows.

Answer 1

Note that sed replaces regular expressions not strings - I'm going to assume that is not an issue for your use case, since your current solution uses it.

One way would be to pre-process map.csv into a sequence of expressions to pass to a single invocation of sed -f:

awk -F, 'NR>1 {printf "s/%s/%s/g\n", $2, $1}' map.csv | sed -f - bigfile

(add -i only once you are satisfied that it is doing the right thing).