Deleting lines from large file if strings in list are found in the first 12 characters of line?

Question

I have a file matrix of +184000 lines * +5400 rows that looks like this

denovo1 someverylaaargenumbers and lotandlotsoftextuntil 5400.........
denovo10 someverylaaargenumbers and lotandlotsoftextuntil 5400........
denovo100 someverylaaargenumbers and lotandlotsoftextuntil 5400.......
denovo1000 someverylaaargenumbers and lotandlotsoftextuntil 5400......
denovo10000 someverylaaargenumbers and lotandlotsoftextuntil 5400.....
denovo100000 someverylaaargenumbers and lotandlotsoftextuntil 5400......
denovo184117 someverylaaargenumbers and lotandlotsoftextuntil 5400......

I have a list of identifiers in second file file that looks like this:

denovo1
denovo100
denovo1000
denovo100000

I wish to purge the lines in matrix 1 if the identifier is found in file 2. Thus:

denovo10 someverylaaargenumbers and lotandlotsoftextuntil 5400........
denovo10000 someverylaaargenumbers and lotandlotsoftextuntil 5400.....
denovo184117 someverylaaargenumbers and lotandlotsoftextuntil 5400......

I have this short unix code that reads line by line and finds the strings in file 2:

while read -r line
do
echo $line
sed -i '' '/$line/d' /my/path/matrix1
done < /my/path/file2

and it does work, but it takes forever because it reads all the lines to the end. Is there some way to make the machine read only the first 12 characters of each line?

Answer 1

With grep:

grep -vwf file matrix > matrix.new
mv matrix.new matrix

• option -f FILE use FILE as pattern input file
• option -w select only those lines containing matches that form whole words
• option -v select non-matching lines

Note that file must not contain any empty lines.

Or if you create your identifier file manually with an anchor ^ to match the start of the line and a space character after each identifier to mark the end of the pattern:

printf '^%s \n' denovo{1,100,1000,100000} > file
grep -vf file matrix > matrix.new
mv matrix.new matrix

Answer 2

Try:

$ awk 'FNR==NR{ids[$1]; next} !($1 in ids)' ids file
denovo10 someverylaaargenumbers and lotandlotsoftextuntil 5400........
denovo10000 someverylaaargenumbers and lotandlotsoftextuntil 5400.....
denovo184117 someverylaaargenumbers and lotandlotsoftextuntil 5400......

How it works:

• FNR==NR{ids[$1]; next}

  While reading the first file, ids, this creates a key in associative array ids with the id. It then skips the rest of the commands and jumps to the next line.

• !($1 in ids)

  While reading the second file, this prints the line if the first field is not a key in associative array ids.

To update the original file

When you are satisfied that the code is working correctly, the file can be changed:

awk 'FNR==NR{ids[$1]; next} !($1 in ids)' ids file >tmp && mv tmp file