get all rows having a column value greater than or equal to particular value

Question

I have to extract the values which are greater than or equal to 0.01 from column number 6 of tab-delimited file(My files contain more than 6 columns). I had tried with following code

for i in $(find ./ `pwd` -name "BC_4_*_*shift.txt" ); do
    awk -F"\t" 'NR==1 || $6>=0.01' $i > $i"_"ctdna_freq.txt;
done

to write this code I had taken help from get all rows having a column value greater than a threshold, Using this code I am able extract values from 6th column which are greater than 0.01 but I am not able to extract the values which are equal to 0.01 following is my input file

chr     pos         ref var p.val       freq.var
chr19   9074573     A   C   6.73E-22    0.586593469
chr19   9091288     G   T   5.96E-188   0.508732726
chr8    124518636   C   T   9.55E-21    0.00005
chr12   56490398    G   T   0.005271732 0.010003218
chr12   56477619    G   A   1.40E-15    0.010001069
chr12   56477619    G   A   1.40E-15    0.010001069
chr3    52677261    C   T   5.13E-06    0.01
chr5    67591010    A   G   4.82E-23    0.01

Expected output

chr     pos         ref var p.val       freq.var
chr19   9074573     A   C   6.73E-22    0.586593469
chr19   9091288     G   T   5.96E-188   0.508732726
chr12   56490398    G   T   0.005271732 0.010003218
chr12   56477619    G   A   1.40E-15    0.010001069
chr12   56477619    G   A   1.40E-15    0.010001069
chr3    52677261    C   T   5.13E-06    0.01
chr5    67591010    A   G   4.82E-23    0.01

Answer 1

#!/usr/bin/env bash
while IFS= read -r i; do
    awk -F'\t' 'NR==1 || $6>=0.01' "$i" > "${i}_ctdna_freq.txt"
done < <(find . -name 'BC_4_*_*shift.txt')

or:

#!/usr/bin/env bash
find . -name 'BC_4_*_*shift.txt' |
xargs -n 1 -I {} awk -F'\t' 'NR==1 || $6>=0.01' "{}" > "{}_ctdna_freq.txt"

Don't do for i in ..., see https://mywiki.wooledge.org/BashFAQ/001, and do always quote your variables, see https://mywiki.wooledge.org/Quotes. Run all your shell scripts through http://shellcheck.net until you're familiar with the fundamentals.

Answer 2

I put your data into a file known as data1.txt

Similarly, I made manual modifications and made many files.

This code does what you want in all. But it outputs to a single file.

find . -name "data*.txt" -type f -exec awk 'NR==1 || $6>=0.01' {} + >>output.txt

Answer 3

command

awk '$6 >= 0.01' file.txt

output

chr     pos         ref var p.val       freq.var
chr19   9074573     A   C   6.73E-22    0.586593469
chr19   9091288     G   T   5.96E-188   0.508732726
chr12   56490398    G   T   0.005271732 0.010003218
chr12   56477619    G   A   1.40E-15    0.010001069
chr12   56477619    G   A   1.40E-15    0.010001069
chr3    52677261    C   T   5.13E-06    0.01
chr5    67591010    A   G   4.82E-23    0.01

Python

#!/usr/bin/python

k=open('file.txt','r')
k.readline()
print ("chr     pos         ref var p.val       freq.var")
for i in k:
    q=i.split(' ')[-1]
    if (float(q) >= 0.01):
        print (i.strip())



output

chr     pos         ref var p.val       freq.var
chr19   9074573     A   C   6.73E-22    0.586593469
chr19   9091288     G   T   5.96E-188   0.508732726
chr12   56490398    G   T   0.005271732 0.010003218
chr12   56477619    G   A   1.40E-15    0.010001069
chr12   56477619    G   A   1.40E-15    0.010001069
chr3    52677261    C   T   5.13E-06    0.01
chr5    67591010    A   G   4.82E-23    0.01