awk cuts strings

Question

When I ran the below command it prints the whole string e.g. Note="Peptidase S59%2C nucleoporin"

awk '$3=="mRNA"'  Nitab-v4.5_gene_models_Chr_Edwards2017.gff | head 
Nt01    maker   mRNA    143295  155540  .   +   .   ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|1|0.90|0.75|12|0|1011;Note="Peptidase S59%2C nucleoporin"
Nt01    maker   mRNA    170633  173860  .   +   .   ID=Nitab4.5_0006317g0020.1;Parent=Nitab4.5_0006317g0020;Name=Nitab4.5_0006317g0020.1;_AED=0.26;_eAED=0.26;_QI=15|0|0|0.83|0.6|0.33|6|0|424;Note="Putative S-adenosyl-L-methionine-dependent methyltransferase"
Nt01    maker   mRNA    156516  160996  .   -   .   ID=Nitab4.5_0006317g0030.1;Parent=Nitab4.5_0006317g0030;Name=Nitab4.5_0006317g0030.1;_AED=0.01;_eAED=0.01;_QI=161|1|1|1|0|0.5|2|358|141;Note="Unknown"
Nt01    maker   mRNA    78554   80638   .   -   .   ID=Nitab4.5_0006317g0040.1;Parent=Nitab4.5_0006317g0040;Name=Nitab4.5_0006317g0040.1;_AED=0.02;_eAED=0.02;_QI=0|0|0|1|1|1|3|0|187;Note="Heavy metal-associated domain%2C HMA"
Nt01    maker   mRNA    111288  129916  .   -   .   ID=Nitab4.5_0006317g0050.1;Parent=Nitab4.5_0006317g0050;Name=Nitab4.5_0006317g0050.1;_AED=0.24;_eAED=0.24;_QI=0|0|0|0.5|1|1|2|0|72;Note="Unknown"
Nt01    maker   mRNA    470560  474346  .   +   .   ID=Nitab4.5_0002367g0010.1;Parent=Nitab4.5_0002367g0010;Name=Nitab4.5_0002367g0010.1;_AED=0.11;_eAED=0.11;_QI=0|0|0|1|1|1|14|0|668;Note="Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation"
Nt01    maker   mRNA    499946  502182  .   +   .   ID=Nitab4.5_0002367g0020.1;Parent=Nitab4.5_0002367g0020;Name=Nitab4.5_0002367g0020.1;_AED=0.26;_eAED=0.26;_QI=0|0.5|0|0.66|0|0|3|0|258;Note="Cellulose synthase"
Nt01    maker   mRNA    496891  497596  .   +   .   ID=Nitab4.5_0002367g0030.1;Parent=Nitab4.5_0002367g0030;Name=Nitab4.5_0002367g0030.1;_AED=0.33;_eAED=0.33;_QI=0|0|0|0.5|0|0.5|2|0|213;Note="Cellulose synthase"
Nt01    maker   mRNA    505125  506853  .   -   .   ID=Nitab4.5_0002367g0040.1;Parent=Nitab4.5_0002367g0040;Name=Nitab4.5_0002367g0040.1;_AED=0.09;_eAED=0.09;_QI=0|0|0|1|0.5|0.66|3|0|230;Note="Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type"
Nt01    maker   mRNA    564383  570328  .   +   .   ID=Nitab4.5_0002367g0050.1;Parent=Nitab4.5_0002367g0050;Name=Nitab4.5_0002367g0050.1;_AED=0.08;_eAED=0.08;_QI=75|1|1|1|1|1|6|146|267;Note="SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12"

However, when I use this following command the string is shortened to e.g. Note="Peptidase

awk '$3=="mRNA"'  Nitab-v4.5_gene_models_Chr_Edwards2017.gff | awk '{print $9}' | head 
ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|1|0.90|0.75|12|0|1011;Note="Peptidase
ID=Nitab4.5_0006317g0020.1;Parent=Nitab4.5_0006317g0020;Name=Nitab4.5_0006317g0020.1;_AED=0.26;_eAED=0.26;_QI=15|0|0|0.83|0.6|0.33|6|0|424;Note="Putative
ID=Nitab4.5_0006317g0030.1;Parent=Nitab4.5_0006317g0030;Name=Nitab4.5_0006317g0030.1;_AED=0.01;_eAED=0.01;_QI=161|1|1|1|0|0.5|2|358|141;Note="Unknown"
ID=Nitab4.5_0006317g0040.1;Parent=Nitab4.5_0006317g0040;Name=Nitab4.5_0006317g0040.1;_AED=0.02;_eAED=0.02;_QI=0|0|0|1|1|1|3|0|187;Note="Heavy
ID=Nitab4.5_0006317g0050.1;Parent=Nitab4.5_0006317g0050;Name=Nitab4.5_0006317g0050.1;_AED=0.24;_eAED=0.24;_QI=0|0|0|0.5|1|1|2|0|72;Note="Unknown"
ID=Nitab4.5_0002367g0010.1;Parent=Nitab4.5_0002367g0010;Name=Nitab4.5_0002367g0010.1;_AED=0.11;_eAED=0.11;_QI=0|0|0|1|1|1|14|0|668;Note="Auxin
ID=Nitab4.5_0002367g0020.1;Parent=Nitab4.5_0002367g0020;Name=Nitab4.5_0002367g0020.1;_AED=0.26;_eAED=0.26;_QI=0|0.5|0|0.66|0|0|3|0|258;Note="Cellulose
ID=Nitab4.5_0002367g0030.1;Parent=Nitab4.5_0002367g0030;Name=Nitab4.5_0002367g0030.1;_AED=0.33;_eAED=0.33;_QI=0|0|0|0.5|0|0.5|2|0|213;Note="Cellulose
ID=Nitab4.5_0002367g0040.1;Parent=Nitab4.5_0002367g0040;Name=Nitab4.5_0002367g0040.1;_AED=0.09;_eAED=0.09;_QI=0|0|0|1|0.5|0.66|3|0|230;Note="Zinc
ID=Nitab4.5_0002367g0050.1;Parent=Nitab4.5_0002367g0050;Name=Nitab4.5_0002367g0050.1;_AED=0.08;_eAED=0.08;_QI=75|1|1|1|1|1|6|146|267;Note="SAC3/GANP/Nin1/mts3/eIF-3

As final results, I would like to retrieve Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin.

What did I miss?

Thank you in advance

Concerning your "final result", do you want to derive the first field from the "ID" or "Name" property? Are they always equal? — AdminBee
– AdminBee, Commented May 26, 2020 at 10:59
Can the quoted string at the end of a line contain a ;, e.g. could it contain a string like in ...258;Note="Cellulose synthase; the coolest synthase"? Can it contain a tab character (now that @terdon has provided the important detail that the fields are space-separated)? — Ed Morton
– Ed Morton, Commented May 26, 2020 at 11:56
@EdMorton tab-separated, not space. And no, the ; is an internal field delimiter and disallowed within values. See github.com/The-Sequence-Ontology/Specifications/blob/master/…. — terdon
– terdon ♦, Commented May 26, 2020 at 12:08

terdon · Accepted Answer · 2020-05-26 11:31:12Z

GFF is a tab-separated format but you are are not using tabs. Unless you use -F'\t' or BEGIN{FS="\t"}, awk will use any whitespace as the field delimiter, and that includes spaces. Since you are cutting on spaces, $9 ends at the first space. You also don't need two commands, what you want to do is:

$ awk -F'\t' '$3=="mRNA"{print $9}' file.gff 
ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|1|0.90|0.75|12|0|1011;Note="Peptidase S59%2C nucleoporin"
ID=Nitab4.5_0006317g0020.1;Parent=Nitab4.5_0006317g0020;Name=Nitab4.5_0006317g0020.1;_AED=0.26;_eAED=0.26;_QI=15|0|0|0.83|0.6|0.33|6|0|424;Note="Putative S-adenosyl-L-methionine-dependent methyltransferase"
ID=Nitab4.5_0006317g0030.1;Parent=Nitab4.5_0006317g0030;Name=Nitab4.5_0006317g0030.1;_AED=0.01;_eAED=0.01;_QI=161|1|1|1|0|0.5|2|358|141;Note="Unknown"
ID=Nitab4.5_0006317g0040.1;Parent=Nitab4.5_0006317g0040;Name=Nitab4.5_0006317g0040.1;_AED=0.02;_eAED=0.02;_QI=0|0|0|1|1|1|3|0|187;Note="Heavy metal-associated domain%2C HMA"
ID=Nitab4.5_0006317g0050.1;Parent=Nitab4.5_0006317g0050;Name=Nitab4.5_0006317g0050.1;_AED=0.24;_eAED=0.24;_QI=0|0|0|0.5|1|1|2|0|72;Note="Unknown"
ID=Nitab4.5_0002367g0010.1;Parent=Nitab4.5_0002367g0010;Name=Nitab4.5_0002367g0010.1;_AED=0.11;_eAED=0.11;_QI=0|0|0|1|1|1|14|0|668;Note="Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation"
ID=Nitab4.5_0002367g0020.1;Parent=Nitab4.5_0002367g0020;Name=Nitab4.5_0002367g0020.1;_AED=0.26;_eAED=0.26;_QI=0|0.5|0|0.66|0|0|3|0|258;Note="Cellulose synthase"
ID=Nitab4.5_0002367g0030.1;Parent=Nitab4.5_0002367g0030;Name=Nitab4.5_0002367g0030.1;_AED=0.33;_eAED=0.33;_QI=0|0|0|0.5|0|0.5|2|0|213;Note="Cellulose synthase"
ID=Nitab4.5_0002367g0040.1;Parent=Nitab4.5_0002367g0040;Name=Nitab4.5_0002367g0040.1;_AED=0.09;_eAED=0.09;_QI=0|0|0|1|0.5|0.66|3|0|230;Note="Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type"
ID=Nitab4.5_0002367g0050.1;Parent=Nitab4.5_0002367g0050;Name=Nitab4.5_0002367g0050.1;_AED=0.08;_eAED=0.08;_QI=75|1|1|1|1|1|6|146|267;Note="SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12"

If you only want to get the value of Note=, you can do:

$ awk -F"\t" '$3=="mRNA"{sub(/.*Note=/,"",$9); print $9}' file.gff 
"Peptidase S59%2C nucleoporin"
"Putative S-adenosyl-L-methionine-dependent methyltransferase"
"Unknown"
"Heavy metal-associated domain%2C HMA"
"Unknown"
"Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation"
"Cellulose synthase"
"Cellulose synthase"
"Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type"
"SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12"

And, to remove the quotes at the beginning and end of the Note, while keeping any that might be part of note itself:

$ awk -F"\t" '$3=="mRNA"{sub(/.*Note=/,"",$9); print $9}' file.gff | sed 's/^"//; s/"$//'
Peptidase S59%2C nucleoporin
Putative S-adenosyl-L-methionine-dependent methyltransferase
Unknown
Heavy metal-associated domain%2C HMA
Unknown
Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation
Cellulose synthase
Cellulose synthase
Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12

Finally, to get the value of both Note and ID, you could do:

$ awk -F"\t" '$3=="mRNA"{n=$9; sub(/.*Note="/,"",n); sub(/"$/,"",n); sub(/.*ID=/,"",$9); sub(/;.*/,"",$9); print $9","n}' file.gff 
Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin
Nitab4.5_0006317g0020.1,Putative S-adenosyl-L-methionine-dependent methyltransferase
Nitab4.5_0006317g0030.1,Unknown
Nitab4.5_0006317g0040.1,Heavy metal-associated domain%2C HMA
Nitab4.5_0006317g0050.1,Unknown
Nitab4.5_0002367g0010.1,Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation
Nitab4.5_0002367g0020.1,Cellulose synthase
Nitab4.5_0002367g0030.1,Cellulose synthase
Nitab4.5_0002367g0040.1,Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
Nitab4.5_0002367g0050.1,SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12

Personally, however, I would do this in perl instead:

$ perl -F'\t' -lane 'if($F[2] eq "mRNA"){/ID=([^\;]+).*Note="([^"]+)/; print "$1,$2"}' file.gff 
Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin
Nitab4.5_0006317g0020.1,Putative S-adenosyl-L-methionine-dependent methyltransferase
Nitab4.5_0006317g0030.1,Unknown
Nitab4.5_0006317g0040.1,Heavy metal-associated domain%2C HMA
Nitab4.5_0006317g0050.1,Unknown
Nitab4.5_0002367g0010.1,Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation
Nitab4.5_0002367g0020.1,Cellulose synthase
Nitab4.5_0002367g0030.1,Cellulose synthase
Nitab4.5_0002367g0040.1,Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
Nitab4.5_0002367g0050.1,SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome

Ed Morton · Accepted Answer · 2020-05-26 12:16:37Z

Assuming the quoted string at the end of each line can't contain a tab or a ;, this is probably all you need:

$ awk -F'\t' -v OFS=',' '$3=="mRNA"{ gsub(/;/,FS); gsub(/[^\t=]+=|"/,""); print $9, $15 }' file
Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin
Nitab4.5_0006317g0020.1,Putative S-adenosyl-L-methionine-dependent methyltransferase
Nitab4.5_0006317g0030.1,Unknown
Nitab4.5_0006317g0040.1,Heavy metal-associated domain%2C HMA
Nitab4.5_0006317g0050.1,Unknown
Nitab4.5_0002367g0010.1,Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimeriion
Nitab4.5_0002367g0020.1,Cellulose synthase
Nitab4.5_0002367g0030.1,Cellulose synthase
Nitab4.5_0002367g0040.1,Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
Nitab4.5_0002367g0050.1,SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12

or (original idea):

$ cat tst.awk
BEGIN { FS="\t"; OFS="," }
$3 == "mRNA" {
    split($NF,f,/;/)
    for (i in f) {
        gsub(/^[^=]+="?|"$/,"",f[i])
    }
    print f[1], f[7]
}

.

$ awk -f tst.awk file
Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin
Nitab4.5_0006317g0020.1,Putative S-adenosyl-L-methionine-dependent methyltransferase
Nitab4.5_0006317g0030.1,Unknown
Nitab4.5_0006317g0040.1,Heavy metal-associated domain%2C HMA
Nitab4.5_0006317g0050.1,Unknown
Nitab4.5_0002367g0010.1,Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation
Nitab4.5_0002367g0020.1,Cellulose synthase
Nitab4.5_0002367g0030.1,Cellulose synthase
Nitab4.5_0002367g0040.1,Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
Nitab4.5_0002367g0050.1,SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12

If it can contain a , then you should reconsider using , as the OFS and use tab or ; instead.

AdminBee · Accepted Answer · 2020-05-26 11:33:17Z

Assuming your "final result" output format is to be derived from the ID= and Note= field, the following should work:

$ awk -F'\t' '$3=="mRNA"{split($9,f,";"); split(f[1],i,"="); split(f[7],n,"\""); printf("%s,%s\n",i[2],n[2])}' file.gff
Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin
Nitab4.5_0006317g0020.1,Putative S-adenosyl-L-methionine-dependent methyltransferase
Nitab4.5_0006317g0030.1,Unknown
Nitab4.5_0006317g0040.1,Heavy metal-associated domain%2C HMA
Nitab4.5_0006317g0050.1,Unknown
Nitab4.5_0002367g0010.1,Auxin response factor%2C B3 DNA binding domain%2C DNA-binding pseudobarrel domain%2C AUX/IAA protein%2C Aux/IAA-ARF-dimerisation
Nitab4.5_0002367g0020.1,Cellulose synthase
Nitab4.5_0002367g0030.1,Cellulose synthase
Nitab4.5_0002367g0040.1,Zinc finger%2C RING-type%2C Zinc finger%2C RING/FYVE/PHD-type
Nitab4.5_0002367g0050.1,SAC3/GANP/Nin1/mts3/eIF-3 p25%2C 26S proteasome non-ATPase regulatory subunit Rpn12

This will split the 9th tab-separated field into ;-separated sub-fields stored in an array f, where the first sub-field (ID=...) is split at the = and the 7th sub-field (Note=" ... ") at the " to separate the interesting parts into auxiliary arrays i and n, respectively. The relevant portions of these are then printed.

This allows for rather compact code but is not necessarily the most efficient way of doing it.

Rakesh Sharma · Accepted Answer · 2020-05-27 01:56:17Z

1

Using GNU awk:

awk -F'\t' -vOFS=, 'NF>=9&&$3=="mRNA"{
N = split($9, a, /[=;]/)
for ( i=1; i<=N; i+=2 )
    h[a[i]] = a[i+1]
print h["ID"], gensub(/"(.*)"/, "\\1", "g", h["Note"])
}' bioinformatic.file

Nitab4.5_0006317g0010.1,Peptidase S59%2C nucleoporin

answered May 27, 2020 at 1:56

Rakesh Sharma

8561 gold badge5 silver badges3 bronze badges

Add a comment |

Stack Exchange Network

awk cuts strings

4 Answers 4

You must log in to answer this question.

Hot Network Questions

awk cuts strings

4 Answers 4

You must log in to answer this question.

Related

Hot Network Questions