1

how can I remove duplicate lines from the data set in a .txt file because the problem is that half content of my lines are repeated and I want only first RESULT line. I tried to use the command but this didn't work well 

sort myfile.txt uniq -u | newfile.txt

here is the content of my file and my lines of my file are 299873

ligand_06278/out.pdbqt:REMARK  Name = 22626427
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.821      8.000
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.4      3.333      6.628
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.4      4.526      7.557
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.5      2.500      4.835
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.5      2.516      7.135
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.6      2.660      7.148
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.8      3.141      6.023
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.9      0.000      0.000
ligand_06279/out.pdbqt:REMARK  Name = 22629712
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3     15.483     18.543
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3      1.944      5.962
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.3      8.946     12.260
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.5     14.453     17.240
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.8     10.330     14.145
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.8      1.727      5.848
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -7.1      7.429     11.509
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -7.3      0.000      0.000
ligand_06280/out.pdbqt:REMARK  Name = 22631372
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.1      0.000      0.000
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.3      5.006      9.020
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.4      2.195      8.687
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.4      2.712      9.301
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.6      2.186      8.354
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.7      5.168      7.981
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.8      1.961      2.580
ligand_06280/out.pdbqt:REMARK VINA RESULT:      -9.8      2.311      8.341
Improve this question
2
  • 1
    What is the expected output? Is it one single line or one line per "section"? Or something else? Commented Mar 28, 2018 at 9:50
  • And why put a so long sample input ? You can makes the same 70% shorter. Please, read MCVE Commented Mar 28, 2018 at 9:54

2 Answers 2

Reset to default
1 
$ awk -F: '$1 != p && /RESULT/ { print; p = $1 }' file
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264

This outputs the first RESULT line from each file mentioned in the input file. It does this by comparing the first column (the filename) to the first column of the previous line and testing whether the current line contains the word RESULT. When a RESULT line is found which has a different filename from the previous, the line is printed as is, and the value of p is updated.

I'm noticing that the input file looks very much like the result of grep having been run over a number of files, possibly grepping for REMARK.

To find all files and get the first line matching REMARK VINA RESULT from each:

find . -type f -path './ligand_*' -name 'out.pdbqt' -exec sed -n '/REMARK VINA RESULT/{p;q;}' {} ';'

or, as a simple loop:

for name in ligand_*/out.pdbqt; do
    grep -F 'REMARK VINA RESULT' "$name" | head -n 1
done

I've used different approaches here, you pick the one that feels most natural.

Improve this answer
9
  • You can use -m1 instead of head in the for loop. Commented Mar 28, 2018 at 10:18
  • @terdon Yes, with some implementations of grep. Commented Mar 28, 2018 at 10:19
  • Oh, I thought -m was POSIX. Sorry. Commented Mar 28, 2018 at 10:27
  • Thank you so much it is really helpful but I also want each "REMARK Name =" line before 'REMARK VINA RESULT' because REMARK Name is not showing in the output . how will it be processed ?? Commented Mar 29, 2018 at 4:51
  • @Bushra This contradicts the question where you say "I want only first RESULT line". Commented Mar 29, 2018 at 5:16
0

You could simply grep for Name= including the following line (-A1) and pass the output though grep RESULT to remove the Name= lines:

$ grep -A1 'Name =' file | grep RESULT
ligand_06278/out.pdbqt:REMARK VINA RESULT:      -8.3      2.094      2.612
ligand_06279/out.pdbqt:REMARK VINA RESULT:      -6.1      9.841     13.115
ligand_06280/out.pdbqt:REMARK VINA RESULT:     -10.0      3.811      7.264
Improve this answer
2
  • thank you for help but this output is not showing REMARK Name line. How will we get this one ?? Commented Mar 29, 2018 at 4:53
  • @Bushra when asking a question, it is important that you explain what you actually need. Kusalananda left you a comment yesterday asking you what your desired output would be, but you never answered so both he and I gave you answers that don't work for you. Anyway, just run grep -A1 'Name =' file and it should be what you want. If not, edit your question and clarify. Commented Mar 29, 2018 at 9:15

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.