How to sort by odd lines then remove repeated values?

Question

I have the following type of file:

transcr_25793 +
YAL039C -
transcr_25793 +
YAL037C-B -
transcr_20649 +
YBL100C -
transcr_7135 +
YBL029C-A -
transcr_11317 +
YBL067C -
transcr_25793 +
YAL038W +
transcr_7135 +
YBL029W +

I was trying to get something like this:

transcr_7135 +
YBL029C-A -
transcr_7135 +
YBL029W +
transcr_11317 +
YBL067C -
transcr_20649 +
YBL100C -
transcr_25793 +
YAL039C -
transcr_25793 +
YAL037C-B -
transcr_25793 +
YAL038W +

Then, afterward, I was looking for something like this:

transcr_7135 +
YBL029C-A -
YBL029W +
transcr_11317 +
YBL067C -
transcr_20649 +
YBL100C -
transcr_25793 +
YAL039C -
YAL037C-B -
YAL038W +

I've scrolled through sort manual and some posts, but couldn't find anything that fit near this, just sort using numerical values to get odd lines...

Answer 1

Pure gawk solution:

awk -F_ 'NR%2{i=$2;next}{a[i]=a[i]"\n"$0}
         END{PROCINFO["sorted_in"]="@ind_num_asc";
             for(i in a) printf "%s","transcr_"i""a[i]"\n"}' file

The trick is to sort indexes of array a numerically with a little help of gawk's PROCINFO special array.

transcr_7135
YBL029C-A -
YBL029W +
transcr_11317
YBL067C -
transcr_20649
YBL100C -
transcr_25793
YAL039C -
YAL037C-B -
YAL038W +

BTW, its a pity awk doesn't offer an option to sort naturally a.k.a. version sort (according to text with numbers).

Answer 2

Not exactly the sorting order you've showed, but maby right as well?

$ cat input.txt|paste - -| sort -k1,1V -k2,2| tr "\t" "\n" | awk '{if($0 in line == 0) {line[$0]; print}}'
    transcr_7135 +
    YBL029C-A -
    YBL029W +
    transcr_11317 +
    YBL067C -
    transcr_20649 +
    YBL100C -
    transcr_25793 +
    YAL037C-B -
    YAL038W +
    YAL039C -

EDIT:

Insert the line number and uses it as a sorting key, should produce the exact output you like:

$ cat input.txt | paste - - | nl | sort -k2,2V -k1,1g | cut -f2- | tr "\t" "\n" | awk '{if($0 in line == 0) {line[$0]; print}}'

Answer 3

With GNU sort and assuming the lines don't contain TAB characters:

paste - - < file | sort -V | tr '\t' '\n' | awk '!seen[$0]++'

Or sort -t$'\t' -sk1,1V to preserve the original order for entries with identical odd lines like in your expected output.

If you don't have GNU sort, and assuming the odd lines always follow that pattern, you can replace sort -V with sort -k1.9n.

Answer 4

for element in $(sed -n 'p;n' a.txt |sort -nk 1.9 |uniq |awk '{print $1}')
do
    echo $element; cat a.txt |grep -A1 $i |grep -v trans |grep -v \\\\--
done

Where a.txt is your input. Tested:

[root@megatron ~]# cat a.txt
transcr_25793 +
YAL039C -
transcr_25793 +
YAL037C-B -
transcr_20649 +
YBL100C -
transcr_7135 +
YBL029C-A -
transcr_11317 +
YBL067C -
transcr_25793 +
YAL038W +
transcr_7135 +
YBL029W +
[root@megatron ~]# for i in $(sed -n 'p;n' a.txt |sort -nk 1.9 |uniq |awk '{print $1}')
do
echo $i; cat a.txt |grep -A1 $i |grep -v trans |grep -v \\\\--
done
transcr_7135
YBL029C-A -
YBL029W +
transcr_11317
YBL067C -
transcr_20649
YBL100C -
transcr_25793
YAL039C -
YAL037C-B -
YAL038W +
[root@megatron ~]#

Answer 5

Pre- and postprocessing with awk; this does not assume that a transcr line is followed by just one Y* line; it's also idempotent -- its output could be piped back as input and it will give the same result.

awk '{print $0~/^transcr/ ? t=$0 : t" "$0}' /tmp/foo | sort -t_ -k2n -k2 -u | awk '{print (NF > 2) ? $3" "$4 : $0}'
transcr_7135 +
YBL029C-A -
YBL029W +
transcr_11317 +
YBL067C -
transcr_20649 +
YBL100C -
transcr_25793 +
YAL037C-B -
YAL038W +
YAL039C -

Answer 6

Allow me to restate you problem. You really want to do two operations:

• Sort the odd-numbered (transcr*) lines by the number in the line, keeping the even-numbered (Y*) lines associated with the previous line
• Merge all of the Y* lines that fall under identical transcr* lines

While I admire the single-line solutions in the other answers, I think that this is complex enough that writing a script may be justified. Plus, once you store the script in a file, you can reuse it and enhance it as your needs change.

This script will take your input and produce lines identical to your final output:

#!/bin/env python

from collections import defaultdict
import fileinput
import re

def pair_lines (iterable):
    ''' Return lines two at a time '''
    args = [iterable] * 2;
    return zip(*args)

dd = defaultdict(list)
for line1,line2 in pair_lines(fileinput.input()):
    # accumulate lists of all second lines that share the same first
    # line value
    dd[line1].append(line2)

def line_index (line):
    ''' Return first substring that looks like an integer '''
    return int(re.search(r'\d+',line).group(0))

# sort by first lines and print results
for line1 in sorted(dd.keys(), key=line_index):
    print(line1, *dd[line1], end='', sep='')

Run with:

reorder.py input.txt > output.txt