1

I am using paste to merge three text files (which do not require sorting) into a single document with three columns.

paste a.txt b.txt c.txt

I'd like elements that the columns have in common to occupy the same row without sharing it with non-matching elements (which they currently do). By the same token, unique elements should have their own rows. The elements in each column should preserve their original order.

Here's a simple example.

Input

1 1 1
2 2 2
3 4 4
5 5 5
1 1 2
3 3 3

Desired Output

1 1 1
2 2 2
3    
  4 4
5 5 5
1 1
    2
3 3 3

Here's a more complicated example.

Input

000 000 000
002 002 001
006 006 006
008 008 007
009 009 009
011 012 010
013 013 013
015 015 014
016 016 016
018 019 017
020 020 020
021 021 022
024 024 024
026 025 025
028 026 026
118 028 027
119 118 118
032 119 117
036 032 032
037 033 033
039 034 034
040 037 037
042 039 038
043 040 040
045 042 041
046 043 043
048 045 044
    046 046
    049 047

Desired Output

000 000 000
        001
002 002
006 006 006
        007
008 008 
009 009 009
        010
011        
    012 
013 013 013
        014
015 015 
016 016 016
        017
018     
    019 
020 020 020
021 021 
        022
024 024 024
    025 025
026 026 026
        027
028 028 
118 118 118
        117
119 119 
032 032 032
    033 033
    034 034
036     
037 037 037
        038
039 039 
040 040 040
        041
042 042 
043 043 043
        044
045 045 
046 046 046
        047
048     
    049

Ideally, I'd like to use tools that are built in to Linux/Unix. I'd also like the output to remain a single document with three columns, e.g., > whatever.csv.

The closest I've been able to get is to run sdiff on the original text files, but although that correctly aligns elements that the files share in common, it does not handle the differences as I would like.

Improve this question
2
  • You say that the original files do not require sorting. Would it be okay to sort them, individually? Commented Dec 5, 2018 at 18:11
  • It would be all right as long as each file/column retains its original order. Commented Dec 5, 2018 at 18:17

2 Answers 2

Reset to default
1 
BEGIN {
    # We assume the default input field separator (changeable with "-F")
    # Output will be tab delimited.
    OFS = "\t"
}
{
    # The number of output records that this input record results in.
    k=0

    # "seen" records which new record a field should be part of.
    # There may be NF new records for each input record if all
    # fields are unique.
    delete seen

    # "a" holds all data for the new output records.
    # It's basically a 2-dimensional NFxNF matrix
    # encodod in a 1-dimensional array.
    delete a

    # Iterate over the fields
    for (i=1; i<=NF; ++i) {
        if (!seen[$i]) {
            # This data has not been seen before (in this input record),
            # assign it to the next output line.

            seen[$i] = ++k
        }

        # Assign the input field to the right spot
        a[(seen[$i]-1)*NF + i] = $i
    }

    # Save NF as this is reset by emptying $0 later.
    nf = NF

    # Create and output new lines
    for (j = 1; j<=k; ++j) {
        $0 = ""

        # Create new output record
        for (i = 1; i<=nf; ++i)
            $i = a[(j-1)*nf + i]

        # Output record
        print
    }
}

Testing on the given data:

$ awk -f script.awk file
1       1       1
2       2       2
3
        4       4
5       5       5
1       1
                2
3       3       3

Testing on other data:

$ cat file
a b c e
1 2 1 1
2 1 1 1
1 1 1 2

$ awk -f script.awk file
a
        b
                c
                        e
1               1       1
        2
2
        1       1       1
1       1       1
                        2
Improve this answer
7
  • This works on my example and on most of my data; consequently, it is the answer. It does not work with some data, doubtless because I have made an assumption of some kind. When I work out what that is, it is probably better to ask a new question that to edit the original post. Commented Dec 5, 2018 at 19:49
  • @MilesO'Brien It's either the handling of the input data, or your interpretation of the output data. It would be interesting to see a lite of data that you believe it does not handle. Commented Dec 5, 2018 at 21:08
  • To be clear, your script does what I described in the OP on all data that I feed it. It's just that I erred in my assumptions about the data and how to handle it. Now, I'm unsure whether I should start a new thread or add to the OP. Commented Dec 5, 2018 at 22:18
  • I have updated the OP with a more complicated example. On reflection, I have decided that the way in which I worded the OP is compatible with both examples. Commented Dec 6, 2018 at 13:35
  • @MilesO'Brien Ah, so it's the order in which the new lines are added that is wrong, I get it. I will mull over this for a while and make a change later today (I'm a bit busy with other work ATM). Commented Dec 6, 2018 at 14:45
0

Here's a "brute force" solution in a shell script using paste and read.

#!/bin/sh

paste a.txt b.txt c.txt |
while read -r a b c; do
    if [ "$a" = "$b" ] && [ "$b" = "$c" ]; then
        printf '%s\t%s\t%s\n' "$a" "$b" "$c"
    elif [ "$a" = "$b" ]; then
        printf '%s\t%s\n\t\t%s\n' "$a" "$b" "$c"
    elif [ "$a" = "$c" ]; then
        printf '%s\t\t%s\n\t%s\n' "$a" "$c" "$b"
    elif [ "$b" = "$c" ]; then
        printf '%s\n\t%s\t%s\n' "$a" "$b" "$c"
    else
        printf '%s\n\t%s\n\t\t%s\n' "$a" "$b" "$c"
    fi
done

There's probably a more elegant solution but I couldn't come up with a good one off the top of my head.

You could probably use awk instead if you prefer it -- I think the result would look very similar. (One advantage of using awk would be that it could potentially do the job of paste at the same time, if that's useful to you.)

Improve this answer

You must log in to answer this question.