176

I have a fixed-width-field file which I'm trying to sort using the UNIX (Cygwin, in my case) sort utility.

The problem is there is a two-line header at the top of the file which is being sorted to the bottom of the file (as each header line begins with a colon).

Is there a way to tell sort either "pass the first two lines across unsorted" or to specify an ordering which sorts the colon lines to the top - the remaining lines are always start with a 6-digit numeric (which is actually the key I'm sorting on) if that helps.

Example:

:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
500123TSTMY_RADAR00
222334NOTALINEOUT01
477821USASHUTTLES21
325611LVEANOTHERS00

should sort to:

:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
222334NOTALINEOUT01
325611LVEANOTHERS00
477821USASHUTTLES21
500123TSTMY_RADAR00
Improve this question
2
  • For the record: the command line I'm using so far is "sort -t\\ -k1.1,1.6 <file>" [the data can contain spaces, but will never contain a backslash] Commented Jan 28, 2013 at 12:51
  • unix.stackexchange.com/questions/11856/… Commented Dec 4, 2019 at 4:42

16 Answers 16

Reset to default
171 
(head -n 2 <file> && tail -n +3 <file> | sort) > newfile

The parentheses create a subshell, wrapping up the stdout so you can pipe it or redirect it as if it had come from a single command.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Thanks; I'm accepting this answer as it seems most complete and concise (and I understand what it's doing!) - it should be "head -n 2", though :-)
Is there a way to have this version work on piped-in data? I tried with tee >(head -n $header_size) | tail -n +$header_size | sort, but head seems to run after the tail|sort pipe, so the header ends up printed in the end. Is this deterministic or a race condition?
You could probably piece together something where you use cat to redirect the stdin to a temporary file, then run the above command on that new file, but it's starting to get ugly enough that it's probably better to use one of the awk-based solutions given in the other responses.
@DamienPollet: See Dave's answer.
@DamienPollet: See freeseek's answer
|
137

If you don't mind using awk, you can take advantage of awk's built-in pipe abilities, e.g.

extract_data | awk 'NR<3{print $0;next}{print $0| "sort -r"}'

This prints the first two lines verbatim and pipes the rest through sort.

Note that this has the very specific advantage of being able to selectively sort parts of a piped input. all the other methods suggested will only sort plain files which can be read multiple times. This works on anything.

Improve this answer

5 Comments

Very nice, and it works with arbitrary pipes, not only files!
Beautiful, awk never stops surprising me. Also, you don't need the $0, print is enough.
@SamWatkins freeseek's answer is less ugly.
What's the -r option doing to sort? Is this supposed to be reverse sort?
I prefer this awk approach by @Dave as it works with arbitrary pipes as opposed to @BobS 's subshell approach which works with files only.
136

In simple cases, sed can do the job elegantly:

your_script | (sed -u 1q; sort)

or equivalently,

cat your_data | (sed -u 1q; sort)
cat your_data | { sed -u 1q; sort; }  # to avoid the subshell

The key is in the 1q -- print first line (header) and quit (leaving the rest of the input to sort).

For the example given, 2q will do the trick.

The -u switch (unbuffered) is required for those seds (notably, GNU's) that would otherwise read the input in chunks, thereby consuming data that you want to go through sort instead.

Improve this answer

9 Comments

IMO this is the simplest solution here and easiest to remember. It works with piped data with no special considerations or awkward quoting and escaping, and does not need to be used multiple times if you are sorting on multiple columns by a chain of piped sort commands with the -s flag. eg. bgzip -dc somefile.tsv.gz | (sed -u 2q; sort -k 3,3 -n | sort -k 2,2 -n -s | sort -k 1,1 -s) | bgzip -c > my_sorted_file.tsv.gz. Key though is the edit adding the -u flag which ought to have solved @RobGilliam's problem above.
Can you explain a bit how pipe and the paenthesis work?
I would use head -n 1 instead of sed -u 1q. This head command is POSIX and much more portable than dealing with sed's -u flag.
@dan On my system, head seems to be buffered like sed, so that doesn't work. (Ubuntu 20.04, head (GNU coreutils) 8.30)
@Gqqnbig The parentheses create a subshell. Speaking of that, you could actually use braces instead, which don't create a subshell, just a group of commands, though you'd need to add spaces plus a semicolon at the end: ... { sed -u 1q; sort; }. The pipe works exactly like a regular pipe, but the input goes into the group of commands instead of one command.
|
55

Here is a version that works on piped data:

(read -r; printf "%s\n" "$REPLY"; sort)

If your header has multiple lines:

(for i in $(seq $HEADER_ROWS); do read -r; printf "%s\n" "$REPLY"; done; sort)

This solution is from here

Improve this answer

6 Comments

nice. for the single header case I use extract_data | (read h; echo "$h"; sort) it's short enough to remember. your example covers more edge cases. :) This is the best answer. works on pipes. no awk.
Ok, I straced this and it seems that bash goes to special lengths to make this work. In general, if you coded this in C or another language it would not work because stdio would read more than just the first header line. If you run it on a seekable file, bash reads a larger chunk (128 bytes in my test), then lseeks back to after the end of the first line. If you run it on a pipe, bash reads one char at a time until it passes the end of the line.
Nice! If you just want to eat the header, it's even easier to remember: extract_data | (read; sort)
This one is almost perfect but you need to use "IFS= read" instead of "read" to keep leading and trailing spaces.
This should be the accepted answer in my opinion. Simple, concise and more flexible in that it also works on piped data.
|
5 
head -2 <your_file> && nawk 'NR>2' <your_file> | sort

example:

> cat temp
10
8
1
2
3
4
5
> head -2 temp && nawk 'NR>2' temp | sort -r
10
8
5
4
3
2
1
Improve this answer

Comments

4

You can use

tail -n +3 <file> | sort ...

tail will output the file contents from the 3rd line.

Improve this answer

3 Comments

But this loses the header, which is not desired.
Losing the header, but otherwise it’s the simplest and I’m most likely to remember. But I really want to print the first line (from a command’s output) then sort the rest of the output by column 6, and print that to the screen. I’ll ask that as a new question.
@OldUncleHo regarding "Losing the header, but otherwise..." - retaining the header is the only interesting thing about this question so any answer that doesn't retain the header isn't answering the question and that's why it appears to be simple, because it doesn't do the only interesting thing that the question requires of an answer.
3

So here's a bash function where arguments are exactly like sort. Supporting files and pipes.

function skip_header_sort() {
    if [[ $# -gt 0 ]] && [[ -f ${@: -1} ]]; then
        local file=${@: -1}
        set -- "${@:1:$(($#-1))}"
    fi
    awk -vsargs="$*" 'NR<2{print; next}{print | "sort "sargs}' $file
}

How it works. This line checks if there is at least one argument and if the last argument is a file.

    if [[ $# -gt 0 ]] && [[ -f ${@: -1} ]]; then

This saves the file to separate argument. Since we're about to erase the last argument.

        local file=${@: -1}

Here we remove the last argument. Since we don't want to pass it as a sort argument.

        set -- "${@:1:$(($#-1))}"

Finally, we do the awk part, passing the arguments (minus the last argument if it was the file) to sort in awk. This was orignally suggested by Dave, and modified to take sort arguments. We rely on the fact that $file will be empty if we're piping, thus ignored.

    awk -vsargs="$*" 'NR<2{print; next}{print | "sort "sargs}' $file

Example usage with a comma separated file.

$ cat /tmp/test
A,B,C
0,1,2
1,2,0
2,0,1

# SORT NUMERICALLY SECOND COLUMN
$ skip_header_sort -t, -nk2 /tmp/test
A,B,C
2,0,1
0,1,2
1,2,0

# SORT REVERSE NUMERICALLY THIRD COLUMN
$ cat /tmp/test | skip_header_sort -t, -nrk3
A,B,C
0,1,2
2,0,1
1,2,0
Improve this answer

Comments

3

It only takes 2 lines of code...

head -1 test.txt > a.tmp
tail -n+2 test.txt | sort -n >> a.tmp

For a numeric data, -n is required. For alpha sort, the -n is not required.

Example file:

$ cat test.txt
header
8
5
100
1
-1

Result:

$ cat a.tmp
header
-1
1
5
8
100
Improve this answer

1 Comment

Isn't this basically the same answer as the accepted answer? (Except BobS's approach puts the result on stdout, allowing you to send the result through other filters before being written to file, if necessary)
1

Here's a bash shell function derived from the other answers. It handles both files and pipes. First argument is the file name or '-' for stdin. Remaining arguments are passed to sort. A couple examples:

$ hsort myfile.txt
$ head -n 100 myfile.txt | hsort -
$ hsort myfile.txt -k 2,2 | head -n 20 | hsort - -r

The shell function:

hsort ()
{
   if [ "$1" == "-h" ]; then
       echo "Sort a file or standard input, treating the first line as a header.";
       echo "The first argument is the file or '-' for standard input. Additional";
       echo "arguments to sort follow the first argument, including other files.";
       echo "File syntax : $ hsort file [sort-options] [file...]";
       echo "STDIN syntax: $ hsort - [sort-options] [file...]";
       return 0;
   elif [ -f "$1" ]; then
       local file=$1;
       shift;
       (head -n 1 $file && tail -n +2 $file | sort $*);
   elif [ "$1" == "-" ]; then
       shift;
       (read -r; printf "%s\n" "$REPLY"; sort $*);
   else
       >&2 echo "Error. File not found: $1";
       >&2 echo "Use either 'hsort <file> [sort-options]' or 'hsort - [sort-options]'";
       return 1 ;
   fi
}
Improve this answer

Comments

1

Applying the Decorate-Sort-Undecorate idiom using any version of the mandatory POSIX tools awk, sort, and cut:

$ awk -v OFS='\t' '{print (NR>2), $0}' file | sort -k1 -k2 | cut -f2-
:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
222334NOTALINEOUT01
325611LVEANOTHERS00
477821USASHUTTLES21
500123TSTMY_RADAR00

Works just fine on incoming piped input too:

$ cat file | awk -v OFS='\t' '{print (NR>2), $0}' | sort -k1 -k2 | cut -f2-
:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
222334NOTALINEOUT01
325611LVEANOTHERS00
477821USASHUTTLES21
500123TSTMY_RADAR00
Improve this answer

Comments

0

Another simple variation on all the others, reading a file once

HEADER_LINES=2
(head -n $HEADER_LINES; sort) < data-file.dat
Improve this answer

3 Comments

Doesn't work in a pipe. Seems like head is buffered, so it reads in a block just to discard most of it, and sort never receives that data. Use Andrea's answer instead.
OP doesn’t mention anything about requiring pipe; use process substitution, if available < <(your-script)
Process substitution does the same thing for me (running Ubuntu 20.04). I know OP didn't mention it, so I didn't downvote, just wanted to comment for the sake of anyone else using a pipe.
0

This is the same as Ian Sherbin answer but my implementation is :-

cut -d'|' -f3,4,7 $arg1 | uniq > filetmp.tc
head -1 filetmp.tc > file.tc;
tail -n+2 filetmp.tc | sort -t"|" -k2,2 >> file.tc;
Improve this answer

Comments

0

Using grep

(plus cat, sort, rm)

I tested this on a MacBook Air (Darwin Kernel Version 20.6.0, etc.). The script uses two working files: f1, f2. The entire input file $1 is copied to f1. From f1 the header and body lines are drawn.

Header lines contain the character ":". We grep them, and store them in f2.

The body lines do not contain the character ":", as far as we know, so we grep the remaining lines (grep -v), sort them, and append the result to f2.

Finally we cat f2 to achieve a result which can be used in a pipe line, and then we remove f1 and f2.

##                             ## Executable file "script"
cat $1 > f1                    ## Or cp $1 f1
grep ":" f1 > f2               ## Header lines contain ":"
grep -v ":" f1 | sort >> f2    ## Body lines do not contain ":"
cat f2                         ## List result
rm f1 f2                       ## Clean up

testfile:

:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
500123TSTMY_RADAR00
222334NOTALINEOUT01
477821USASHUTTLES21
325611LVEANOTHERS00

Invoke the script as "./script testfile" (or "cat testfile | ./script > result" etc.).

:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
222334NOTALINEOUT01
325611LVEANOTHERS00
477821USASHUTTLES21
500123TSTMY_RADAR00

The working files are handy for debugging, but force a clean up. I thought one would do, maybe zero, and yes, that works if you invoke it as "./script testfile". It doesn't work in a pipe, e.g. "cat testfile | ./script". You cannot rewind the standard input. But the version with two working files functions in a pipeline.

Improve this answer

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
0

if you don't mind a non-portable solution, then

echo '
:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
500123TSTMY_RADAR00
222334NOTALINEOUT01
477821USASHUTTLES21
325611LVEANOTHERS00' | 

gnu-sort -g   # -g := --general-numeric-sort   

bsd-sort -g   # -g := --general-numeric-sort
              #       --sort=general-numeric (either)

:0:12345
:1:6:2:3:3:8:4:2
010005TSTDOG_FOOD01
222334NOTALINEOUT01
325611LVEANOTHERS00
477821USASHUTTLES21
500123TSTMY_RADAR00

this assumes the header is in this exact format, and the sorting order might be different for random beyond-ASCII unicode headers

The -g flag on bsd-sort also worked for this input, but I don't have much insight regarding whether they're fully interchangeable for this particular flag.

Improve this answer

4 Comments

It works and I can see how, but it's more by coincidence than by design.
@RobGilliam : how so ? This is by design, per OP's specs, which said ...or to specify an ordering which sorts the colon lines to the top.
I am the OP so ... Two issues are that it doesn't leave the header lines alone, it sorts them (at least on MacOS, today), and it's using a numeric ordering, e.g. 123456000ABCDEFGHZZ would appear after 987654XYZPQRSTUVXX. The first wouldn't have been a problem for me as the first line always started :0: and the second :1: - the bit in the question re: sorting colons to the top implies this. The second one might have been problematic - ten years and at least 3 jobs have passed since I was doing this and I can't remember whether the second field could start with a digit or not.
then you oughta provide more precise test cases. and is the -g flag failing for your sort ? Which macOS you're on ? I'm on 14.5 myself.
-1

With Python:

import sys
HEADER_ROWS=2

for _ in range(HEADER_ROWS):
    sys.stdout.write(next(sys.stdin))
for row in sorted(sys.stdin):
    sys.stdout.write(row)
Improve this answer

1 Comment

pre-supposes the system has Python installed (mine doesn't)
-7 
cat file_name.txt | sed 1d | sort

This will do what you want.

Improve this answer

1 Comment

1) This only removes the header line and sorts the rest, it doesn't sort everything below the header line leaving the header intact. 2) it removes the first line only, when the header is actually two lines (read the question). 3) Why do you use "cat file_name.txt | sed 1d" when "sed 1d < file_name.txt" or even just "sed 1d file_name.txt" has the same effect?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.