sort (difference between default and numerical option -n)

Question

Per the man pages we have the following description of the --numeric-sort option of the sort command.

-n, --numeric-sort
              compare according to string numerical value

I assume, by string numeric value, we mean comparing each string character consecutively by its ASCII value?

The info pages read

‘-n’
‘--numeric-sort’
‘--sort=numeric’
     Sort numerically.  The number begins each line and consists of
     optional blanks, an optional ‘-’ sign, and zero or more digits
     possibly separated by thousands separators, optionally followed by
     a decimal-point character and zero or more digits.  An empty number
     is treated as ‘0’.  The ‘LC_NUMERIC’ locale specifies the
     decimal-point character and thousands separator.  By default a
     blank is a space or a tab, but the ‘LC_CTYPE’ locale can change
     this.

     Comparison is exact; there is no rounding error.

     Neither a leading ‘+’ nor exponential notation is recognized.  To
     compare such strings numerically, use the ‘--general-numeric-sort’
     (‘-g’) option.

After reading both docs, I still do not see explicitly explained which collation order is used for the -n option.

How does the --numeric-sort option differ from the default? My naive guess would be that numbers take precedence over letters, but I am not reading this in the documentation.

And which documentation states this explicitly, i.e. where could I have found this info by just looking up the documentation?

Answer 1

When you have multi-digit numbers, sort -n considers the entire number; by default the file

3
2
1
20
30

sorts like this:

1
2
20
3
30

which is probably not what you wanted. With -n, you get:

1
2
3
20
30

The numeric sort also deals with negative numbers, decimal points, and thousands separators (as determined by your locale). If there's trailing "non-number" text, it is ignored in the sorting order. If the line starts with something non-numeric, that line is counted as 0.

More exactly, the logic is like this: the (primary) sort key is an initial numeric string. (That is, "the number begins each line".) This string is defined to consist of possible blanks, a minus sign, zero or more digits, and possibly . and , (or whatever). Trailing letters don't factor in — they are not part of "the number". If the line does not start with numbers, that is treated as an invisible ("empty") number equal to 0. (Or, "a number with zero digits".)

So, having sorted on "the number" (compare using -k to give a sort key), if there are any remaining lines, those lines are sorted with the default sort. (That is, 1a before 1b — and 1a20 before 1a3.) The whole line is sorted in this way, not the line except the sort key, which gives some odd behavior in this case (0cookies sorts before biscuits — for the secondary sort, there's no "invisible 0" added).

In general, use -n when you actually want to sort lines (or fields) which consist of numbers. If you have a bunch of things that aren't numbers, or are numbers mixed in with other strings, you'll still get a consistent result, but it might not be what you want.

If you do have a mix of letters and numbers (and lines which contain both), you might prefer -V, which does a version sort according to special rules which divide the string into logical components — but be careful because this will put 1.10 higher than 1.9.

Answer 2

By default sort sorts character by character using a locale specified sort order. Generally that's pretty close to ASCII order, but there may be some regional variations. From the man page:

***  WARNING  ***  The  locale  specified  by the environment affects sort order.
Set LC_ALL=C to get the traditional sort order that uses native byte values.

Native byte value usually means by ASCII value, so digits come before uppercase letters which come before lowercase letters. But ordering is still character by character, so 10 comes before 2 because 1 comes before 2.

When the -n or --numeric-sort option is specified, runs of digits are treated as numbers (not individual characters), and sorted numerically from smallest number to largest.

The documentation is not entirely explicit on the details, so here are the rules of the -n flag derived experimentally:

1. Lines that begin numerically are sorted by numeric value (smaller numbers come first)
2. Trailing characters on numeric lines do not affect the numeric portion, but the trailing characters are sorted alphanumerically if the numeric portion is the same.
3. Lines that begin non-numerically are sorted as if they were zero, and then by rule 2.

Observe:

$ printf %s\\n 2z 111 10 20b 20a aa2 aa10 | sort -n
aa10
aa2
2z
10
20a
20b
111

By Rule 3, lines aa10 and aa2 are treated as zeros, and sorted by the remaining characters (including the digits, which are considered characters).

By Rule 2, lines 2z, 20a and 20b are treated as numbers and the trailing character only comes into effect when the numbers are the same.

And by Rule 1, all lines that begin with a number are sorted by numeric value.

Without the -n flag, sorting is done character by character where digit characters come before letter characters. Observe:

$ printf %s\\n 2z 111 10 20b 20a aa2 aa10 | sort
10
111
20a
20b
2z
aa10
aa2