I have a file looks like:
1
2 4 5 6 7 19
20
22
24 26 27
29 30 31 32 34 40 50 56 58
234 235 270 500
1234 1235 1236 1237
2300
considering that my real data file is huge. So I want to check what is the maximum number in this data file. Also I want to check on average how many columns exists within rows. as an example in this small example, the maximum number of column is 9 (in 5th row) and on average there is 3.33 column within rows. Any Suggestion please ?
$ awk 'NF > m { m = NF } { s += NF } END { printf("Max = %d\nAvg = %g\n", m, s/NR) }' data.in
Max = 9
Avg = 3.33333
The awk script will keep track of the maximum number of fields (columns) in m and the sum of the number of fields in s. When reaching the end of the input stream, it will output the stats collected.
The number of fields in the current record (line) is NF, and the number of records read so far is NR.
The following version will also keep track of the record with the most number of fields:
awk 'NF > m { m = NF; r = NR } { s += NF } END { printf("Max = %d (%d)\nAvg = %g\n", m, r, s/NR) }' data.in
Max = 9 (6)
Avg = 3.33333
You could use the "dc" utility for math work:
dc -e "
[zsmlksn]sb
[lk1+skzls+ss]sa
[[Max = ]nlmn[(]nlnn[)]n10an[Avg = ]n5klslk/1/n10an]sp
[lpxq]sq
[?z0=qlaxzlm<bcl?x]s?
0ddddsksmsnssd=?
"
Which has been shown below towards the working of the above
tr '\t-' ' _' data.in | # dc wants negative numbers to begin with underscores
dc -e "
[
z sm # store current num of cols in register "m"
lk sn # store row num in register "n"
]sb
[
lk 1 + sk # increment the number of rows
z ls + ss # add num of cols to running sum of cols
]sa
[
[Max=]n
lmn # put max number of cols on ToS & print it
[(]n
lnn # put row num at which max number of cols are present on ToS & print it
[)]n
10an
[Avg=]n
5k ls lk /1/n # accuracy of 5 digits, compute average = sum of cols / total num of cols
10an
]sp
[
lpx # print the results by invoking the macro "p"
q # quit
]sq
# while loop for reading in lines
[
? z 0 =q # quit when no columns found in the current line read in
lax # macro "a" does: rows++, sum+=cols
z lm <b # update max cols value stored in register "lm" when cols > lm
c # clear out the line and read in the next line
l?x # read the next line
]s?
# initializations+set the ball rolling:
# register "sk" -> line kount
# register "sm" -> max cols
# register "sn" -> row number corresp. to max cols
# register "ss" -> sum of cols
0
d d d d
sk sm sn ss
d=?
"
Max = 9(6)
Avg = 3.33333