Timeline for What should be the buffer size for the sort command?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 26, 2014 at 23:43 | answer | added | James K. Lowden | timeline score: 5 | |
| Jul 23, 2014 at 7:16 | comment | added | Stéphane Chazelas |
Try LC_ALL=C sort -rk2,2 (or with -f if you want case insensitive)
|
|
| Jul 23, 2014 at 0:49 | comment | added | iruvar | Yes I read the man page too. Notice I said "play around with" not "set it to as high as you can" | |
| Jul 23, 2014 at 0:30 | history | tweeted | twitter.com/#!/StackUnix/status/491742072228294659 | ||
| Jul 23, 2014 at 0:18 | comment | added | Sambit Tripathy | ‘--parallel=n’ Set the number of sorts run in parallel to n. By default, n is set to the number of available processors, but limited to 8, as there are diminishing performance gains after that. Note also that using n threads increases the memory usage by a factor of log n. source: GNU | |
| Jul 23, 2014 at 0:13 | comment | added | iruvar |
you may also want to play around with --parallel option seeing the number of processors at your disposal
|
|
| Jul 22, 2014 at 23:13 | comment | added | Sambit Tripathy | @1_CR Just realized that my command sorts the entire content starting from position 2 and that's why it is slower. I have reduced the buffer size to 200G and trying it. | |
| Jul 22, 2014 at 23:03 | comment | added | Sambit Tripathy | @polym the available RAM is 2 TB and file size is 150G, so decided to use 50% of the total, hoping this should be good enough. It got 32 CPUs @ 2.6 Ghz. | |
| Jul 22, 2014 at 23:00 | comment | added | Sambit Tripathy | @1_CR I am looking to sort on field 2 | |
| Jul 22, 2014 at 23:00 | comment | added | iruvar | Are you looking to sort by field 2 or sort by all fields starting 2? | |
| Jul 22, 2014 at 22:58 | review | First posts | |||
| Jul 22, 2014 at 23:00 | |||||
| Jul 22, 2014 at 22:55 | history | asked | Sambit Tripathy | CC BY-SA 3.0 |