2

I am loading a data into ElasticSearch and later visualizing it in Kibana. However, the Kibana does not support nanosecond precision. As my file contains 900,000 lines, how do I iterate over it to accomplish following:

I want to preprocess the file in bash.

INPUT file1.csv:

2018-10-1711:54:59.4422378,OUTLOOK.EXE,12052,11316,Thread Profiling,Thread 11316,SUCCESS,User Time: 0.0000000; Kernel Time: 0.0000000; Context Switches: 3,company\username,0

EXPECTED OUTPUT file2.csv:

2018-10-1711:54:59.442,OUTLOOK.EXE,12052,11316,Thread Profiling,Thread 11316,SUCCESS,User Time: 0.0000000; Kernel Time: 0.0000000; Context Switches: 3,company\username,0

How to round up the date to 3 digits? I want to round up base on the 4th digit.eg. 4425000 = 442, 4426000 = 443

Improve this question
2
  • Command line only. I want to manipulate the file as it is one time load. Commented Nov 8, 2018 at 14:33
  • I want to round up base on the 4th digit.eg. 4425000 = 442, 4426000 = 443 Commented Nov 8, 2018 at 14:38

1 Answer 1

Reset to default
2 
awk -F, 'BEGIN { OFS=FS=","; } 
  {
    seconds=substr($1, index($1, ".")-2, 10); 
    ms=substr(seconds, 7); 
    seconds=substr(seconds, 1, 6); 
    if (ms > 5000) 
      seconds += 0.001; 
    $1=sprintf("%s%6.3f", substr($1, 1, index($1, ".") - 2), seconds);
    print
  }' < input

This simply brute-forces the timestamp fields out of the first parameter then checks to see whether the time should be rounded up or not. With the new time value in hand, it reassembles the timestamp field back into $1 and then prints the new line.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.