Revisions to Break up large log files

replaced http://unix.stackexchange.com/ with https://unix.stackexchange.com/

Source Link

edited Apr 13, 2017 at 12:36

1

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

edited body

Source Link

edited Apr 10, 2016 at 13:22

terdon ♦

252.2k
69
480
718

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07 2307T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08 0008T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08 0008T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07 23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08 00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08 00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

deleted 154 characters in body; edited tags

Source Link

edited Apr 7, 2016 at 14:47

terdon ♦

252.2k
69
480
718

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07 23:59:58 CheckForCallAction [ERROR] Exception caught<br>caught
Undated line 1<br>1
Undated line 2<br>2
2014-04-08 00:00:03 MobileAppRequestFilter [DEBUG] Action<br>Action
undated line 3<br>3
2015-04-08 00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \<br>\
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }<br>
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }<br>
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I had to add html line breaks to this post make the text lines wrap properly. Hopefully that does not cause any difficulties.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07 23:59:58 CheckForCallAction [ERROR] Exception caught<br>
Undated line 1<br>
Undated line 2<br>
2014-04-08 00:00:03 MobileAppRequestFilter [DEBUG] Action<br>
undated line 3<br>
2015-04-08 00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \<br>
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }<br>
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }<br>
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

I had to add html line breaks to this post make the text lines wrap properly. Hopefully that does not cause any difficulties.

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07 23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08 00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08 00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.