Break up large log files

Question

I am trying to break a large log file into smaller files based on date.

The file is of the following form, where some lines may not have a date. Those lines should be included with the previous dated line.

2014-04-07T23:59:58 CheckForCallAction [ERROR] Exception caught
Undated line 1
Undated line 2
2014-04-08T00:00:03 MobileAppRequestFilter [DEBUG] Action
undated line 3
2015-04-08T00:00:03 MobileAppRequestFilter [DEBUG] ActionB

I found How to extract logs between two time stamps which is close to what I want, except my log file does not include a "[" at the start of the date, or "]" and the end of the date.

The command from that link is:

awk -F'[[]|[]]' \
  '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 }
   $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 }
   p { print $0 }' > test1.log  logwith[.log

I have been trying for several days to modify this, but I just can't seem to get it.

A desired enhancement would be to not have to specify a start and end date, but rather automatically name the output files by either year, or year-month.

Answer 1

Use T as the field delimiter and check for date-like strings explicitly. For example, to split by year:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){d=substr($1,1,4)}{print > d".log"}' logfile 

And by year+month:

awk -FT '($1~/^[0-9]+-[0-9]+-[0-9]+$/){split($1,d,"-")}{print > d[1]d[2]".log"}' logfile 

Here, we check that the first field (defined by T, so the whole date on lines starting with dates, that's what -FT means) is a set of 3 numbers separated by -. If it is, to get the year, we extract the first 4 characters (d=substr($1,1,4)) and, to get the month, we split the 1st field on -, saving the resulting strings in the array d (split($1,d,"-")), and use the 1st two elements of the array (d[1]d[2]) for the file name.