I recently had to investigate a field issue for our large enterprise application. I was horrified by the logs that I had to comb through in an attempt to find the problem and at the end of the day the logs did not help at all identifying/isolating the bug.

Note: I understand not all bugs are discoverable through logs. This does not change the fact that the logs are horrible.

There are some obvious problems with our logging that we can already attempt to fix. I do not want to list those here and I cannot simply show you our log files so you can give advice on what to do.

Instead, in order to assess how bad we are doing on the logging front, I would like to know:

1. What are some guidelines, if any, when it comes to logging for an application, especially large application.
2. Are there any patterns we should follow or anti-patterns we should be aware of?
3. Is this an important thing to fix or can it even be fixed or all log files are simply huge and you need supplemental scripts to analyze them?

Side note: we use log4j.