How to parse a data file to extract specific data and format for other use?

There may be many ways of doing this, but in awk we can do something like

awk 'gsub(/^[0-9.]+ \[ /,$1 " ") { a=2; while (a<=NR && $a != "]") { if ($a !~ /=/) {print $1 " " $a } ; a++ } }'

Let's break this down into something more readable:

gsub(/^[0-9.]+ \[ /,$1 " ") { .... }

This will match lines that begin with numbers and dots, followed by a space and a [. So it will match 10.20.30.40 [ but won't match junk23. It will replace it with the first field and then run the stuff inside the {...}.

So a line like

198.0.12.20 [ joker penguin character=villian ] blah  

will enter that section looking like

198.0.12.20 joker penguin character=villian ] blah  

Now the middle part, which might be easier to understand if we write it on multiple lines:

a=2;
while (a<=NR && $a != "]")
{
  if ($a !~ /=/) {print $1 " " $a }
  a++;
}

So we start at the 2nd field ("joker" in this case) and continue on until we run out of words or we see the ]. For each word we find, if it doesn't contain an = then we print the first field (the IP address) and the word.

The results:

198.0.12.20 joker
198.0.12.20 penguin
198.0.13.20 super_man
198.0.13.20 bat_man
198.0.14.20 lois_lane

(There's probably better ways of doing this!)