I have a series of text files with a consistent format like:
FirstName: Mary
LastName: Smith
Address: 123 Anywhere St
City: Nowhere
State: TX
Zip: 77777
I need to extract several lines from these files and output them into a csv file in a format like:
<filename>,<FirstName>,<City>,<Zip>
I can get the fields I want with a simple grep but I don't know how to get the output the way I need it.
If you only have one record per file then this is a simple read loop.
#!/bin/bash
read_data()
{
local first last addr city state zip
file=$1
while read -r header data
do
case $header in
FirstName:) first=$data ;;
LastName:) last=$data ;;
Address:) addr=$data ;;
City:) city=$data ;;
State:) state=$data ;;
Zip:) zip=$data ;;
*) echo Ignoring bad line $header $data >&2
esac
done < $file
echo "$file,$first,$last,$addr,$city,$state,$zip"
}
for file in *srcfiles*
do
read_data $file
done
The read_data function reads each line and splits up the line into a "header" and a "data". Once we get to the end of the file we just print out the results.
We call that function once for each source file via the for loop.
Note some potential gotcha's: If there are commas in the data then this will break things, so you might want to do
echo "\"$file\",\"$first\",\"$last\",\"$addr\",\"$city\",\"$state\",\"$zip\""
as the output to enclose everything inside "..." layout. If there's any " in the data then this may cause the CSV to be malformed as well.
Adjust the echo line to match the format you want.
Quick and dirty approach, may suit your requirements.
grep . *|perl -ne 'if(/FirstName: (.*)/){$f=$1}if(/City: (.*)/){$c=$1}if(/^(.*):Zip: (.*)/){print "$1,$f,$c,$2\n"}'
Example:
grep . *
f1.txt:FirstName: Mary
f1.txt:LastName: Smith
f1.txt:Address: 123 Anywhere St
f1.txt:City: Nowhere
f1.txt:State: TX
f1.txt:Zip: 77777
f2.txt:FirstName: Joe
f2.txt:LastName: Bloggs
f2.txt:Address: 444 Anywhere St
f2.txt:City: Nowhere2
f2.txt:State: TXA
f2.txt:Zip: 77737
grep . *|perl -ne 'if(/FirstName: (.*)/){$f=$1}if(/City: (.*)/){$c=$1}if(/^(.*):Zip: (.*)/){print "$1,$f,$c,$2\n"}'
f1.txt,Mary,Nowhere,77777
f2.txt,Joe,Nowhere2,77737
If there's a single record per file and you have GNU awk, you could do
gawk -F': +' -vOFS=, '
BEGINFILE{delete rec}
{rec[$1] = $2}
ENDFILE{print FILENAME, rec["FirstName"], rec["City"], rec["Zip"]}
' file1.txt file2.txt ...
$2 mean that additional words will get lost? (eg "City: New York") would return "New".