Why is column adding a newline in the middle of my row where one is not present in the original data?

Question

I'm analyzing some packetfilter logs and wanted to make a nice table of some output, which normally works fine when I use column -t. I can't use a tab as my output field separator (OFS) in this case because it jacks up the multi-word string fields with the table view.

My original data consists of rows like this:

2018:01:24-09:31:21 asl ulogd[24090]: id="2103" severity="info" sys="SecureNet" sub="ips" name="SYN flood detected" action="SYN flood" fwrule="50018" initf="eth0" srcmac="12:34:56:78:90:ab" dstmac="cd:ef:01:23:45:67" srcip="192.168.1.123" dstip="151.101.65.69" proto="6" length="52" tos="0x00" prec="0x00" ttl="128" srcport="59761" dstport="80" tcpflags="SYN"

I'm getting the data into a comma-delimited (CSV) format using:

grep -EHr "192\.168\.1\.123" | 
cut -d':' -f2- | 
awk -F '"' 'BEGIN{
    OFS=","; 
    print "name","action","srcip","srcport","dstip","dstport","protocol","tcpflags"
}
{
    print $10,$12,$22,$36,$24,$38,$26,$(NF-1)
}'

This works fine and produces this kind of output (IP addresses all changed, I don't really have an internal host flooding this site):

name,action,srcip,srcport,dstip,dstport,protocol,tcpflags
SYN flood detected,SYN flood,192.168.1.123,59761,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,59764,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,59769,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,59771,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,59772,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,59890,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,60002,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,60005,151.101.65.69,80,6,SYN
SYN flood detected,SYN flood,192.168.1.123,60006,151.101.65.69,80,6,SYN

For some reason, whenever I use column to display the table output (-t), it adds a newline after the first column where no newline exists in the original data. For example:

$ cat mydata.csv | column -s ',' -t
name
                                action     srcip           srcport  dstip          dstport  protocol  tcpflags
SYN flood detected
                                SYN flood  192.168.1.123   59761    151.101.65.69  80       6         SYN
SYN flood detected
                                SYN flood  192.168.1.123   59764    151.101.65.69  80       6         SYN
SYN flood detected
                                SYN flood  192.168.1.123   59769    151.101.65.69  80       6         SYN

The expected output would be like follows:

name                 action     srcip           srcport  dstip          dstport  protocol  tcpflags
SYN flood detected   SYN flood  192.168.1.123   59761    151.101.65.69  80       6         SYN
SYN flood detected   SYN flood  192.168.1.123   59764    151.101.65.69  80       6         SYN
SYN flood detected   SYN flood  192.168.1.123   59769    151.101.65.69  80       6         SYN

Adding -x to column makes no difference either, nor does specifying the number of columns with -c (I have plenty of screen width in the terminal). Why is it doing that when there is no newline in the original data?

I really don’t think it is a character in my data because it is also happening with the header column which I created in my awk BEGIN block.

copying your example output and running cat mydata.csv | column -s ',' -t on it produces the expected output. centos 7 bash, fwiw. perhaps the output has a non-printing character in it that you aren't seeing. — WEBjuju
– WEBjuju, Commented Jan 24, 2018 at 23:56
@WEBjuju I thought that, too, but then it should definitely not be happening in my header field which I create in awk, since I created that myself and know it has no weird characters. This appears to be something else. — Dan
– Dan, Commented Jan 25, 2018 at 0:06
cool - can you attach the csv file to the question? i'll try again... — WEBjuju
– WEBjuju, Commented Jan 25, 2018 at 0:10
@WEBjuju not sure I can attach a file to a question, and it would take a lot of effort as this is not my actual data (i.e. I did find/replace on IP addresses with bogus data). — Dan
– Dan, Commented Jan 25, 2018 at 0:11
What I can do is post the original data so you can see if my awk is the problem — Dan
– Dan, Commented Jan 25, 2018 at 0:12

Kusalananda · Accepted Answer · 2018-01-25 16:58:26Z

I can reproduce your issue if I insert a row in the CSV file who's first comma-separated value is a very long string.

name                                                                                            
                   action     srcip          srcport  dstip          dstport  protocol  tcpflags
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  59761    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  59764    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  59769    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  59771    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  59772    151.101.65.69  80       6         SYN     
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxx  SYN flood  192.168.1.123  59890    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  60002    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  60005    151.101.65.69  80       6         SYN     
SYN flood detected                                                                              
                   SYN flood  192.168.1.123  60006    151.101.65.69  80       6         SYN

Note that there is no newline between the name and action columns in the actual output, but a line wrap (due to the line being so long) giving the illusion of a newline followed by indentation.

This means that you should look in your data for an entry with a very long name value.

This was the problem. I did not know this was the behavior of column - I wish there was a way to truncate a column when using -t. Thanks — Dan
– Dan, Commented Jan 25, 2018 at 17:49
But this makes sense as it is just padding the column based on the longest value, which in this case exceeds the screen width. Thanks! — Dan
– Dan, Commented Jan 25, 2018 at 18:47

Stack Exchange Network

Why is column adding a newline in the middle of my row where one is not present in the original data?

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Why is column adding a newline in the middle of my row where one is not present in the original data?

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions