Can't redirect cut output

Question

When I try to redirect the output of cut it always seems to be empty. If don't redirect it, the output shows in terminal as expected. This is true for OS X 10.10 and Linux 4.1.6.

This works:

root@karla:~# nc 10.0.2.56 30003
[...] lots of lines [...]

This works:

root@karla:~# nc 10.0.2.56 30003 | cat
[...] lots of lines [...]

This works:

root@karla:~# nc 10.0.2.56 30003 | cut -d, -f 15,16
[...] lots of lines [...]

This doesn't

root@karla:~# nc 10.0.2.56 30003 | cut -d, -f 15,16 | cat
[nothing]

This again DOES

root@karla:~# cat messung1 | cut -d, -f15,16 | cat
[...] lots of lines [...]

This is not limited to cat after cut. grep, tee and standard redirection using > don't work either.

What's wrong there?

Answer 1

It's not that much that there's no output as that it's coming in chunks.

Like many programs, when its output is no longer a terminal, cut buffers its output. That is, it only writes data when it has accumulated a buffer-full of it. Typically, something like 4 or 8 KiB though YMMV.

You can easily verify it by comparing:

(echo foo; sleep 1; echo bar) | cut -c2-

With:

(echo foo; sleep 1; echo bar) | cut -c2- | cat

In the first case, cut outputs oo\n and then ar\n one second later, while in the second case, cut outputs oo\nar\n after 1 seconds, that is when it sees the end of its input and flushes its output upon exit.

In your case, since stdin is nc, it would only see the end of its input when the connection is closed, so it would only start outputting anything after it has accumulated 4KiB worth of data to write.

To work around that, several approaches are possible.

• On GNU or FreeBSD systems, you can use the stdbuf utility that can tweak the buffering behaviour of some commands (it doesn't work for all as it uses a LD_PRELOAD hack to pre-configure the stdio buffering behaviour).

  ... | stdbuf -oL cut -d, -f15,16 | cat
  

  would tell cut to do a line-based buffering on its stdout.

• some commands like GNU grep have options to affect their buffering. (--line-buffered in the case of GNU grep).

• you can use a pseudo-tty wrapper to force the stdout of a command to be a terminal. However most of those solutions have some drawbacks and limitations. The unbuffer expect script for instance, often mentioned to address this kind of problem has a number of bugs for instance.

  One that doesn't work too bad is when using socat as:

  ... | socat -u 'exec:"cut -d, -f15,16",pty,raw' -
  

• You can replace your text utility with a higher-level text processing tool that has support for unbuffered output.

  • GNU awk for instance has a fflush() function to flush its output. So your cut -d, -f15,16 could be written:

    awk -F, -vOFS=, '{print $15,$16;fflush()}'
    

  • if your awk lacks the fflush() function, you can use system("") instead. That's normally the command to execute a command. Here, we would be executing an empty command, but actually using it for the fact that awk flushes its stdout before running the command.

  • or you can use perl:

    perl -F, -lane 'BEGIN{$,=",";$|=1} print @F[14..15]'