A tee >( process ) is truncating its stdout when writing a file

Question

When I use tee to pipe stdout directly to a "specific block of code" (which then writes the modified data to a file), I always get the full complement of exptected output lines in the file.

However, when I tee the same data to a process >("same block of code"), it can drop up to 4 lines from the output. This number varies; sometimes it writes all the lines.

How can I get around this issue, and why is it only happening in the >( process ) ?

More info: I am 'tee'-ing to 2 process-substitutions and also to normal stdout pipe when this line dropping occurs.

using GNU bash, version 4.1.5(1)-release (i486-pc-linux-gnu) on Ubuntu 10.04.2 LTS.

Here is the actual script. The second process-substitution is the one in question, the one beginning with >(tr $'\x60' $X01

#  
# Run 'locate' and direct the output to a temp file
  errflag=""
  locitmct=0  # Count of located items 
  columns=4  # The number of columns in the main dialog
  colmnb=0  # Column number (NB: columns 1 and 2 are processed together) 
  X01=$'\x01'
  eval locate $zenargs |tee \
      >(zenity --progress --pulsate --auto-close) \
      >(tr $'\x60' $X01 \
         |sed -n "s/^\(.*\/\)\(.*\)/\1\2\n\2\n\1/p" \
            |while IFS= read -r line ; do \
              #
              #
              # process the data
              #
              #
            done > "$listf" )\
      >/dev/null
#
cat "$listf"
#

Answer 1

It has been 10 hours since I asked the question, and I've worked out a solution... Note: As I've previously mentioned in a comment (under the question), wait does not work with these processes. I assume that is because these 'process-substitutions' are not 'child' processes, which I believe is what wait waits for. (I tried wait with no args) ....

Comments on its features and failings would be appreciated. I don't quite understand how stdin is picked up by zenity and tr when echo is the first command, but I just thought I'd give it a go... It seems to work (in this case), but is this method safe?

There are quite likely to be some well tried and tested methods out there to tackle this, so other answers would be worthwile....

---

#!/bin/bash
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# Set up flag-files for processes to send PIDs to main process
# The first thing each process does is: echo -n "$BASHPID " > flag-file  
for i in {1..2};do cp /dev/null "$listf".pid$i;done
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
eval locate $zenargs |tee \
    >(echo -n "$BASHPID " > "$listf".pid1 ; \
      zenity --progress --pulsate --auto-close) \
    >(echo -n "$BASHPID " > "$listf".pid2 ; \
      tr $'\x60' $X01 \
       |sed -n "s/^\(.*\/\)\(.*\)/\1\2\n\2\n\1/p" \
          |while IFS= read -r line ; do \
              #
              #
              # process the data 
              #
              #
          done > "$listf" ) \
    >/dev/null
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
####  WAIT for processes to terminate  
pids=$(cat "$listf".pid{1,2})
while [[ "$pids" == *[0-9]* ]] ; do
   sleep .1 # GNU
   for pid in $pids ; do
      if ! kill -0 "$pid" 2>/dev/null; then
         pids="${pids/$pid/}"
      fi
   done 
done
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
cat "$listf"
#

Answer 2

Starting from 8.24 version of coreutils there is another way:

tee -p </dev/zero >(head -c10M | wc -c ) > >(head -c1 | wc -c ) | numfmt --to=iec

The output is

1

10M

The crucial points here are

1. Use -p which will cause tee to warn on errors writing to any output not a pipe. This means that EPIPE error on fwrite will not cause tee to exit and tee will run as long as there is at least one output file opened for writing. If you would like to get warning about EPIPE printed out to stderr use --output-error=warn instead -p.

2. Use > >(head -c1 | wc -c ) to redirect tee's stdout to another >(process substitution). This is important for two reasons. You need to take care of stdout output produced by tee. If you use >/dev/null for example tee would run as long as stdin has data (forever in this case). At the same time it will allow output of >(process substitution) to be send to stdout for the further processing. Other possibilities but not so elegant are to close tee's stdout output by other means. Two possibilities are >/dev/full and >&-. These will however issue a warnings (tee: standard output: No space left on device and tee: standard output: Bad file descriptor respectively)