Pipe (pass the output of previous command to the next command) with "|" when using "bsub"

Question

Hi I guess this is a very basic question:

Is there any specific syntax to do the "piping" when using "bsub"?

I am asking that because when I had an issue like this:

# try to run the same command successfully ran in another host
bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimmed_file.fastq &

but I met an error:

fastq_quality_trimmer: input file (-) has unknown file format (not FASTA or FASTQ), first character = J (74)

the second command seems not be able find out the output of the first command.

# try to run command without using "|"
bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq -o Trimmed_file.fastq
# seemed to work!

bsub fastq_quality_trimmer -Q33 -t 18 -l 20 -i Trimmed_file.fastq -o Trimme_file2.fastq &
# Also seemed to work!

# try to pipe again, did not work...
bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimme_file2.fastq &

# when call command without bsub, it seemed to work. 
fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimme_file2.fastq &
# so the issue seemed to be "bsub"

I would like to use bsub because my host is using gate node (the previous host has no gate node, so it does not matter whether I use "bsub" or not), and I do not want to create "traffic jam" by running command in the gate node. Any suggestion?

Answer 1

The bsub command will output the job submission ID and the name of the queue that it has been submitted to, if the submission is successful.

bsub will not provide you with the output of fastx_trimmer, which means that it can not be used for direct input to fastq_quality_trimmer via a pipe.

You will need to submit the whole pipeline to the queue to have it work properly, either by putting the commands into a shell script and submit that (ideally, if you plan on running it many times), or by quoting the whole pipeline properly in the call to bsub.

Answer 2

I received suggestions from my friend by email. Two solutions: 1. put the real command in "".

bsub "fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimmed_file.fastq”

2. write a bash script for bsub.

I tried solution 1. it worked very well. Put it here in case other people have similar issues.

Answer 3

I do not have the wherewithal to test using the specific programs mentioned in your example.

However, what you're asking for is possible with the -I (dash capital eye) option to bsub. The only catch is that the job submission message is sent to stdout, too, which will get mixed in with the expected actual output of your program:

Job <123456> is submitted to default queue <normal>.

That seems like a bug in LSF (should go to stderr). A workaround courtesy this SO answer is:

BSUB_QUIET=1 bsub -I cmd-that-writes-to-stdout | cmd-that-reads-from-stdin

Note that the actual value of the BSUB_QUIET environment variable apparently doesn't matter; it just needs to be defined in the environment.

For one-off commands, you can pipe the output of one bsub -I into another bsub -I, no problem:

$ export BSUB_QUIET=1

$ bsub -I grep ^root /etc/passwd
root:x:0:0:root:/root:/bin/bash

$ bsub -I grep ^root /etc/passwd | bsub -I awk -F: '{print \$6}'
/root

You would have to be careful to quote or escape shell metacharacters like $, though, if they appear somewhere in your command(s). This is another good reason to simply use a standalone script, as others suggested.

Hope that helps.