Works on command line but not in shell script

Question

I have a file formated like below:

436541,000454056,Smith,john,jsmith,[email protected]

I want to extract out ",jsmith" and write it to another file "test.txt".

So I use:

grep -o '\,[a-z][a-z0-9]{1,7}' source.txt > test.txt

from the command line and it works fine.

when I use it from a shell script the test.txt file is empty

#!/bin/bash
grep -o '\,[a-z][a-z0-9]{1,7}' source.txt > test.txt

any suggestions?

Answer 1

By default, grep uses Basic Regular Expressions (BRE) which don't support {N}. If it works on the commandline, you most probably have grep aliased to grep -E or grep -P. You can check by running

alias | grep grep

Aliases are not enabled in scripts. As explained in man bash:

Aliases are not expanded when the shell is not interactive, unless  the
expand_aliases  shell option is set using shopt.

So, in a non-interactive shell, which is what you get when you run a script, aliases won't work. You have two options, either enable the aliases explicitly in your script and then source your ~/.bashrc file to get the alias definitions (there's no need to escape the ,, by the way):

#!/bin/bash
shopt -s expand_aliases
source ~/.bashrc
grep -o ',[a-z][a-z0-9]{1,7}' source.txt > test.txt

Or, far simpler, use grep -E in the script itself:

#!/bin/bash
grep -Eo ',[a-z][a-z0-9]{1,7}' source.txt > test.txt

You might also want to consider tools like awk that are designed to work on field-delimited data though. Chances are they will make your life much simpler.

Answer 2

If your text file has the same structure you could also use awk

awk -F"," '{print $5}' INPUTFILE.txt >> test.txt 

There is also cut as Costas mentioned in the comments

cut -d"," -f5 INPUTFILE.txt >> test.txt