sed with numerical quantifiers - how to?

Question

I'm trying to edit fasta headers of multiple files, in order to remove a forward slash and everything after it (as long as 'everything after it' is equal or less than 10 characters). Header lines are marked by a '>'.

for i in ./*.fa;do sed -r 's/(>.*)\/.\{,10\}\n/\1\n/' "$i"; done

I've also tried

for i in ./*.fa;do sed -r 's/(>.*)\/.{,10}\n/\1\n/' "$i"; done

but it doesn't seem to be any better. My hunch is that it's the {,10} quantifier which breaks things. I'm not sure though. Help would be much appreciated!

For example, if the following was in a file:

>header1_some_extra_data_here/1-1000
ATGCGGGTACCCCA
>code/header2_some_extra_data
AGGTCCCCGGGAAAAA

I'd like the following to be the output:

>header1_some_extra_data_here
ATGCGGGTACCCCA
>code/header2_some_extra_data
AGGTCCCCGGGAAAAA

Answer 1

Your sed substitutions will not work as expected because you'll never be able to match a newline in the input data. This is because sed reads your file line by line, i.e. with the newlines as delimiters, and the expression(s) are applied to the lines individually, without the delimiting newlines.

Instead, changing your code slightly:

for fasta in ./*.fa; do
    sed 's;^\(>.*\)/.\{0,10\}$;\1;' "$fasta"
done

The few changes I've done are:

1. Use ; as the delimiter for the s/// command instead of the default /. This allows us to not escape the / in the pattern. Almost any character may be used as the delimiter, but one should probably pick one that does not occur in the pattern or in the replacement text.
2. Use only the standard basic regular expression syntax. In your pattern, (...) is extended regular expression syntax and \{...\} is basic regular expression syntax. I settled on using the basic syntax for portability. This also means dropping the -r option which enables the extended syntax in GNU sed.
3. Anchor the pattern to the start and end of the line with ^ and $ respectively.
4. Don't try to insert a newline in the replacement bit.

An alternative and shorter sed expression would be

sed '/^>/s;/.\{0,10\}$;;'

This applies a substitution to all lines that start with the > character (/^>/ acts as the "address" for the subsequent s/// command). The substitution simply deletes the / and the bit after it to the end of the line if that bit is 10 characters or fewer long.

Answer 2

Here's a slightly different way:

1. Save the output to new files:

   for file in *fa; do 
        sed -E 's|^\s*(>.{10,}.*)/.*|\1|' "$file" > "$file.fixed"; 
    done
   

2. Edit the files in place:

   sed -i -E 's|^\s*(>.{10,}.*)/.*|\1|' *.fa 
   

The -E option enables extended regular expressions. This lets us use () for capturing and {} for repeats without needing to escape them. I also changed the delimiter to | for clarity, and added ^\s* (note that this might not be supported by your sed; if it isn't, you can use ^ * instead) since sometimes you can have whitespace before the > in fasta files.

Then, the trick is to match a > followed by 10 or more characters until a /, capture these characters with the parentheses so they become \1 and replace the entire line with only the matched portion.

Note that this will find the longest stretch of >10 characters until the last /. So, if there are multiple / on the same line, that will keep all but the last one. For example:

$ echo ">header1_some_extra_data_here/1-1000/foo/bar/baz" | 
    sed -E 's|^\s*(>.{10,}.*)/.*|\1|' 
>header1_some_extra_data_here/1-1000/foo/bar
    

To avoid that, and remove everything after the first / as long as you've already matched 10 characters, use:

sed -E 's|^\s*(>.{10}[^/]*)/.*|\1|'