Trying to extract a substring and version number from a filename using bash

Question

I'm currently trying to extract a substring and version number from a filename using bash.

There are two formats the filenames will be in:

example-substring-1.1.0.tgz
example-substring-1.1.0-branch-name.tgz

For the first scenario I was able to extract the version number using sed like so:

echo example-substring-1.1.0.tgz | sed "s/.*-\(.*\)\.[a-zA-Z0-9]\{3\}$/\1/"

However this won't work for the second scenario.

Eventually I would like to create a script that will store the first substring and version in an associative array like below.

example_array["example-substring"]="1.1.0"
example_array["example-substring"]="1.1.0-branch-name"

This is proving tricky however as I can't seem to find a good way that will work for both scenarios. And for scenarios where the version includes the branch name I can't know before hand how many words the branch name will consist of.

I think variable expansion may be the way to go but wasn't able to get it to output what I want.

Answer 1

To be able to really test this we need sample input that contains more problematic cases, e.g. a string like -1.2.3 which looks like a version number appearing in the branch name:

$ cat file
example-substring-foo-1.1.0.tgz
example-substring-bar-1.1.0-branch-name.tgz
example-substring-rainy-1.1.0-branch-1.2.3.tgz

Normally I would do the pattern matching part in sed or awk, e.g. using any awk:

$ awk 'match($0,/-([0-9].*)\.[^.]+$/) {
    printf "\"%s\" \"%s\"\n", substr($0,1,RSTART-1), substr($0,RSTART+1)
}' file
"example-substring-foo" "1.1.0.tgz"
"example-substring-bar" "1.1.0-branch-name.tgz"
"example-substring-rainy" "1.1.0-branch-1.2.3.tgz"

rather than a shell loop but since you want to populate a shell array with the result anyway:

$ cat tst.sh
#!/usr/bin/env bash

declare -A example_array

while IFS= read -r ver; do
    if [[ $ver =~ -([0-9].*)\.[^.]+$ ]]; then
        example_array["${ver::-${#BASH_REMATCH[0]}}"]="${BASH_REMATCH[1]}"
    fi
done < "$@"

for idx in "${!example_array[@]}"; do
    printf 'example_array["%s"]="%s"\n' "$idx" "${example_array[$idx]}"
done

$ ./tst.sh file
example_array["example-substring-rainy"]="1.1.0-branch-1.2.3"
example_array["example-substring-bar"]="1.1.0-branch-name"
example_array["example-substring-foo"]="1.1.0"

Answer 2

With Perl

echo "example-substring-1.1.0-branch-name.tgz" |
    perl -wne'print join " ", /(.+)\-([0-9]+\.[0-9]+\.[0-9]+.*)\.tgz/'

Prints two words

example-substring 1.1.0-branch-name

This is thus its return to the shell script, from which this would be invoked I presume, and then one can form needed structures in the shell script.^† Tested also without the branch name, and with a few other variations of the input string.

Since the example-substring can contain digits as well (why not?), and so can the branch name (why not?), the regex pattern has no restrictions and both the leading and (possible) trailing parts are matched simply by .+ and .*.

But then we need something more specific for the version number and I've used an assumption that it always consists of three numbers separated by dots. I've also assumed the fixed rest of the string, the file extension .tgz. These can be relaxed somewhat if needed.

---

^† One can directly read a list (key value key value...) into an associative array

#!/bin/bash

eval declare -A ver=( $( 
    echo "example-substring-1.1.0-branch-name.tgz" | 
    perl -wnE'say join " ", /(.+)\-([0-9]+\.[0-9]+\.[0-9]+.*)\.tgz/' ))

echo ${ver["example-substring"]}

Or it may be more suitable to assign to variables first

str="example-substring-1.1.0-branch-name.tgz"

read -r str val <<< $( 
perl -wE'say join " ", $ARGV[0] =~ /(.+)\-([0-9]+\.[0-9]+\.[0-9]+.+)\.tgz/' 
    -- "$str" )

ver[$str]=$val

or even just using positional parameters

set -- $(
    perl -wE'say join " ", $ARGV[0] =~ /(.+)\-([0-9]+\.[0-9]+\.[0-9]+.+)\.tgz/' 
        -- "$str" )

ver[$1]=$2

There are of course other ways to pass arguments to a Perl script or a command-line program ("one-liner"), and other ways to take its output in bash.

Let me know if this Perl code needs commentary.

Answer 3

If you're willing to use grep instead of sed, then lookaheads and lookbehinds will allow you to define patterns to extract what you care about.

Consider the pattern: .+(?=-\d+\.\d+\.\d+) This will match anything that is followed by -<numbers>.<numbers>.<numbers>. ?= marks a conditional lookahead, which is an expression that must match the next characters, but is excluded from the final match of the pattern. When used with your examples:

$ echo example-substring-1.1.0.tgz | grep -Po '.+(?=-\d+\.\d+\.\d+)'
example-substring
$ echo example-substring-1.1.0-branch-name.tgz | grep -Po '.+(?=-\d+\.\d+\.\d+)'
example-substring

(The P flag enables PCRE2, and the o flag only prints the match)

Also consider the pattern: (?<=-)\d+\.\d+\.\d+.*(?=\.tgz$) It uses lookbehinds to assert that, immediately before the pattern, there is a -, and uses lookaheads to assert that the pattern ends with .tgz. When used with your examples:

echo 'example-substring-1.1.0.tgz' | grep -Po '(?<=-)\d+\.\d+\.\d+.*(?=\.tgz$)'
1.1.0
$ echo 'example-substring-1.1.0-branch-name.tgz' | grep -Po '(?<=-)\d+\.\d+\.\d+.*(?=\.tgz$)'
1.1.0-branch-name

Answer 4

This might work for you (GNU sed):

sed -E 's/^([^-]+-)+([0-9.]+).*\..*/\2/' file

Match filenames that have one or more words separated by -'s, followed by digits separated by .'s and then end an extension preceded by . and return the digits separated by .'s.

Answer 5

It may be possible to to do what you need just with Bash's built-in Pattern Matching. This Shellcheck-clean code demonstrates the idea:

#! /bin/bash -p

shopt -s extglob

files=( example-substring-1.1.0.tgz example-substring2-1.1.0-branch-name.tgz )

declare -A example_array

for f in "${files[@]}"; do
    base=${f%.*}    # remove suffix
    substring=${base%%-+([0-9]).*}
    example_array["$substring"]=${base#"$substring-"}
done

declare -p example_array

This outputs:

declare -A example_array=([example-substring2]="1.1.0-branch-name" [example-substring]="1.1.0" )

• shopt -s extglob enables "extended globbing" (including patterns like +([0-9])). See the extglob section in glob - Greg's Wiki.
• See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for an explanation of ${f%.*}, ${base%%-+([0-9]).*}, and ${base#"$substring-"}.
• In general, declare -p var prints the value of a variable in an unambiguous way. It avoids looping, and pitfalls, when printing the values of, both kinds of, arrays.