Looking at the help caller text, the first word of the returned text is the line number.
We can use the bash read command to split it into words:
read lineno rest <<< "$(caller)"
Using a regex is not so onerous
[[ $(caller) =~ ^([[:digit:]]+) ]] && lineno=${BASH_REMATCH[1]}