In general, which characters in a regular expression need escaping?
For example, the following is not syntactically correct:
echo '[]' | grep '[]'
grep: Unmatched [ or [^
This, however, is syntatically correct:
echo '[]' | grep '\[]'
[]
Is there any documentation on which characters should be escaped in a regular expression, and which should not?
There are multiple types of regular expressions and the set of special characters depend on the particular type. Some of them are described below. In all the cases special characters are escaped by backslash \. E.g. to match [ you write \[ instead. Alternatively the characters (except ^) could be escaped by enclosing them between square brackets one by one like [[].
The characters which are special in some contexts like ^ special at the beginning of a (sub-)expression can be escaped in all contexts.
As others wrote: in shell if you do not enclose the expression between single quotes you have to additionally escape the special characters for the shell in the already escaped regex. Example: Instead of '\[' you can write \\[ (alternatively: "\[" or "\\[") in Bourne compatible shells like bash but this is another story.
grep, sed.[\*^$"$(printf '%s' "$string" | sed 's/[.[\*^$]/\\&/g')"grep -E, sed -E (old GNU versions: sed -r).[\(*^$+?{|"$(printf '%s' "$string" | sed 's/[.[\(*^$+?{|]/\\&/g')"
sed 's/regex/replace/' or in sed 's#regex#replace#, you would have to escape / or # characters, respectively. If you use it in awk '/regex/', you would have to escape / character in regex.
], }), but you quote ). Is this for a reason?
). I have removed it from the answer. Thank you for noticing. --- According to the current POSIX draft we have an official support of ERE in sed: sed -E. --- The escaping for ERE should also work for PCRE and other regexes (JavaScript, Python) but I have not tested it. GNU still marks grep -P as experimental / incomplete in some cases. (gnu.org/software/grep/manual/grep.html#grep-Programs)
sed -E for quite a while. Also, I removed ) from "Special in some contexts." If it's somehow special, please undo the change.
)]}, are special in some context in ERE, but there is normally no use in escaping them.
This depends on the application. In your example [ must be quoted as an argument for grep but not echo.
For the shell (from the POSIX specs):
Quoting is used to remove the special meaning of certain characters or words to the shell. Quoting can be used to preserve the literal meaning of the special characters in the next paragraph, prevent reserved words from being recognized as such, and prevent parameter expansion and command substitution within here-document processing (see Here-Document).
The application shall quote the following characters if they are to represent themselves:
| & ; < > ( ) $ ` \ " ' <space> <tab> <newline>and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of IEEE Std 1003.1-2001:
* ? [ # ˜ = %The various quoting mechanisms are the escape character, single-quotes, and double-quotes. The here-document represents another form of quoting; see Here-Document.
Specific programs (using regexes, perl, awk) could have additional requirements on escaping.
. or + which also require escaping if you want to use them literally?
Each application will have its own set of 'special' characters. The issue that you ran into was with grep not the shell. For which characters need to be quoted in grep, read the manpage's section on "REGULAR EXPRESSIONS".
For the shell, that characters that should be quoted are:
;'"`#$&*?[]<>{}\
and any whitespace.
Depending on the shell, other characters may need to be quoted as well:
!^%
Look under "SHELL GRAMMAR" on the shell's manpage.
bash included), ! is still expanded in double quotes, only single quotes will stop its expansion (or turning the shell option off).
] should not be quoted, [ not always. I didn't find any reference to { and }
grep uses BRE (Basic Regular Expressions) as its regex method.
A general rundown would be "escape any special character or metacharacter to get its literal, escape to create escape sequences (\n, \r, etc)", although this is not always true, for example, you have to escape ( and ) to get their special meaning (backreference).
The shell may transform the command line before the command execution. Both the shell and grep may use quoting to remove the special meaning of some characters. Nonetheless, grep and shells have different special characters. Moreover, unescaped special characters that did not result from an existing expansion are removed, before the command execution, by the shell.
echo '[]' | grep '[]'
The shell transmits the argument [] to grep and it is parsed as a malformed bracket expression by grep.
echo '[]' | grep \[]
Above, we can see a similar case. The backslash is removed and [] is transmitted as argument to grep. grep recognizes a malformed bracket expression.
echo '[]' | grep '\[]'
Finally, in this case, the quotes are removed by the shell and \[] is transmitted as argument to grep but, in this specific case ¹, \[ is interpreted by grep as a literal bracket. Quotes are needed to prevent the interpretation of the backslash as a special character by the shell.
Based on pabouk's answer, you can simplify the sed transformation in one variable expansion.
filePath="/my/file.ext"
filePathBreRegex="${filePath//[.\/\\*^\$\[]/\\&}" # the \/ is only for regexes delimited by /
filePathEreRegex="${filePath//[.\/\\*+?^\$|({\[]/\\&}"
It maybe is not better readable but neither is worse if you know bash.
The ${varName//pattern/substitution} expands varName by replacing matches of the glob-like pattern pattern with the string substitution where an unescaped & has a special meaning. It stands for the substituted pattern match itself.
Some of the special characters must be escaped inside the pattern's character class [] including [. It would be handy if there were an actual command to do this such as
function toBreRegex () { echo "${1//[.\/\\*^\$\[]/\\&}"; }
function toEreRegex () { echo "${1//[.\/\\*+?^\$|({\[]/\\&}"; }
To be safe with different types of regular expressions you could use square brackets to enclose almost every character to make them seen as regular characters by regex engines:
sed -e 's/[^^]/[&]/g' -e 's/\^/\\^/g'
Even then you could not use '[', '^' or '\' as separator with sed s (e.g. echo abc | sed 's['"$(echo abc | sed -e 's/[^^]/[&]/g' -e 's/\^/\\^/g')"'[.[g' don't work, but why would anyone use those characters anyway?)
An un-escaped [ starts a character class that expects to have at least one character.
So ] is treated as that character.
However the character class needs to end with a ] also; thus the "Unmatched [" message.
As you obviously did not intend to define a character class, you need to escape the [ as \[ ([[] and []] are both stupid, but valid character classes).
In general any special character (as described in man regex(7)) has to be escaped to loose the special property (like for \*).
However some non-special characters become special characters when escaped (like + is not special in sed until escaped).
Finally and unfortunately there's more than one regular expression syntax, so you have to study the corresponding manual pages to be sure.
