Command substitution and double quotes: why results are different?

Question

Here is an example where backticks and $() behave differently:

$ echo "$(echo \"test\")"
"test"
$ echo "`echo \"test\"`"
test

My understanding was this is because "backslashes (\) inside backticks are handled in a non-obvious manner"

But it seems like this is something else because when I remove outer double quotes the results became similar:

$ echo $(echo \"test\")
"test"
$ echo `echo \"test\"`
"test"

Could someone explain me how it works and why "`echo \"test\"`" removes double quotes?

Answer 1

You are right, it is something else in this case.

The solution is still in the same link, but the second point:

• Nested quoting inside $() is far more convenient.

  [...]

  `...` requires backslashes around the internal quotes in order to be portable.

Thus,

echo "`echo \"test\"`"

does not equal this:

echo "$(echo \"test\")"

but this:

echo "$(echo "test")"

---

You need to compare it instead with this:

echo "`echo \\"test\\"`"

Answer 2

The reason why the command substitution with backticks removes the quoting is caused by the way it is implemented:

• The text within the backticks is passed through the lexical parser a second time before it is finally executed.

  This happens because the results from passing the lexical parser a first time (that eats up one quoting level) is kept as a string and fed through the complete parser again during the final execution. So as a result backtick based command substitution removes two levels of quoting.

• The text within$(...) is kept unchanged before it is fed through the complete parser during the final execution.

  • this is implemented in ksh by recording the input text as a side efect of the parser while it is parsing for the closing ')̈́' that matches the $(.

  • this is implemented in bosh and mksh by recording the binary syntax tree output from the parser while it is parsing for the closing ')̈́' that matches the $( and reconstructing the related equivalent original input for that binary tree for the final execution.

  As a result, $(...) based command substitution removes only a single quoting level (as it happens for normal command execution as well).

Because of this implementation, backtick based command substitution causes unexpected behavior in special if grep patterns are part of such a command. This is why $(...) is recommended to be used in favor.

Answer 3

The reason is what happens to a (double) quoted string.

As soon as the shell parser finds an (unquoted) " it declares the whole string that ends on the next (unquoted) " as quoted. Inside quoted strings, most backslashes don't get removed. In UN-quoted strings backslashes are removed.

From POSIX (remove most backslashes):

A <backslash> that is not quoted shall preserve the literal value of the following character, …

This will show that in practice (set -x show what the shell executes after parsing):

$ ksh -c 'set -x; echo test\h\jtest; set +x'
+ echo testhjtest
testhjtest

unquoted backslashes get removed. But quoted ones don't :

$ ksh -c 'set -x; echo "test\h\jtest"; set +x'
+ echo 'test\h\jtest'
test\h\jtest

From POSIX Double quotes:

The <backslash> shall retain its special meaning as an escape character (see Escape Character (Backslash)) only when followed by one of the following characters

That's why backslashes get removed inside quoted strings for:

$   `   "   \   <newline>

Example:

$ ksh -c 'set -x; echo "test\"\h\j\"test"; set +x'
+ echo 'test"\h\j"test'
test"\h\j"test

That is what happens when the first character found is a (double) quote.

When what is found is an initial backquote (`) the parsing rules are different. First, the string starts unquoted (internal double quotes may start a quoted section) and there is only one quoting scope (until the closing (`). Note the single " on the first line below:

$ set -x; echo `echo \"\`echo \\"test\\" \`\"`; set +x
+++ echo '"test"'
++ echo '""test""'
+ echo '""test""'
""test""
+ set +x

A $(…), instead, starts a new quoting scope every time.

$ set -x; echo $(echo \"$(echo \\"test\\" )\"); set +x
+++ echo '\test\'
++ echo '"\test\"'
+ echo '"\test\"'
"\test\"
+ set +x

The result is completely different.