It's a combination of two things. First, brace expansion is not a pattern that matches file names: it's a purely textual substitution — see What is the difference between `a[bc]d` (square brackets) and `a{b,c}d` (braces)? . Second, when you use the result of a command substitution outside double quotes (ls $(…)), what happens is only pattern matching (and word splitting: the “split+glob” operator), not a complete re-parsing.

With ls $(echo 'test?.txt'), the command echo 'test?.txt outputs the string test?.txt (with a final newline). The command substitution results in the string test?.txt (without a final newline, because command susbtitution strips trailing newlines). This unquoted substitution undergoes word splitting, yielding a list consisting of the single string test?.txt since there are no whitespace characters (more precisely, no characters in $IFS) in it. Each element of this one-element list then undergoes conditional wildcard expansion, and since there is a wildcard character ? in the string the wildcard expansion does happen. Since the pattern test?.txt matches at least one file name, the list element test?.txt is replace by the list of file names that match the patterns, yielding the two-element list containing test1.txt and test2.txt. Finally ls is called with two arguments test1 and test2.

With ls $(echo 'test{1,2}'), the command echo 'test{1,2}' outputs the string test{1,2} (with a final newline). The command substitution results in the string test{1,2}. This unquoted substitution undergoes word splitting, yielding a list consisting of the single string test{1,2}. Each element of this one-element list then undergoes conditional wildcard expansion, which does nothing (the element is left as is) since there is no wildcard character in the string. Thus ls is called with the single argument test{1,2}.

For comparison, here's what happens with ls $(echo test{1,2}). The command echo test{1,2} outputs the string test1 test2 (with a final newline). The command substitution results in the string test1 test2 (without a final newline). This unquoted substitution undergoes word splitting, yielding two strings test1 and test2. Then, since neither of the strings contains a wildcard character, they're left alone, so ls is called with two arguments test1 and test2.

It's a combination of two things. First, brace expansion is not a pattern that matches file names: it's a purely textual substitution — see What is the difference between `a[bc]d` (square brackets) and `a{b,c}d` (braces)? . Second, when you use the result of a command substitution outside double quotes (ls $(…)), what happens is only pattern matching (and word splitting: the “split+glob” operator), not a complete re-parsing.

With ls $(echo 'test?.txt'), the command echo 'test?.txt' outputs the string test?.txt (with a final newline). The command substitution results in the string test?.txt (without a final newline, because command substitution strips trailing newlines). This unquoted substitution undergoes word splitting, yielding a list consisting of the single string test?.txt since there are no whitespace characters (more precisely, no characters in $IFS) in it. Each element of this one-element list then undergoes conditional wildcard expansion, and since there is a wildcard character ? in the string the wildcard expansion does happen. Since the pattern test?.txt matches at least one file name, the list element test?.txt is replace by the list of file names that match the patterns, yielding the two-element list containing test1.txt and test2.txt. Finally ls is called with two arguments test1 and test2.

With ls $(echo 'test{1,2}'), the command echo 'test{1,2}' outputs the string test{1,2} (with a final newline). The command substitution results in the string test{1,2}. This unquoted substitution undergoes word splitting, yielding a list consisting of the single string test{1,2}. Each element of this one-element list then undergoes conditional wildcard expansion, which does nothing (the element is left as is) since there is no wildcard character in the string. Thus ls is called with the single argument test{1,2}.

For comparison, here's what happens with ls $(echo test{1,2}). The command echo test{1,2} outputs the string test1 test2 (with a final newline). The command substitution results in the string test1 test2 (without a final newline). This unquoted substitution undergoes word splitting, yielding two strings test1 and test2. Then, since neither of the strings contains a wildcard character, they're left alone, so ls is called with two arguments test1 and test2.