Based on what you've told us so far (I'm trying to ... leave just the title:) and the sample input you provided (## [Some title](#some-title)) this might be what you're trying to do, using any awk:
$ awk -F'[][]' '{print $2}' file
Some title
or any sed:
$ sed 's/.*\[\([^]]*\)].*/\1/' file
Some title
but without more truly representative sample input and expected output that's just a guess.
As for what's wrong with your sed script:
sed 's/^\(\#*\) *\[\([^\]]*\)\].*/\1 \2/'
In the first regexp segment \(\#*\):
- You're escaping the literal char
# as \# which is undefined behavior per POSIX when you wanted just #.
- You're using
#* which matches zero or more #s when you wanted 1 or more which is ##* or #\{1,\} in a BRE as sed uses by default (or #+ if you were using an ERE).
In the separating spaces part *:
- You're using
<blank>* which matches zero or more #s when you wanted 1 or more which is <blank><blank>* (oror <blank>+<blank>\{1,\}` in a BRE (or +``` if you were using an ERE).
In the last regexp segment \[\([^\]]*\)\].*:
- You're using
[^\]] and so escaping ] which is undefined behavior per POSIX when you wanted just [^]].
- You're using
\] at the end which is undefined behavior per POSIX since there's no unescaped [ before it when you wanted just ].
If you fixed all of those issues you'd get:
$ sed 's/^\(##*\) *\[\([^]]*\)].*/\1 \2/' file
## Some title
or
$ sed 's/^\(#\{1,\}\) \{1,\}\[\([^]]*\)].*/\1 \2/' file
## Some title
and since you're using GNU sed which supports EREs you could write that as:
$ sed -E 's/^(#+) +\[([^]]*)].*/\1 \2/' file
## Some title
And then to leave just the title as you said you wanted just means removing the first capture group:
$ sed 's/^##* *\[\([^]]*\)].*/\1/' file
Some title
$ sed 's/^#\{1,\} \{1,\}\[\([^]]*\)].*/\1/' file
Some title
$ sed -E 's/^#+ +\[([^]]*)].*/\1/' file
Some title