TL;DR: $n \le C(f) \le n+1$.
We can easily prove that $C(f) \ge n$. Consider the set of $x \in \{(,[\}^n$. There are $2^n$ such $x$-values. Each matches a different set of $y$-values. So, you need to send at least $n$ bits to distinguish among these $x$-values.
Also, we trivially have $C(f) \le 2n$, as there are only $2^{2n}$ different values for $x$.
We can obtain a better upper bound on $C(f)$, as follows. Let the signature of $x$ be the shortest string $x'$ such that, for all $y$, $xy \in \text{Dyck}(2)$ iff $x'y \in \text{Dyck}(2)$. Then every signature $x$' is in $\{[,(\}^{\le n}$, i.e., it uses only ['s and ('s and has length at most $n$. Let $N$ count the number of strings in $\{[,(\}^{\le n}$, i.e.,
$$N = \sum_{i+j \le n} {i+j \choose i},$$
where the sum is over all $i,j \in \mathbb{N}$ such that $i+j \le n$. Here $i$ counts the number of ['s in $x'$, and $j$ counts the number of ('s in $x'$. Letting $k=i+j$, we can calculate
$$N = \sum_{k=0}^n \sum_{i=0}^k {k \choose i}
= \sum_{k=0}^n 2^k = 2^{n+1}-1.$$
Now since every signature is one of these strings, there are at most $N$ different signatures. A valid strategy is for Alice to send the signature of $x$ to Bob (or Bob to send the signature of $y$ to Alice), and that contains everything needed to compute the answer. Since there are at most $N$ different signatures, we have $C(f) \le \lg N = \lg (2^{n+1}-1)$, so $C(f) \le n+1$.
