How can you bind ^[ or M-[ shortcut to an action in nano?

You can't.

Terminals send characters, not keys. (See How do keyboard input and text output work? for more details.) But not all keys have a corresponding character. When you press a key or key chord that doesn't have a corresponding character, the terminal sends a sequence of characters that represents it, or in a few case a non-printable control character). These sequences always start with a particular character, which is called the escape character. This character is also what Ctrl+[ sends.

So if you could bind ^[ (Ctrl+[), that would break all keys that send escape sequences. For example, Up sends either the three characters (^[, [, A) or the three characters (^[, O, A), depending on the terminal. If you could rebind ^[ then the Up key would do the action of ^[ and then insert [ and A.

Alt+char sends the escape character followed by char. So if you rebound M-[, you'd really be rebinding the two-character sequence (^[, [), which would break some cursor and function keys.

Nano technically allows rebinding ^[ (as of version 2.5.3) but this has no effect because when it reads ^[, it classifies this as the start of an escape sequence (I'm simplifying a bit) and it never looks up a binding for ^[. Nano explicitly forbids rebinding M-[.

There are ways around this on some terminals, but only a few editors take advantage of them. Nano is a relatively simple editor, which primarily targets users who don't use terminals where such ways exist, and doesn't support this feature.