Terminal emulators are moderately complex pieces of software.
Most of them are written by a very small team, or often just a single person. Bugs can occur here and there and can go unnoticed for quite a while. Probably nobody ever did any security review on any of them to catch corner cases.
Knowing the C/C++ programming languages, in which many of these terminals are written, it's enough to have a buffer overflow for things to go uncontrolled afterwards. Maybe such a buffer overflow can be achieved by passing it an overlong escape sequence.
Example: A few years ago VTE (the engine behind GNOME Terminal and others) had a bug where a specially crafted escape sequence could cause its parser to enter an infinite loop, thereby locking the entire GNOME Terminal (all its tabs and windows) or whichever other terminal emulator application using this widget. Not the end of the world, but something to be aware of. That issue was fixed immediately after being discovered, but how do you know that there aren't any more in this or any other terminal emulator?
Then there's the story with intentional behaviors that scratch boundaries or by intent do not-so-safe things. Most terminals allow to change the title; what if you set it to something overlong, or invalid UTF-8, or lots of emojis and BiDi text and whatnot; can you be sure that the terminal emulator and the windowing system will handle it safely? Can you be sure that if the title is changed, or the bell is rang, or a desktop notification is displayed (if the emulator supports such an extension) thousands of times per second then it will be handled safely? How about the escape sequence which causes the terminal to be resized, and what if you pass it enormous values; will the terminal emulator handle it correctly and will your desktop or window manager do so as well?
The most interesting bit of the story that occurs to me is OSC 52 clipboard access. Many terminals implement it, meaning that an escape sequence can set your clipboard's contents to something harmful (e.g. a command that deletes your files or uploads them to a remote party – now if in turn you accidentally paste it to another terminal and execute it then you have problems). VTE, due to these security concerns, refuses to implement this feature, and guess what: this is probably the most often requested feature, people hate that we don't have it. People would rather have a slightly insecure terminal with this convenience feature than sacrifice convenience for security.
There's also the case where the terminal seemingly misbehaves. The escape sequences leave it with invisible text, hidden cursor. The output ends in the middle of an escape sequence so that the subsequent shell prompt is misinterpreted. A few query-response escape sequences populated the input buffer with synthesized keypresses (n.b. I assume you could even get to a deadlock if too much input is synthesized).
Another aspect you should take into account is: How trusted the said file is? Is it a local file, or something you download from a trusted site, or download from a not-so-trusted one? If it's a local file, who and how does it generate it; e.g. if it's a locally running service then why would it ever place escape sequences in that file? If it's a locally running service that logs the data it receives over the wire, does it sanitize its logs (e.g. removes/replaces raw ESC and similar control bytes) beforehand?
So, here you are. If your data is extremely valuable, if you are working inside a government or secret service or such (but then I guess you wouldn't be asking this question here) then you should be paranoid: you cannot know what known "misfeatures" or yet undiscovered bugs you might run into. Get used to viewing your files with cat -v, less, a graphical text viewer, or alike. If you are an average user with average files then the risk is extremely low, perhaps negligible.