sed regex not working for hidden characters

Question

I have a list that has for some reason got some non alphanumeric characters appended to the end.

I would have thought it would be a simple case of running :

sed 's/[^A-Za-z0-9]$//'

To replace these trailing characters. Unfortunatley the above seems to be having zero effect.

For example, if I hex dump the file, I will see lines such as :

63 6f 6d be 0a 

Obviously, the culprit here is the hex character

be

But running the following pipe

.... | sed 's/[^A-Za-z0-9]//' | hexdump -C

I still see "be" appearing !

Answer 1

You can filter the characters you don't want with tr; in your case

... | tr -d -c '[:print:][:cntrl:]'

-d deletes any character matching the specified characters, -c complements the character sets (so it keeps only what matches in this case), [:print:] matches all printable characters including space, and [:cntrl:] matches control characters (such as carriage return and newline, which you probably want to keep).