diff only works on text files. To run diff on binary files, you first need to convert them to text, e.g. with xxd or hexdump. This is easy enough to do on-the-fly with process substitution.
e.g.
$ cat file1
A B C D E
$ cat file2
A B X D E
$ diff -u <(xxd file1) <(xxd file2)
--- /dev/fd/63 2022-03-06 15:40:23.811027810 +1100
+++ /dev/fd/62 2022-03-06 15:40:23.811027810 +1100
@@ -1 +1 @@
-00000000: 4120 4220 4320 4420 450a A B C D E.
+00000000: 4120 4220 5820 4420 450a A B X D E.
And, yes, file1 and file2 are text files, but text files are a subset of binary files that happen to only contain "text" characters. It was easier to create text files for this example.
Worth noting: Even tiny changes in a binary file (e.g. the addition or deletion of even one byte) can cause diff's output to be enormous. That's because every line of xxd or hexdump's output after that tiny change will be different. Hence doing this is not recommended. You could redirect diff's output to /dev/null and check the exit code, but if you only wanted to know if the files were different, it would be better to just run cmp instead.
Solution: Use a binary diff tool, like one of those shown below, programs whose purpose is to "diff" binary files, e.g.which are often used to generate binary patchespatch-files for binaries. For example:
$ apropos diff | grep binary
radiff2 (1) - unified binary diffing utility
bsdiff (1) - generate a patch between two binary files
xdelta3 (1) - VCDIFF (RFC 3284) binary diff tool
e.g.
$ radiff2 file1 file2
0x00000004 43 => 58 0x00000004
$
radiff2 also has a unidiff output option, which may be more readable (i don't know whether the output is any smaller than diffing xxd dumps of two large files, though):
$ radiff2 -u file1 file2
-0x00000004:43 "C D E\n"
+0x00000004:58 "X D E\n"
If the files are the same, radiff2 won't output anything:
$ radiff2 file1 file1
$