comm should tell you that one of the files isn’t sorted:
comm: file 1 is not in sorted order
It expects the files to be sorted using the current locale’s collation order (as determined by LC_COLLATE); it won’t accept numerical order.
To compare the files, you can pre-sort them (lexicographically as you point out):
comm <(sort file1) <(sort file2)
If you want the result to be sorted numerically, sort it again:
comm <(sort file1) <(sort file2) | sort -n
This produces
1
2
3
4
5
6
7
8
9
11
12
13
15
16
17
18
19
20
21
22
23
705
707
709
711
712
826
827
839
846
847
848
872
873
874
875
891