How to know if a text file is a subset of another

Question

I am trying to find a way to determine if a text file is a subset of another..

For example:

foo
bar

is a subset of

foo
bar
pluto

While:

foo
pluto

and

foo
bar

are not a subset of each other...

Is there a way to do this with a command?

This check must be a cross check, and it has to return:

file1 subset of file2 :    True
file2 subset of file1 :    True
otherwise             :    False

Answer 1

If those file contents are called file1, file2 and file3 in order of appearance, then you can do it with the following one-liner:

 # python3 -c "x=open('file1', mode='rb').read(); y=open('file2', mode='rb').read(); print(x in y or y in x)"
 True
 # python3 -c "x=open('file2', mode='rb').read(); y=open('file1', mode='rb').read(); print(x in y or y in x)"
 True
 # python3 -c "x=open('file1', mode='rb').read(); y=open('file3', mode='rb').read(); print(x in y or y in x)"
 False

Answer 2

With perl:

if perl -0777 -e '$n = <>; $h = <>; exit(index($h,$n)<0)' needle.txt haystack.txt
then echo needle.txt is found in haystack.txt
fi

-0octal defines the record delimiter. When that octal number is greater than 0377 (the maximum byte value), that means there's no delimiter, it's equivalent to doing $/ = undef. In that case, <> returns the full content of a single file, that's the slurp mode.

Once we have the content of the files in two $h and $n variables, we can use index() to determine if one is found in the other.

That means however that the whole files are stored in memory which means that method won't work for very large files.

For mmappable files (usually includes regular files and most seekable files like block devices), that can be worked around by using mmap() on the files, like with the Sys::Mmap perl module:

if 
  perl -MSys::Mmap -le '
    open N, "<", $ARGV[0] || die "$ARGV[0]: $!";
    open H, "<", $ARGV[1] || die "$ARGV[1]: $!";
    mmap($n, 0, PROT_READ, MAP_SHARED, N);
    mmap($h, 0, PROT_READ, MAP_SHARED, H);
    exit (index($h, $n) < 0)' needle.txt haystack.txt
then
  echo needle.txt is found in haystack.txt
fi

Answer 3

From http://www.catonmat.net/blog/set-operations-in-unix-shell/:

Comm compares two sorted files line by line. It may be run in such a way that it outputs lines that appear only in the first specified file. If the first file is subset of the second, then all the lines in the 1st file also appear in the 2nd, so no output is produced:

$ comm -23 <(sort subset | uniq) <(sort set | uniq) | head -1
# comm returns no output if subset ⊆ set
# comm outputs something if subset ⊊ set

Answer 4

I found a solution thanks to this question

Basically I am testing two files a.txt and b.txt with this script:

#!/bin/bash

first_cmp=$(diff --unchanged-line-format= --old-line-format= --new-line-format='%L' "$1" "$2" | wc -l)
second_cmp=$(diff --unchanged-line-format= --old-line-format= --new-line-format='%L' "$2" "$1" | wc -l)

if [ "$first_cmp" -eq "0" -o "$second_cmp" -eq "0" ]
then
    echo "Subset"
    exit 0
else
    echo "Not subset"
    exit 1
fi

If one is subset of the other the script return 0 for True otherwise 1.

Answer 5

If f1 is a subset of f2 then f1 - f2 is an empty set. Building on that we can write an is_subset function and a function derived from it. As per Set difference between 2 text files

sort_files () {
  f1_sorted="$1.sorted"
  f2_sorted="$2.sorted"

  if [ ! -f "$f1_sorted" ]; then
    cat "$1" | sort | uniq > "$f1_sorted"
  fi

  if [ ! -f "$f2_sorted" ]; then
    cat "$2" | sort | uniq > "$f2_sorted"
  fi
}

remove_sorted_files () {
  f1_sorted="$1.sorted"
  f2_sorted="$2.sorted"
  rm -f "$f1_sorted"
  rm -f "$f2_sorted"
}

set_union () {
  sort_files "$1" "$2"
  cat "$1.sorted" "$2.sorted" | sort | uniq
  remove_sorted_files "$1" "$2"
}

set_diff () {
  sort_files "$1" "$2"
  cat "$1.sorted" "$2.sorted" "$2.sorted" | sort | uniq -u
  remove_sorted_files "$1" "$2"
}

rset_diff () {
  sort_files "$1" "$2"
  cat "$1.sorted" "$2.sorted" "$1.sorted" | sort | uniq -u
  remove_sorted_files "$1" "$2"
}

is_subset () {
  sort_files "$1" "$2"
  output=$(set_diff "$1" "$2")
  remove_sorted_files "$1" "$2"

  if [ -z "$output" ]; then
    return 0
  else
    return 1
  fi

}

Answer 6

I had to do this just now, and while searching for an answer, I thought of an approach using diff + grep in bash:

#!/bin/bash

subset()
{
    ! diff --ignore-blank-lines "$1" "$2" | grep '^<' > /dev/null
}

crosscheck()
{
    subset "$1" "$2" || subset "$2" "$1" 
}

echo -e 'foo\nbar'        > file1
echo -e 'foo\nbar\npluto' > file2
echo -e 'foo\npluto'      > file3

echo; echo '  file1'; cat file1
echo; echo '  file2'; cat file2
echo; echo '  file3'; cat file3

echo
crosscheck file1 file2 && echo file1 is a subset of file2, or file2 is a subset of file1, or they are the same
crosscheck file2 file3 && echo file2 is a subset of file3, or file3 is a subset of file2, or they are the same
crosscheck file3 file1 || echo file3 and file1 are neither one subset of the other

rm file1 file2 file3

Answer 7

Here is a (POSIX compatible) solution in AWK which checks if file1 is a superset of file 2:

awk 'FILENAME == ARGV[1] { lines[$0] = 1; next } \
    FILENAME == ARGV[2] && ! lines[$0] { exit 1 }' file1 file2