Thanks to @StephenHarris

ipset command's output is generated on stderr (not stdout) and 2>&1 captures the output to the variable.

str=$((/usr/sbin/ipset test IPsetName 1.1.1.1) 2>&1)

if [[ $str = *"The set with the given name does not exist"* ]]; then
   echo "IPsetName not found"
fi

Now this if statement works as expected!

Thanks to @StephenHarris

ipset command's output is generated on stderr (not stdout) and 2>&1 captures the output to the variable.

str=$(/usr/sbin/ipset test IPsetName 1.1.1.1 2>&1)

if [[ $str = *"The set with the given name does not exist"* ]]; then
   echo "IPsetName not found"
fi

Now this if statement works as expected!