Bash script to process list of (folder) names

Question

I am trying to create a bash script that iterates a text file wich contains a list of folder names, and checks if any of these folders are missing from a certain path, in this case a directory called 2021.

My list.txt file contains a list like this one:

100001
100002
100003
100004
...

and I have a directory (the so called 2021) that contains sub-folders whos names I want to check

100001
100002
100004
...

Now, in this example, the script should state that folder 100003 is missing. The purpose of the script is just to find which of the folders listed in the text file are missing from the path

So far, I have:

while IFS= read -r line; do
    if [[ ! -d "2021/$line" ]]; then
        echo $line is missing
    fi
done <<< $(cat list.txt)

but this code just prints all the folders that exist inside 2021 directory, so I am guessing that there is something wrong inside the IF statement, but I can't figure this out.

Answer 1

I assume you are running the shell script outside the path which contains 2021 folder.
Having this structure:

$> tree
#output: 
.
├── 2021
│   ├── 100000
│   ├── 100001
│   ├── 100002
│   ├── 100003
│   ├── 100006
│   ├── 100007
│   ├── 100008
│   ├── 100009
│   ├── 100010
│   ├── 100011
│   ├── 100019
│   ├── 100020
├── list.txt
└── script

You should be able to run your current script without changing anything by using simply: ./script

The problem is that if you run script outside the script directory, the code will not work successfully. So I suggest you change your script to this:

#!/bin/bash

filepath="$(dirname $(readlink -f $0) )"
cd $filepath

while read -r line; do
    if [[ ! -d "2021/$line" ]]; then
        echo $line is missing
    fi
done <<< $(cat list.txt)
#Or:
#  done < /paht/to/list.txt

I removed IFS= . I'm not sure why you have it, but if you really have to use then just add it

So you will be able to run your script from any directory, e.g. ./path/to/dir/script

---

You can also get the missing directories by using the following command:

#With this code you will get only the missing dirs. The text 'is missing'
#won't appear here
diff --new-line-format="" --unchanged-line-format=""  <(sort /path/to/list.txt)  <(find . -maxdepth 1 -type d ! -path . -printf '%P\n' | sort)

If your file list.txt is actually sorted then you can change <(sort /path/to/list.txt) to simply /path/to/list.txt

If you use the code above make sure you are placed under 2021 directory as well as changing /path/to/list.txt to the actual path of list.txt

Answer 2

After hours searching for the problem, I've finally found the solution.

The problem was that the file was initially created under windows OS and then used in a bash shell, which means, in Windows lines end with both a line feed and a carriage return, but in Unix they end only with a single line feed.

So I am guessing that, when the IF compared the name in the file with the name in the folder, it obviously couldn't be a match, because the name inside the list file had an extra carriage return, invisible to any eye.

So I've recreated the file under unix, and now, Lo and Behold, I am getting the expected result flawlessly. Thanks to all who tried to help.

Answer 3

I using zsh instead of bash is an option, you could use its superior globs and its array conjunction (${A:*B}) and subtraction (${A:|B}) operators:

#! /bin/zsh -
list=path/to/list.txt
dir=path/to/2021
# or list=${1?} dir=${2?}
# for instance to pass those as arguments to the script

expected=( ${(f)"$(<$list)"} )
# or expected=( ${(f)"$(dos2unix < $list)"} )
# if that file is in MS DOS/Windows format

got=( $dir/*(ND/:t) )
# or got=( $dir/*(ND-/:t) )
# to also consider symlinks to directories like your [[ -d ... ]] does

report() {
  if (( $# > 1 )); then
    print -rC1 -- $1: ' - '$^@[2,-1]
   else
    print -r   -- $1: none
  fi
}

report Found      ${expected:*got}
report Missing    ${expected:|got}
report Unexpected ${got:|expected}