Check whether files in a file list with changing date exist in a certain directory print missing

Question

So I have a list file that goes like this:

SOURCE_FILE|SOURCE_SYSTEM|DELIVERY_FREQUENCY|IMPORT_DIRECTORY|PROCESSING_BLOCKS|DEPENDENT_DELIVERY_CODES|AFFECTED_TARGET_SYSTEMS

example:

SUBMIT_303_EXOTICIIBOND_YYYYMMDD_V01_T23.xml.gz|STF|DAILY|fromSTF|FLOWS|SMTMM|CFP;
YYYYMMDD_PES_TER_99_99.tar.gz|PALO|MONTHLY|fromPO|IRBDD|PES_TER|CFP

and a directory with the files:

SUBMIT_303_EXOTICIIBOND_20220720_V01_T23.xml.gz;
20220720_PES_TER_99_99.tar.gz).

I need to check if the file from the list really exists and if not write it into file.

I was able to create the script to sort out the names of files that I need cat file_path | grep "DAILY" | cut -d '|' -f1 so it's just the names of files.

Any suggestions on how to implement the output into !-f with the daily changing dates?

Answer 1

I've been diving deeper into gawk recently:

gawk -F'|' '
  @load "filefuncs"
  $3 == "DAILY" {
    result = stat($1, statdata)
    if (result != 0)
      print $1
  }
' file.list > files.not.found

Or, expressed as a one-liner:

gawk -l filefuncs -F'|' '$3 == "DAILY" && stat($1, data) {print $1}' file.list > files.not.found

Ref https://www.gnu.org/software/gawk/manual/html_node/Extension-Sample-File-Functions.html