Delete files - Keep newest based on filename pattern

Question

I have a directory with a lot of files. Each file have the same pattern <id>_data_<date>.csv. What i want to do is to delete all files but to keep the newest per <id>.

Example directory:

10020077_data_2017-07-18_001.csv
10020078_data_2017-07-18_001.csv
10020209_data_2019-04-23_001.csv
10020209_data_2019-04-24_001.csv
10020209_data_2019-04-25_001.csv
10020209_data_2019-04-26_001.csv
10020209_data_2019-04-27_001.csv
10020209_data_2019-04-28_001.csv
10020272_data_2019-04-23_001.csv
10020272_data_2019-04-24_001.csv
10020272_data_2019-04-25_001.csv
10020272_data_2019-04-26_001.csv
10020272_data_2019-04-27_001.csv
10020272_data_2019-04-28_001.csv
10020286_data_2019-04-23_001.csv

Expected result:

10020077_data_2017-07-18_001.csv
10020078_data_2017-07-18_001.csv
10020209_data_2019-04-23_001.csv <-- delete
10020209_data_2019-04-24_001.csv <-- delete
10020209_data_2019-04-25_001.csv <-- delete
10020209_data_2019-04-26_001.csv <-- delete 
10020209_data_2019-04-27_001.csv <-- delete
10020209_data_2019-04-28_001.csv
10020272_data_2019-04-23_001.csv <-- delete
10020272_data_2019-04-24_001.csv <-- delete
10020272_data_2019-04-25_001.csv <-- delete
10020272_data_2019-04-26_001.csv <-- delete
10020272_data_2019-04-27_001.csv <-- delete
10020272_data_2019-04-28_001.csv
10020286_data_2019-04-23_001.csv

In this case i can not work with find -mtime because some ids get new files daily and some others only once a month or sometimes only once a year.

My idea was to group the filenames per id and than to keep the last item. How can i solve this with bash?

Answer 1

There's no particular need for bash here; a simple sh script could do it, utilizing the positional array twice. The outer loop picks up all of the desired data files (wildcards for the id and date portions); it pulls off the id portion then starts a subshell to loop through all of the files with that id. That subshell then loops through the naturally date-sorted list of those files and deletes all except for the last one, preserving the newest one.

#!/bin/sh

set -- *_data_*.csv
for f in "$@"
do
  id=${f%%_*}
  # a subshell so we don't clobber $@
  (
        set -- "${id}"_data_*.csv
        while [ "$#" -gt 1 ]
        do
          rm -- "$1"
          echo "DELETE: $1"
          shift
        done
  )
done

I added an echo ... DELETE statement so I could demonstrate the results on the filenames you provided:

DELETE: 10020209_data_2019-04-23_001.csv
DELETE: 10020209_data_2019-04-24_001.csv
DELETE: 10020209_data_2019-04-25_001.csv
DELETE: 10020209_data_2019-04-26_001.csv
DELETE: 10020209_data_2019-04-27_001.csv
DELETE: 10020272_data_2019-04-23_001.csv
DELETE: 10020272_data_2019-04-24_001.csv
DELETE: 10020272_data_2019-04-25_001.csv
DELETE: 10020272_data_2019-04-26_001.csv
DELETE: 10020272_data_2019-04-27_001.csv

Answer 2

You can do it with a sequence of commands in a one-liner too, as long as you have mktemp, tee, sort, grep, xargs, and of course rm available on your system. If you don't have tac you can replace it with sort -r:

(temp_all=$(mktemp) && temp_last=$(mktemp) && { tac | tee $temp_all | sort -un > $temp_last ; } && grep -vf $temp_last $temp_all ; rm -f $temp_last $temp_all)

The above expects the whole list of files in stdin (it can come from whatever means you see fit, a find, an ls, a file, etc.) and shows the list of files to delete. You can then pipe such list to an xargs rm

Broken down:

(
temp_all=$(mktemp) && \
temp_last=$(mktemp) && \ # make a couple of temp files
{ 
    tac | \              # reverse the list of files and ...
    tee $temp_all | \    # pipe it into one temp entirely and also ...
    sort -un > $temp_last ; \ # into a sort that makes names unique into the other temp
} && \
    grep -vFf $temp_last $temp_all ; \ # use grep to filter out names
rm -f $temp_last $temp_all  # remove temp files
)

This can deal with any number of input names, but requires that no name has embedded newlines. It seems reasonable for your case.

Answer 3

Put all file names in l.txt
Proceed with below steps and it worked fine

da=`awk -F "_" '{print $3}' l.txt | sort | uniq| sort -nr| sed -n '1p'`


 for id in `awk -F "_" '{print $3}' l.txt | sort | uniq`
> do
> find  path -maxdepth 1 -type f -newermt $da -iname "$id*"  | sed -n '2,$p'| awk '{print "rm" " " $1}'| sh;done

Answer 4

I know there are already a lot of answers, but here as alternative in Python. You don't need to loop through the files twice.

#!/usr/bin/env python                                                           

import os
import glob

if __name__ == '__main__':
    newest_dict = dict()

    for f in glob.glob('*.csv'):
        id = f[:8]

        if id not in newest_dict:
            newest_dict[id] = f
        else:
            nf = newest_dict[id]

            f_ts = f[14:24]
            nf_ts = nf[14:24]

            if f_ts > nf_ts:
                newest_dict[id] = f
                print("Deleting", nf)
                os.remove(nf)                                                  
            else:
                print("Deleting", f)
                os.remove(f)