Merging every nth files in a folder and delete used one

Question

I have folder with 30000 txt files, each file is 50-60kb. I need to merge them into 2.5mb txt files.And remove the one that was merging. My code would need to be something like: for f in *,50; do cat file1,file2...file49 > somefile.txt;done Of course this is pseudocode. I would need to merge files in package of 50 files, then remove the used one. Can someone please help me?

Answer 1

With zsh:

files=( ./input-file*(Nn.) )
typeset -Z3 n=1
while
 (( $#files > 0 )) &&
   cat $files[1,50] > merged-file$n.txt &&
   rm -f $files[1,50]
do
  files[1,50]=()
  ((n++))
done

There ./input-file*(Nn.) expands to the files that match ./input-file*, but with 3 glob qualifiers further classifying that:

• N: nullglob: makes the glob expand to nothing instead of aborting with an error when there's no match. That one you often want when setting an array from a glob and it's fine for that array to be empty in the end:
• n: numericglobsort: change the sorting from a default of lexical to numerical (in effect combination of both), so that input-file2 sorts before input-file10 for instance.
• .: restrict to regular files (ignore directories, symlinks, fifos...)

typeset -Z3 n makes $n a variable zero-padded to width 3, so we get merged-file001.txt, ... merged-file049.txt...

Then we loop as long as there are elements in the $files array and there's no error, concatenating batches of 50 at a time (and whatever's left for the last batch).

The same with bash 4.4+ and GNU tools:

readarray -td '' files < <(
  LC_ALL=C find . -maxdepth 1 -name 'input-file*' -type f -print0 |
    sort -zV
)
n=0
set -- "${files[@]}"
while
 (( $# > 0 )) &&
   printf -v padded_n %03d "$n" &&
   cat "${@:0:50}" > "merged-file$padded_n.txt" &&
   rm -f "${@:0:50}"
do
  shift "$(( $# >= 50 ? 50 : $# ))"
  ((n++))
done

Where find does the job of zsh's ./input-file*(N.), sort -V does the numeric (version) sort, and we use positional parameters and shift in the loop as bash arrays are quite limited.

Answer 2

This script is:

1. For bash (as tagged),
2. Avoiding find (which fails on invalid characters),
3. Making sure that only plain files are processed (no dirs),
4. Using sort to sort numerically (well, by version) and
5. Joining on k files (variable count)
6. Removing one file at a time (avoid copying a block of files that wont get erased)

dir="myDir"

readarray -td $'\0' files < <(
   for f in ./"$dir"/in-file*; do
       if [[ -f "$f" ]]; then printf '%s\0' "$f"; fi
   done |
       sort -zV
)

k=50
rm -f ./"$dir"/joined-files*.txt
for i in "${!files[@]}"; do
   n=$((i/k+1))
   cat "${files[i]}"  >> ./"$dir"/joined-files$n.txt &&
       rm -f "${files[i]}"
done