How to concatenate files from different sub-directories?

Question

I have a large folder containing many sub-directories each holding many .txt files. I want to concatenate all of these files into one .txt file. I am able to do it for each of the sub-directories with cat *.txt>merged.txt, but I am trying to do it for all of the files in the large folder. How do I do this?

Answer 1

try with

find /path/to/source -type f -name '*.txt' -exec cat {} + >mergedfile

find all '*.txt' files in /path/to/source recursively for sub-directories and concatenate all into one mergedfile.

To concatenate each sub-directories files within its directory, do:

find . -mindepth 1 -type d -execdir sh -c 'cat $1/*.txt >> $1/mergedfile' _ {} \;

Answer 2

If you are using Bash and the number of text files is contained (i.e. does not exceed the maximum argument number limit, which is very large but not infinite), you can easily achieve this with the globstar feature:

$ shopt -s globstar
$ cat **/*.txt > merged.txt

A more general, although less elegant approach, will be to use find as the driver and make it call cat on each file, appending the output:

$ find -name \*.txt -exec sh -c 'cat {} >> merged.out' \;

Calling sh is needed here because you want to append the result of each cat. Make sure the output file has a different extension or lies outside of the tree you're merging, or find may try to concatenate the output with itself.

Answer 3

If you have to do the concatenation in a particular order, then the below will concatenate the files in lexicographical order (sorted by pathnames) in bash:

shopt -s globstar
for name in **/*.txt; do
    [ -f "$name" ] && cat <"$name"
done >merged.out

This is similar to the find command

find  . -type f -name '*.txt' -exec cat {} ';' >merged.out

except that the ordering may be different, symbolic links to regular files would be included (add a && [ ! -L "$name" ] if you don't want them) and hidden files (and files in hidden directories) would be excluded (use shopt -s dotglob to add them back).