Concatenate specific files into one single file

Question

In a folder containing X files, I need to concatenate Y files (where X > Y) together into a single text file. I have the filenames (of Y files) that I need to concatenate.

Answer 1

You can use the cat command (see man cat for more information) to concatenate the text files.

If you want to create a new file

cat [FILE1] [FILE2]... > new_file

or if you want to append to an existing file use it like this

cat [FILE1] [FILE2]... >> file 

Answer 2

If the Y filenames are listed in a list file, a simple combination of xargs and cat is enough:

xargs cat <list >>concatenation_of_files

In the case you've been careful and you've listed files one per line (to avoid problems with spaces in filenames), then just add a -d delimiter option:

xargs -d'\n' cat <list >>concatenation_of_files

(This assumes concatenation_of_files is initially inexistent or empty).

Answer 3

Answers are using bash.

Let's say you want to concat json files in your directory:

cat *json > all.json

Including subdirectories:

shopt -s globstar
cat **/*json > all.json

However, if you have thousands to millions of files, you will encounter:

bash: /bin/cat: Argument list too long

In which case, do the following which also outputs the current file being processed:

shopt -s globstar
rm concat.json
find . -path "**/*.json" | while read -r file; do
    echo -ne "\\r$file"
    cat "$file" >> concat.json
done

Posted on this question, as this other question became closed.

Answer 4

(I know this is REALLY late and not actually an answer but a suggestion if you are doing this.)

Do not use the same file extension as the files that you are concatenating.

cat *.json > all.json.txt

Then come back and...

mv all.json.txt all.json

Not actually sure if this solves the problem in the question but this will avoid the looping issue.