split file into N pieces with same name but different target directories

Question

I want to split sourcefile.txt which contains 10000 lines, (increasing everyday) into 30 equal files. I have directories called prog1 to prog30 and I would like to save split the file into these directories with the same filename. For example /prog1/myfile.txt, /prog2/myfile.txt to /prog30/myfile.txt.

Here is my bash script called divide.sh runs in prog directory

#!/bin/bash
programpath=/home/mywebsite/project/a1/
array=/prog1/
totalline=$(wc -l < ./sourcefile.txt)   
divide="$(( $totalline / 30 ))"   
split --lines=$divide $./prog1/myfile.txt    
exit 1
fi

Answer 1

#!/bin/bash

# assuming the file is in the same folder as the script
INPUT=large_file.txt
# assuming the folder called "output" is in the same folder
# as the script and there are folders that have the patter
# prog01 prog02 ... prog30
# create that with mkdir output/prog{01..30} 
OUTPUT_FOLDER=output

OUTPUT_FILE_FORMAT=myfile

# split 
# -n -> 30 files
# $OUTPUT_FILE_FORMAT -> should start with this pattern
# --numeric-suffixes=1 -> end of file name should start from 01 
split -n 30 $INPUT $OUTPUT_FILE_FORMAT --numeric-suffixes=1

# move all files to their repective directories
for i in {01..30} 
do
    mv $OUTPUT_FILE_FORMAT$i $OUTPUT_FOLDER/prog$i/myfile.txt
done

echo "done :)"

exit

The split command is more than enough for this task. However the solution here requires you to make your folder names start from prog01 and not prog1

Answer 2

The awk only solution (N here equals 30 files):

awk 'BEGIN{ cmd="wc -l <sourcefile.txt"; cmd|getline l; l=int((l+29)/30); close(cmd) } 
    NR%l==1{trgt=sprintf("prog%d",((++c)))}{print >trgt"/myfile.txt"}' sourcefile.txt

Or let shell run and return the number of lines in sourcefile.txt and pass to awk as suggested by jthill.

awk 'NR%l==1{trgt=sprintf("prog%d",((++c)))}{print >trgt"/myfile.txt"}' 
    l=$(( ($(wc -l <sourcefile.txt)+29)/30 )) sourcefile.txt

Answer 3

split + bash solution:

lines=$(echo "t=$(wc -l ./sourcefile.txt | cut -d' ' -f1); d=30; if(t%d) t/d+1 else t/d" | bc)
split -l $lines ./sourcefile.txt "myfile.txt" --numeric-suffixes=1

for f in myfile.txt[0-9]*; do 
    dir_n="prog"$(printf "%d" "${f#*txt}")  # constructing directory name
    mv "$f" "$dir_n/myfile.txt"
done

---

Assuming that you already have folders called prog1 to prog30 (as you mentioned)

• lines - contains the integer number of lines per output file

  • t - total number of lines of file ./sourcefile.txt
  • d=30 is a divider

• --numeric-suffixes=1 - split's option, tells to use numeric suffixes starting at 1

Answer 4

Sed version for fun:

lines=$(wc -l <sourcefile.txt)
perfile=$(( (lines+29)/30 ))     # see https://www.rfc-editor.org/rfc/rfc968.txt
last=0
sed -nf- sourcefile.txt <<EOD
$(while let $((last<lines)); do 
        mkdir -p prog$((last/perfile+1))
        echo $((last+1)),$((last+perfile)) w prog$((last/perfile+1))/myfile.txt
        : $((last+=perfile))
        done)
EOD

Answer 5

Steps

1. count the lines in file and divide by 30 lines = cat ${file} | wc -l

2. get the amount of files you need (bash will round it up to an integer) numOfFiles = ${lines} / 30

3. use split to divide the file split -l ${lines} -d --additional-suffix=-filename.extension ${file}

Expected result

x01-filename.extension, x02-filename.extension... xN-filename.extension

Wrap it into a for loop to process more than one file at a time

#!/bin/bash    
for FILE in $(find ${pathToWorkingDir} -type f -name "filename.extension")
do
    split -l ${lines} -d --additional-suffix=-filename.extension ${file}
    if [ $? -eq 0 ]; then
        echo "${file} splitted file correctly"
    else
        echo "there was a problem splitting ${file}"
        exit 1 #we exit with an error code
    fi
done
exit 0 #if all processed fine we exit with a success code

Answer 6

Parallelized with GNU Parallel:

parallel -j30 -a sourcefile.txt --pipepart --block -1 'mkdir -p prog{#};cat >prog{#}/myfile.txt'

This will run 30 jobs in parallel, splitting sourcefile.txt into one part per job (i.e. 30) and give the parts to cat that saves into prog{jobnumber}/myfile.txt.

Running in parallel GNU Parallel needs filehandles. This means that if you do not change the number of available file handles, you can at most run around 250 jobs in parallel.

But you can easily split into more blocks: They will just not be split all in parallel:

parallel -j30 -a sourcefile.txt --pipepart --block 1M 'mkdir -p prog{#};cat >prog{#}/myfile.txt'

Here we split into blocks around 1 MB with 30 jobs in parallel.