Copy directory structure with random number of files

Question

Is there an elegant and fast way to copy a certain directory structure and only select a random amount of files to be copied with it. So for example you have the structure:

--MainDir
  --SubDir1
    --SubSubDir1
      --file1
      --file2
      --...
      --fileN
    --...
    --SubSubDirN
      --file1
      --file2
      --...
      --fileN
  --...

I want to copy the entire folder structure but choose only a specific number of random files from {files1-filesN} of each SubSubDir to be copied along.

Answer 1

Since you tagged this as linux I'll assume GNU utilities.

Copy directory structure from $src to $dest:

find "$src" -type d -print0 | cpio -padmv0 "$dest"

Also copy a random sample of $nfile files from each leaf subdirectory of $src:

find "$src" -type d -links 2 -exec \
    sh -c 'find "$1" -type f -print0 | shuf -z -n "$2"' sh {} "$nfiles" \; | \
    cpio -padmv0  "$dest"

Here the first find finds leaf subdirectories (-links 2), then the second find finds files in each of these subdirectories. shuf chooses a random sample of files, and finally cpio copies them.

Answer 2

First find all directories:

find MainDir -type d

Then parse these directories to a script

find MainDir -type d -exec ./randomCopy.sh 2 {} \;

, which

1. Creates the target directory
2. Copy's an random amount of files.

In this case 2 random files are copied.

In my example the script randomCopy.sh looks like this:

#!/bin/bash                                                                                                                                                                                                                                                                    
cnt="$1"                                                                                                                                                                                                                                                                     
dir="$2"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
mkdir -p "TARGET/$dir"                                                                                                                                                                                                                                                       

# see: https://stackoverflow.com/questions/414164/how-can-i-select-random-files-from-a-directory-in-bash                                                                                                                                                                     
find "$dir" -maxdepth 1 -type f | sort -R | tail -n $cnt | while read file; do                                                                                                                                                                                               
  # copy the file                                                                                                                                                                                                                                                            
  cp "$file" "TARGET/$dir/"
done

And don't forget to make the script executable: chmod +x randomCopy.sh.

Replace the string TARGET with your target directory or use a third script-option.

This proof of concept is running inside my test directory, but there may be a lot to improve.