Ksh Script to ftp multiple directories simultaneously

Question

I'm new to this site and i have this problem:

I have a directory in unix with multiple directories in it. Each directory have around 5k files in it. So we are talking about 40k to 50k files. I need to send this over to a windows server, using FTP ( only because i only have FTP ). So, i have a script that,loops thru each directory and sent this files. However, this is painful slow, so i want to do it simultaneously. This is what i have right now, it start to send this files and then somehow never finishes. Log just show it was working with a 221 message in the end. However, this doesn't guarantee that all my files are sent. I do a manual count on them and can see that sometimes, when a folder have 5000 files sometimes just 800 are sent. Log doesn't shoow way.

Also my script keep running long after the transfer stops. i can see it by using ps -ef.

Can someone take a look and advise any improvement or why im getting this weird behavior ?

Some info on my setup:

• HP-UX 9000/859 B.10.20 E
• Ksh version : How ? tried --version , echo $KSH_VERSION ,swlist, nothing worked

My Script :

#! /usr/bin/ksh

if [[ $# -eq 0 ]]; then
  print "No arguments, Please enter password for ftp process"
  exit
fi

exec 4>~/ftpParallel.log

#Directory to send
CONVERTED_DIR=/data/history/
#FTP Variables
HOST=xxxxx.com
PORT=8009
USER=yyyyy
PASS=$1

ftpFiles(){
    #   Do some processing and lets get the group and the dategroup, Format will be#    /DATA/BRCPCB/201101
    GROUP=$1
    DATEGROUP=$2
    #now mount the destdir based on the curent dir
    DESTDIR=/DATA/$GROUP
    cd $CONVERTED_DIR/$GROUP/$DATEGROUP
    i=0
    ftp -nv >&4 2>&4 |&
    print -p open $HOST $PORT
    print -p user $USER $PASS
    print -p mkdir $DESTDIR
    print -p mkdir $DESTDIR/$DATEGROUP
    print -p cd $DESTDIR/$DATEGROUP
    ls | while read filename ; do
      [[ -f $filename ]] && print -p put $filename
      (( i += 1 ))
    done
    print -p close
    print -p bye
    print -p "$DATEGROUP send $i files"
}

#Get All Folders structure, we will need it to iterate and search for PeakPro Files Later
a=`find $CONVERTED_DIR -type d  2>/dev/null | awk 'BEGIN{FS="/"}{if($NF ~/^[0-9]{6}$/)print $(NF-1),$NF}'` 
echo "$a" | while read item ; do
   ftpFiles $item & #this will make the function be called in background
done
wait
exit 0

---

UPDATE:

I've changed the code as requested and found new interesting things. It appears that my FTP jobs keep running, even after they stopped: This are the FTP logs :

$ tail -5 ftpParallel200103.log
150 Opening ASCII mode data connection for C31905.CVFS.
226 Transfer complete.
15931 bytes sent in 0.01 seconds (2117.55 Kbytes/s)
200 PORT command successful.
150 Opening ASCII mode data connection for C31905.RVFS.
$ tail -5 ftpParallel200104.log
200 PORT command successful.
150 Opening ASCII mode data connection for WG4829.RVFS.
226 Transfer complete.
12110 bytes sent in 0.01 seconds (1011.91 Kbytes/s)
221  
$ tail -5 ftpParallel200105.log
150 Opening ASCII mode data connection for C51047.CVFS.
226 Transfer complete.
159734 bytes sent in 0.15 seconds (1027.98 Kbytes/s)
200 PORT command successful.
150 Opening ASCII mode data connection for C51047.RVFS.
$

As you can see, only 1 of them finished ( Code 221 - FTP BYE ). Others never finished and job keep running ( i've started it using nohup ftpParallel.sh & ) :

 rcsanto  8314  8299  5 10:15:27 ttyq6     0:00 ps -ef
 rcsanto 25834 25833  0 05:35:00 ?         0:00 ls
 rcsanto 25828 25826  0 05:35:00 ?         0:00 ls
 rcsanto 25813 25808  0 05:35:00 ?         0:27 ftp -nv
 rcsanto 25815 25808  0 05:35:00 ?         0:19 ftp -nv
 rcsanto 25833 25816  0 05:35:00 ?         0:01 ftpParallel.sh ./ftpParallel.sh roJabSuP08WJjco
 rcsanto  8299  8298  0 10:15:15 ttyq6     0:00 -sh
 rcsanto  8315  8299  1 10:15:27 ttyq6     0:00 grep rcsanto
 rcsanto 25808     1  0 05:34:46 ?         0:00 ftpParallel.sh ./ftpParallel.sh roJabSuP08WJjco
 rcsanto 25826 25815  0 05:35:00 ?         0:02 ftpParallel.sh ./ftpParallel.sh roJabSuP08WJjco
 rcsanto 25816 25808  0 05:35:00 ?         0:08 ftp -nv
 rcsanto 25825 25813  0 05:35:00 ?         0:02 ftpParallel.sh ./ftpParallel.sh roJabSuP08WJjco
 rcsanto 25827 25825  0 05:35:00 ?         0:00 ls

This is the nohup LOG :

FTP starting at: Tue Feb  5 04:51:48 CST 2013        rm: /homrm: /homrm: /homrm: /homrm: /homrm: /home/rcsante/rcsante/rcsante/rcsante/rcsante/rcsanto/ftpParo/ftpParo/ftpParo/ftpParo/ft
    allel200allel200104.log non-existent
    107.log non-exi106.log304.log non-existent
     non-existent
     stent
     non-existent

I believe the variables are being messed up somehow. IT also appears that the FTP dies within an hour of running, maybe its timming out ??

-rw-rw-rw-   1 rcsanto    pp_user     249853 Feb  5 05:51 ftpParallel200103.log
-rw-rw-rw-   1 rcsanto    pp_user     937693 Feb  5 06:22 ftpParallel200104.log
-rw-rw-rw-   1 rcsanto    pp_user     172395 Feb  5 05:47 ftpParallel200105.log
-rw-rw-rw-   1 rcsanto    pp_user      88497 Feb  5 05:41 ftpParallel200106.log
-rw-rw-rw-   1 rcsanto    pp_user     981598 Feb  5 06:24 ftpParallel200107.log
-rw-rw-rw-   1 rcsanto    pp_user     819814 Feb  5 06:21 ftpParallel200304.log

Thanks for all your help.

Answer 1

It seems to me that your loop, at the end of the script, is not working. You collect a number of couples in one variable, namely a. All those couple are separated by spaces because all newlines printed by awk are converted in spaces by the shell. Then you print it, via echo as only one line. And you read it back in one variale, namely item. So item has the same content than a. This means that your loop only loops once. If I understood what you are doing, you may problably change it this way:

find $CONVERTED_DIR -type d 2>/dev/null \
  | awk 'BEGIN{FS="/"}{if($NF ~/^[0-9]{6}$/)print $(NF-1),$NF}'` \
  | while read group dategroup
do
  ftpFiles $group $dategroup & #this will make the function be called in background
done

Beside this, having a parallel transfer instead of a serial one, will probably not speed up your procedure as expected, because you are transferring data over the same wire. Moreover, you have the overhead of opening many sockets instead of only one.

Last note: I suggest you to have different log files, because otherwise all output from different ftp will be mixed together.

update: The internal function may probably be rewritten this way:

( echo open $HOST $PORT
  echo user $USER $PASS
  echo mkdir $DESTDIR
  echo mkdir $DESTDIR/$DATEGROUP
  echo cd $DESTDIR/$DATEGROUP
  ls | while read filename ; do
      [[ -f $filename ]] && echo put $filename
      (( i += 1 ))
    done
  echo close
  echo bye
  echo "$DATEGROUP send $i files" >&4 ) | ftp -nv >&4 2>&4