How to traverse a file and extract words using their index?

Question

The given file must not be stored in variable and then traversed due to memory size restrictions:

Example:

var=$(cat FILE)
for i in $var
do
  echo $i
done

How do you traverse all strings in a file in the same way as the example above but extract each whitespace-separated directly from the file?

Example:

fileindex=1
totalfilecount=$(cat FILE | wc -w)
while (( ${fileindex} <= ${totalfilecount} ))
do
  onefilename= ??? missing command using fileindex
  ((fileindex+=1))
done

Is there a command that can view a file as an array and allow you to extract words using their index positions?

The idea is to process every word in the file as though the file were an array.

Input file example:

one two
three four
five six

Here is the scenario that requires the above funtionality:

• we have server_A and server_B
• server_A needs to connect to server_B via sftp (sftp only) and 'get' some files
• BOTH 'ls' or 'ls -l' commands in sftp can be using wild cards to filter specific files
• each file needs to be processed individually (for various reasons) on the fly
• the files cannot be copied as a group to server_B and then processed individually
• a list of files must first be created on server_A and then each file in that list is copied from server_B and processed one file at a time

Where is the problem?

The problem is how the 'ls' command can create a dual column list of words if the list is long thus not allowing simple processing as with 'ls -l' which always creates a single column list.

This leads us to my initial question, if such a solution exists.

Answer 1

You can do this per word using awk, which should meet your memory requirements:

awk -v RS=\  '{
    # Do something with the word
    print
}' file

You can specify which string you want by using NR.

$ awk -v RS=\  'NR==2{print}' <<< 'foo bar baz'
bar

Answer 2

When you say “strings” you mean “words”, right? Strings of characters separated by whitespace. And according to your examples, you want to access them sequentially.

You can do:

$ sed 's/[ \t]\+/\n/g' YOUR_FILE | while read -r word ; do PROCESS $word ; done

Example of use:

% echo word1 word2 > YOUR_FILE
% echo word3 word4 >> YOUR_FILE
% echo word5 word6 >> YOUR_FILE
% sed 's/[ \t]\+/\n/g' YOUR_FILE | while read -r word ; do echo _${word}_ ; done
_word1_
_word2_
_word3_
_word4_
_word5_
_word6_