Write data file line by line in python

Question

My code generates a 2 dimensional array, which I want to save into a file. This can be easily done using numpy.savetxt as shown in the sample code. But I wish to do something different. In the following code I generate random walks for 5 walkers. For each walk, I generate positions of the walker for some time and store the positions in a list. After each walk is completed , I add the whole list containing walker's positions, as a new row to an already existing array S_data. At the end of the next walk (or iteration), I add the list of position for the next walker and so on.. So by the end of each walker's iteration, a new row is added to S_data. At the end of all walks , this S_data is finally saved to an external file using numpy.savetxt .

import numpy as np

no_of_walkers = 5
t_max = 10

S_data=np.zeros(t_max)

for R in range(no_of_walkers):
    position = []
    x = 0.0
    for t in range(t_max):
        x = x + np.random.randint(-1, 2)
        position.append(x)
    S_data = np.vstack([S_data, position])

S_data = np.delete(S_data, obj=0, axis=0)

np.savetxt('data_file.txt', S_data)

What I wish to do is, at the end of each walk, instead of adding the position list to S_data, I want to write it to an external file. Again at the end of next iteration, I will add the position list for the next walker as a new line to the external file, and so on. I want to do this because, I will have to run this code for large number of walkers and also for longer times, so then my code will consume lot of RAM, which I want to avoid. Is there a pythonic way to export lists to the same external file at the end of each iteration, without overwriting the previous data?

Answer 1

If I understand correctly, it seems like you are trying to append to a file after each iteration (if your data file already exists), if so use:

f = open("data_file.txt", "a+")

The "a+" indicates you wish to append with any subsequent calls to write. Docs here. If you are doing a bunch of iterations, be sure to open the file outside of any loops.

Of course you could also loop to write each element of the new row after each iteration and then just write a newline with something like this:

#open file
for item in newRow:
  #Your formatting will determine the specifics of this write
  f.write(item)
f.write("\n")

Something like this does what I think you are getting at:

# Use with for file safety
with open('output.txt', 'w') as f:    
    for R in range(no_of_walkers):
        x = 0.0
        for t in range(t_max):
            x = x + np.random.randint(-1, 2)
            f.write(str(x) + ' ')
        f.write('\n')

output.txt

1.0 1.0 1.0 1.0 1.0 0.0 0.0 -1.0 -2.0 -2.0 
0.0 1.0 0.0 0.0 -1.0 0.0 1.0 0.0 1.0 1.0 
-1.0 -1.0 -1.0 -2.0 -1.0 -1.0 -2.0 -1.0 -2.0 -3.0 
-1.0 -1.0 0.0 1.0 0.0 1.0 2.0 1.0 2.0 1.0 
1.0 0.0 -1.0 -2.0 -1.0 -1.0 0.0 1.0 1.0 0.0 

As for efficiency, I'm pretty sure the way you were originally trying to write is better. See this post.