Faster way to build text file in python

Question

I have two 3d numpy arrays, call them a and b, 512x512x512. I need to write them to a text file:

a1 b1
a2 b2
a3 b3
...

This can be accomplished with a triple loop:

lines = []
for x in range(nx):
    for y in range(ny):
        for z in range(nz):
            lines.append('{} {}'.format(a[x][y][z], b[x][y][z])
print('\n'.join(lines))

But this is brutally slow (10 minutes when I'd prefer a few seconds on a mac pro).

I am using python 3.6, latest numpy, and am happy to use other libraries, build extensions, whatever is necessary. What is the best way to get this faster?

akuiper · Accepted Answer · 2017-03-31 21:51:11Z

4

You could use np.stack and reshape the array to (-1, 2) (two columns) array, then use np.savetxt:

a = np.arange(8).reshape(2,2,2)
b = np.arange(8, 16).reshape(2,2,2)

np.stack([a, b], axis=-1).reshape(-1, 2)

#array([[ 0,  8],
#       [ 1,  9],
#       [ 2, 10],
#       [ 3, 11],
#       [ 4, 12],
#       [ 5, 13],
#       [ 6, 14],
#       [ 7, 15]])

Then you can save the file as:

np.savetxt("*.txt", np.stack([a, b], axis=-1).reshape(-1, 2), fmt="%d")

answered Mar 31, 2017 at 21:51

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Henry Over a year ago

Wow I definitely could have googled that one, thank you for the complete example!

juanpa.arrivillaga Over a year ago

@Henry you almost never should be writing for-loops with numpy. At the very least, it is admitting defeat!

plasmon360 · Accepted Answer · 2017-03-31 21:53:55Z

1

you could use flatten() and dstack(), see example below

a = np.random.random([5,5,5]).flatten()
b = np.random.random([5,5,5]).flatten()
c = np.dstack((a,b))
print c

will result in

[[[ 0.31314428  0.35367513]
  [ 0.9126653   0.40616986]
  [ 0.42339608  0.57728441]
  [ 0.50773896  0.15861347]
....

answered Mar 31, 2017 at 21:53

plasmon360

4,1991 gold badge21 silver badges19 bronze badges

Comments

Carolyn Conway · Accepted Answer · 2017-03-31 21:50:46Z

-1

It's a bit difficult to understand your problem without knowing what kind of data you have in those three arrays, but it looks like numpy.savetxt could be useful for you.

Here's how it works:

import numpy as np
a = np.array(range(10))
np.savetxt("myfile.txt", a)

And here's the documentation: https://docs.scipy.org/doc/numpy/reference/generated/numpy.savetxt.html

answered Mar 31, 2017 at 21:50

Carolyn Conway

1,5461 gold badge17 silver badges21 bronze badges

1 Comment

hpaulj Over a year ago

But savetxt expects a 2d array

Collectives™ on Stack Overflow

Faster way to build text file in python

3 Answers 3

2 Comments

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

1 Comment

Related