Write numpy array to csv file with a specific number of columns

Question

I have a numpy array of shape (444,445), and I need to dump it as a csv file. One can achieve this by:

np.savetxt('outfile.txt',array, delimiter=',',fmt="%s")

I use the fmt="%s" option, because at the end of each row (the 444 element of the array, is NaN).

What I would like to accomplish is to write a csv file that is 5 column wide, with 39,516 total lines (that is, 89 sections each of which consist of 5 columns and 444 lines), and finally the NaN written as an empty element on the end of the 444th line. In this way, one has the number of elements of the matrix that are equivalent: 89x5x444=444x445, or 197,580 pieces of data.

For instance:

  1 xxxx,xxxx,xxxx,xxxx,xxxx,
  2 xxxx,xxxx,xxxx,xxxx,xxxx,
    ...
    ...
 89 xxxx,xxxx,xxxx,xxxx,
 90 xxxx,xxxx,xxxx,xxxx,xxxx,
 91 xxxx,xxxx,xxxx,xxxx,xxxx,
    ...
    ...
178 xxxx,xxxx,xxxx,xxxx,

I have added the line number to be more clear in my question. I do not want it in the actual output.

What would be an efficient and pythonic way of doing so?

For the moment, I am trying to adapt the answer to this question to my case:

Write to list to rows with specific number of columns.

@RickTeachey well, in some way yes. Because 5 columns times 89 lines is 444. I tried reshaping my array to (89,5,444) but np.savetxt does not give me back what I need. — muammar
– muammar, Commented Jun 5, 2015 at 11:56
if your array is of shape 444x445, there are 197,580 pieces of data. if i understand, what you want is a csv file that is 39,516 lines long (with 5 columns), correct? and the 89th line, 178th line, etc, will end in 'nan'. is that right? — Rick
– Rick, Commented Jun 5, 2015 at 13:09
Actually the 89th line, 178th line, etc, should not end with 'nan' but have one less element. I don't think this can be achieved with a single call to numpy.savetxt(). With masked arrays you can obtain something like xxxx,xxxx,, i.e. not a row with one less element, but a row with an empty element. — Stefano M
– Stefano M, Commented Jun 5, 2015 at 13:26

Community · Accepted Answer · 2017-05-23 11:43:39Z

Hope I well understand what you are asking for

# Reshape it

array_.reshpe(89,444,5)

# Change it's dtype to str so you can replace NaN by white spaces

array_.astype(str)

# Replace nan by white spaces

array_[array_ == 'nan'] = ''


# Finaly, save it SEE EDIT

Edit

I think that np.savetxt wouldn"t work with numpy arrays with more than 2 dimension, so, and referring to this answer we can try this:

# Write the array to disk
with file('test.txt', 'w') as outfile:
    # I'm writing a header here just for the sake of readability
    # Any line starting with "#" will be ignored by numpy.loadtxt
    outfile.write('# Array shape: {0}\n'.format(array_.shape))

    # Iterating through a ndimensional array produces slices along
    # the last axis. This is equivalent to array_[i,:,:] in this case
    for data_slice in array_:

        # The formatting string indicates that I'm writing out
        # the values in left-justified columns 7 characters in width
        # with 2 decimal places.  
        np.savetxt(outfile, data_slice, fmt='%-7.2f')

        # Writing out a break to indicate different slices...
        outfile.write('# New slice\n')

There is also reshape. I like a lot your solution, but when you try to save the .astype(str) array without the nan -> ` TypeError: float argument required, not numpy.ndarray`.
@muammar, I realize that there is a problem with saving a numpy array with (a,b,c) as shape. take a look to the edit and tell me if it works or you want to customize it
To save in disk, the fmt has to be set to fmt='%s', otherwise it fails. Thank you very much for your help. I will mark your solution as the correct answer.

Collectives™ on Stack Overflow

Write numpy array to csv file with a specific number of columns

1 Answer 1

5 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Linked

Related