Updating a NumPy array by adding columns

Question

I am working with a large dataset and I would like to make a new array by adding columns, updating the array by opening a new file, taking a piece from it and adding this to my new array.

I have already tried the following code:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            Powers = np.append(Powers,Pname, axis = 1)
            np.savetxt("Powers.txt", Powers)

However, what it does here is just adding the stuff from Pname in the bottom of the array, making a large 1D array instead of adding new columns and making an ndarray.

I have also tried this with numpy.insert, numpy.hstack and numpy.concatenate and I tried changing the shape of Pname. Unfortunately, they all give me the same result.

If you want to append or otherwise concatenate on axis=1, you need to start with arrays that have at least 2 axes. Powers=np.zeros((32446,1)) — hpaulj
– hpaulj, Commented Dec 4, 2015 at 17:43

rayryeng · Accepted Answer · 2015-12-04 17:42:51Z

Have you tried numpy.column_stack?

Powers = np.column_stack([Powers,Pname])

However, the array is empty first, so make sure that the array isn't empty before concatenating or you will get a dimension mismatch error:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            if len(Powers) == 0:
                Powers = Pname[:,None]
            else:
                Powers = np.column_stack([Powers,Pname])
            np.savetxt("Powers.txt", Powers)

len(Powers) will check the amount of rows that exist in Powers. At the start, this should be 0 so at the first iteration, this is true and we will need to explicitly make Powers equal to a one column 2D array that consists of the first column in your file. Powers = Pname[:,None] will help you do this, which is the same as Powers = Pname[:,np.newaxis]. This transforms a 1D array into a 2D array with a singleton column. Now, the problem is that when you have 1D arrays in numpy, they are agnostic of whether they are rows or columns. Therefore, you must explicitly convert the arrays into columns before appending. numpy.column_stack takes care of that for you.

However, you'll also need to make sure that the Powers is a 2D matrix with one column the first time the loop iterates. Should you not want to use numpy.column_stack, you can still certainly use numpy.append, but make sure that what you're concatenating to the array is a column. The thing we talked about above should help you do this:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            if len(Powers) == 0:
                Powers = Pname[:,None]
            else:      
                Pname = Pname[:,None]
                Powers = np.append(Powers, Pname, axis=1) 
            np.savetxt("Powers.txt", Powers)

The second statement ensures that the array becomes a 2D array with a singleton column before concatenating.

Yes I did! Should have mentioned that one. It won't run, gives me the error: 'all the input array dimensions except for the concatenation axis must match exactly' The problem I think is that in first run Powers is empty, so it doesn't have the same dimensions as Pname. Thanks for the suggestion though!
@user90465 ah yeah now that makes sense. Try my edit. On the first iteration, we initialize Powers to be a singleton 2D array with 1 column,then we append from there.
What do you mean exactly by checking if the array is empty first? I tried your edit, but now I keep getting the message that dimensions don't match
@user90465 you can check if the array is empty by checking if the length is equal to 0, or if the number of rows is equal to 0. I've made some more edits as there were syntax errors. Please try the new edit.

Collectives™ on Stack Overflow

Updating a NumPy array by adding columns

1 Answer 1

4 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Related