Combining two numpy arrays

Question

I have a numpy array ar1 with dimensions (1196, 14, 64, 1).

I have a numpy array ar2 with dimensions (1196,).

I want to combine these two because I want to shuffle them later. I thought that this was the kind of thing that zip was made for, but when I zip them together:

ar3 = np.asarray(zip(ar1,ar2))

then print(ar3.shape) gives ()

I have also tried np.concatenate but apparently all the input array dimensions for the concatenation axis must match exactly.

I tried np.hstack but I got all the input arrays must have same number of dimensions

How can I combine these two arrays that have the same size along axis 0? Perhaps I don't need to combine then and should just shuffle them separately using the same indices.

(I already actually combined and saved these arrays using numpy.savez, but when I load this file into my code, I assume that I have to separate them first and then recombine the in an array as I am trying to do. If I could just pull out a combines array from the .npz file then that would be even better)

Péter Leéh · Accepted Answer · 2020-08-17 15:19:40Z

1

Use np.concatenate along with np.broadcast_to, because concatenate needs all the arrays to have the same number of dimensions. This should do the trick:

x = np.ones((1196, 14, 64, 1))
y = np.arange(1196)
output = np.concatenate((x, np.broadcast_to(y[:, None, None, None], x.shape[:-1] + (1,))), axis=-1)
# output.shape --> (1196, 14, 64, 2)

answered Aug 17, 2020 at 15:19

Péter Leéh

2,1092 gold badges12 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hpaulj · Accepted Answer · 2020-08-17 17:28:19Z

In [111]: ar1 = np.ones((1196,14,62,1))                                                              
In [112]: ar2 = np.zeros((1196))

In py3, zip is generator-like. You have expand it with list to get an array:

In [113]: np.array(zip(ar1,ar2))                                                                     
Out[113]: array(<zip object at 0x7f188a465a48>, dtype=object)

That's a single element array, with shape ().

If you expand the zip, and make an array, the result is a object dtype array:

In [119]: A = np.array(list(zip(ar1,ar2)))                                                           
/usr/local/bin/ipython3:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  #!/usr/bin/python3
In [120]: A.shape                                                                                    
Out[120]: (1196, 2)
In [121]: A.dtype                                                                                    
Out[121]: dtype('O')
In [122]: A[0,0].shape                                                                               
Out[122]: (14, 62, 1)
In [123]: A[0,1].shape                                                                               
Out[123]: ()

One column is the (14,62,1) arrays, the other the ar2 values.

The other answer suggests expanding ar2 to match ar1 in shape, and concatenating on the last axis. The result is twice the size of ar1. broadcast_to does a 'virtual' replication (no increase in memory), but that doesn't carry over in the concatenate. You get 14*62 copies of every ar2 value.

But ar2 could be broadcast to (1196,1,62,1) and concatenate on axis 1 (a 62x replication), or (1196,14,1,1) and axis 2 concatenate (14x). To concatenate, the arrays have to match on all but one axis.

But for savez and load you don't need to concatenate the arrays. You can save and load them separately. savez puts them in separate npy files. savez shows how to load each array.

And you can shuffle them separately.

Make a shuffling index:

In [124]: idx = np.arange(1196)                                                                      
In [125]: np.random.shuffle(idx)                                                                     
In [126]: idx[:10]                                                                                   
Out[126]: array([ 561,  980,   42,   98, 1055,  375,   13,  771,  832,  787])

and apply it to each array:

In [127]: ar1[idx,:,:,:];                                                                            
In [128]: ar2[idx];

Collectives™ on Stack Overflow

Combining two numpy arrays

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related