0

I need to populate a 2D array whose shape is 3xN, where N is initially unknown. The code looks as follows:

import numpy as np
import random

nruns = 5
all_data = [[]]
for run in range(nruns):
    n = random.randint(1,10)
    d1 = random.sample(range(0, 30), n)
    d2 = random.sample(range(0, 30), n)
    d3 = random.sample(range(0, 30), n)
    data_tmp = [d1, d2, d3]
    all_data = np.concatenate((all_data,data_tmp),axis=0)

This gives the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-103-22af8f04e7c0> in <module>
     10     d3 = random.sample(range(0, 30), n)
     11     data_tmp = [d1, d2, d3]
---> 12     all_data = np.concatenate((all_data,data_tmp),axis=0)
     13 print(np.shape(data_tmp))

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 0 and the array at index 1 has size 4

Is there a way to do this without pre-allocating all_data? Note that in my application, the data will not be random, but generated inside the loop.

Many thanks!

Improve this question
1
  • Do you understand what the error is saying the problem is? Commented Dec 30, 2020 at 0:38

3 Answers 3

Reset to default
2

You could store the data generated in each step of the for loop into a list and create the array when you are done.

In [298]: import numpy as np
     ...: import random

In [299]: nruns = 5
     ...: all_data = []

In [300]: for run in range(nruns):
     ...:     n = random.randint(1,10)
     ...:     d1 = random.sample(range(0, 30), n)
     ...:     d2 = random.sample(range(0, 30), n)
     ...:     d3 = random.sample(range(0, 30), n)
     ...:     all_data.append([d1, d2, d3])

In [301]: all_data = np.hstack(all_data)

In [302]: all_data
Out[302]: 
array([[13, 28, 14, 15, 11,  0,  0, 19,  6, 28, 14, 18,  1, 15,  4, 20,
         9, 14, 15, 13, 27, 28, 25,  5,  7,  4, 10, 22, 12,  6, 23, 15,
         0, 20, 14,  5, 13],
       [10,  9, 23,  4, 25, 28, 17, 14,  3,  4,  5,  9,  7, 18, 23,  9,
        14, 15, 25, 26, 29, 12, 21,  0,  5,  6, 11, 27, 13, 26, 22, 14,
         6,  5,  7, 23,  0],
       [13,  0,  7, 14, 29, 26, 12, 16, 13,  3,  9,  6, 11,  2, 19, 17,
        28, 14, 25, 24,  3, 12, 22,  7, 23, 18,  5, 14,  0, 14, 15,  8,
         3,  2, 26, 21, 16]])
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Perfect, this is exactly what I was looking for! It is also easy generalizable to cases where I have more than 3 columns of data. Thanks!
1

See if this is what you need, i.e. populate along axis 1 instead of 0.

import numpy as np
import random

nruns = 5
all_data = [[], [], []]
for run in range(nruns):
    n = random.randint(1,10)
    d1 = random.sample(range(0, 30), n)
    d2 = random.sample(range(0, 30), n)
    d3 = random.sample(range(0, 30), n)
    data_tmp = [d1, d2, d3]
    all_data = np.concatenate((all_data, data_tmp), axis=1)
Improve this answer

1 Comment

It'll be great to add some explanations to point what went wrong in PO.
1

How about using np.random only:

nruns = 5

# set seed for repeatability, remove for randomness
np.random.seed(42)

# randomize the lengths for the runs
num_samples = np.random.randint(1,10, nruns)

# sampling with the total length
all_data = np.random.randint(0,30, (3, num_samples.sum()))
# or, if `range(0,30)` represents some population
# all_data = np.random.choice(range(0,30), (3,num_samples.sum()) )
print(all_data)

Output:

[[25 18 22 10 10 23 20  3  7 23  2 21 20  1 23 11 29  5  1 27 20  0 11 25
  21 28 11 24 16 26 26]
 [ 9 27 27 15 14 29 29 14 29 18 11 22 19 24  2  4 18  6 20  8  6 17  3 24
  27 13 17 25  8 25 20]
 [ 1 19 27 14 27  6 11 28  7 14  2 13 16  3 17  7  3  1 29  5 21  9  3 21
  28 17 25 11  1  9 29]]
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.