ValueError when apply function to data frame in python-pandas

Question

Let's say I have a very simple data frame:

import pandas as pd
df = pd.DataFrame(np.full((6), 1))

Now I am going to define a function that generate a numpy array with random length and add the given value to the tail:

import numpy as np
def func(row):
    l = np.full((np.random.random_integer(5)), 1)
    return np.hstack(l, row)

When I try to apply the function to df to get a 2-D array:

df.apply(func, axis=1),

I got such an error:

ValueError: Shape of passed values is (6, 2), indices imply (6, 1)

Do you know what is the problem and how to fix it? Thank you in advance!

Answer 1

Firstly you want np.random.random_integers, secondly hstack takes a tuple so pass a tuple, thirdly you need to return something that it can align with so in this case a Series:

In [213]:
df = pd.DataFrame(np.full((6), 1))
def func(row):
    l = np.full((np.random.random_integers(5)), 1)
    return pd.Series(np.hstack((l, row)))

In [214]:    
df.apply(func, axis=1)

Out[214]:
     0    1    2    3    4    5
0  1.0  1.0  1.0  NaN  NaN  NaN
1  1.0  1.0  NaN  NaN  NaN  NaN
2  1.0  1.0  NaN  NaN  NaN  NaN
3  1.0  1.0  1.0  NaN  NaN  NaN
4  1.0  1.0  1.0  1.0  1.0  NaN
5  1.0  1.0  1.0  1.0  1.0  1.0

Note that I get a ton of warnings about the above:

C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(3, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(2, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(1, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(4, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(5, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)

To get a np array from a df call attribute values:

df.apply(func, axis=1).values