4
\$\begingroup\$

I am working with a data set that has many variables. Currently I am storing the data in many indented dictionaries in the following way:

import numpy as np

X_POSITIONS = [0,1.5,1]
Y_POSITIONS = [0,1,2]
CHANNELS = ['left pad', 'right pad', 'top pad', 'bottom pad']

data = {}
for x in X_POSITIONS:
    data[x] = {}
    for y in Y_POSITIONS:
        data[x][y] = {}
        for ch in CHANNELS:
            data[x][y][ch] = np.random.rand() # Here I would place my data.

This works fine, but it is cumbersome if by some reason I need to change the order of the keys. Consider the following function:

def do_something_with_single_channel(data_from_one_channel):
    for x in data_from_one_channel:
        for y in data_from_one_channel[x]:
            print(f'x = {x}, y = {y}, data[x][y] = {data_from_one_channel[x][y]}')

Before calling this function the whole data object has to be rearranged:

new_data = {}
for ch in CHANNELS:
    new_data[ch] = {}
    for x in X_POSITIONS:
        new_data[ch][x] = {}
        for y in Y_POSITIONS:
            new_data[ch][x][y] = data[x][y][ch]

do_something_with_single_channel(new_data['left pad'])

Since this seems to be a very common thing to do, I am sure that there must already exist something better than dictionaries for this purpose. Ideally I imagine something that can be accessed in the same way as the arguments of a function by giving names to the variables and forgetting about the order, for example something of the form

do_something_with_single_channel(data[channel='left pad'])

Does something like this exists? What's the name?

\$\endgroup\$
0

1 Answer 1

Reset to default
5
\$\begingroup\$

Consider a nested dictionary comprehension:

data = {x:{y:{ch: np.random.rand()
          for ch in CHANNELS}
          for y in Y_POSITIONS}
          for x in X_POSITIONS}

pprint.pprint(data)

{0: {0: {'bottom pad': 0.4563182938806024,
         'left pad': 0.7109389987303294,
         'right pad': 0.04972926343584316,
         'top pad': 0.49018200203439044},
     1: {'bottom pad': 0.9197368747212471,
         'left pad': 0.49675239597387033,
         'right pad': 0.8846734838851381,
         'top pad': 0.2536908682927299},
     2: {'bottom pad': 0.19202917332705682,
         'left pad': 0.4680743827356374,
         'right pad': 0.9824888617543756,
         'top pad': 0.7871922090543111}},
 1: {0: {'bottom pad': 0.532524614137474,
         'left pad': 0.5500941768186839,
         'right pad': 0.046363378683273115,
         'top pad': 0.507924966038481},
     1: {'bottom pad': 0.18606527132423667,
         'left pad': 0.2926470569818338,
         'right pad': 0.4542221348696881,
         'top pad': 0.07304292627461106},
     2: {'bottom pad': 0.255962925458759,
         'left pad': 0.8206558157675303,
         'right pad': 0.028156806394849743,
         'top pad': 0.4617476628686388}},
 1.5: {0: {'bottom pad': 0.5537703566924752,
           'left pad': 0.14192043274483335,
           'right pad': 0.04030969407542717,
           'top pad': 0.4145838174513119},
       1: {'bottom pad': 0.10519991606894175,
           'left pad': 0.33471726599841756,
           'right pad': 0.10389744180143101,
           'top pad': 0.3927574328293768},
       2: {'bottom pad': 0.2950323578101469,
           'left pad': 0.9335998766267041,
           'right pad': 0.9337763647877098,
           'top pad': 0.6591832120994695}}}

However, to retain data of disparate types for later analyses, use pandas Data Frames which are essentially equal length Pandas Series (or 1-D Numpy arrays). And using itertools.product, you can return all possible combinations from multiple iterables:

from itertools import product 

import numpy as np
import pandas as pd

data = list(product(X_POSITIONS, Y_POSITIONS, CHANNELS))

# CAST LIST OF VALUES INTO DATA FRAME AND ASSIGN COLUMN OF RANDOM NUMs
df = (pd.DataFrame(data, columns=['X_POSITIONS', 'Y_POSITIONS', 'CHANNELS'])
        .assign(value = np.random.rand(len(data), 1)))

Output (N=36 for 3 * 3 * 4 product)

print(df)

#     X_POSITIONS  Y_POSITIONS    CHANNELS     value
# 0           0.0            0    left pad  0.761372
# 1           0.0            0   right pad  0.440973
# 2           0.0            0     top pad  0.182679
# 3           0.0            0  bottom pad  0.564203
# 4           0.0            1    left pad  0.954728
# 5           0.0            1   right pad  0.539686
# 6           0.0            1     top pad  0.957724
# 7           0.0            1  bottom pad  0.232217
# 8           0.0            2    left pad  0.488761
# 9           0.0            2   right pad  0.883579
# 10          0.0            2     top pad  0.010666
# 11          0.0            2  bottom pad  0.022114
# 12          1.5            0    left pad  0.129402
# 13          1.5            0   right pad  0.763472
# 14          1.5            0     top pad  0.475217
# 15          1.5            0  bottom pad  0.160637
# 16          1.5            1    left pad  0.521797
# 17          1.5            1   right pad  0.865391
# 18          1.5            1     top pad  0.263130
# 19          1.5            1  bottom pad  0.576295
# 20          1.5            2    left pad  0.004636
# 21          1.5            2   right pad  0.137856
# 22          1.5            2     top pad  0.156635
# 23          1.5            2  bottom pad  0.198684
# 24          1.0            0    left pad  0.143598
# 25          1.0            0   right pad  0.660144
# 26          1.0            0     top pad  0.588416
# 27          1.0            0  bottom pad  0.294899
# 28          1.0            1    left pad  0.915973
# 29          1.0            1   right pad  0.348533
# 30          1.0            1     top pad  0.391135
# 31          1.0            1  bottom pad  0.951016
# 32          1.0            2    left pad  0.015479
# 33          1.0            2   right pad  0.719314
# 34          1.0            2     top pad  0.976324
# 35          1.0            2  bottom pad  0.191481

To subset data frame by your indicator values:

print(df[df['CHANNELS'] == 'right pad'])

#     X_POSITIONS  Y_POSITIONS   CHANNELS     value
# 1           0.0            0  right pad  0.988888
# 5           0.0            1  right pad  0.091176
# 9           0.0            2  right pad  0.334674
# 13          1.5            0  right pad  0.706215
# 17          1.5            1  right pad  0.032422
# 21          1.5            2  right pad  0.024871
# 25          1.0            0  right pad  0.554525
# 29          1.0            1  right pad  0.790112
# 33          1.0            2  right pad  0.650198
\$\endgroup\$
4
  • \$\begingroup\$ Thanks, this looks better with data frames! \$\endgroup\$ Commented Jan 8, 2021 at 22:09
  • \$\begingroup\$ I have implemented my code with pandas.DataFrame and it works, but accessing single elements is incredibly slow. My df has about 1.5e6 rows and later I need to group the elements according to the values of some of the columns, for example df[(df['x']=1)&(df['y']=2)&(df['channel']='left pad')]. I need to do this for each value of x and y and channel and this takes a considerable amount of time, with the indented dictionaries implementation is super fast (however a bit too rigid). \$\endgroup\$ Commented Jan 9, 2021 at 22:26
  • \$\begingroup\$ According to a time measurement using time.time() with the data frame it takes about 150 s while with the dictionaries it takes 20 s. \$\endgroup\$ Commented Jan 9, 2021 at 22:49
  • \$\begingroup\$ There are many pandas methods for calculations. Pandas works best in sets not scalar values, especially vectorized calculations. It is not clear what exactly you need to do. Look into groupby to run calculations across all groups: df.groupby(['x','y','channel'])['value'].agg(['sum','mean','median','min','max']). \$\endgroup\$ Commented Jan 10, 2021 at 6:10

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.