1

Let's say I have a (simplified) class as below. I am using it for a program configuration (hyperparameters).

# config.py
class Config(object):      # default configuration
    GPU_COUNT = 1
    IMAGES_PER_GPU = 2
    MAP = {1:2, 2:3}

    def display(self):
        pass

# experiment1.py
from config import Config as Default
class Config(Default):     # some over-written configuration
    GPU_COUNT = 2
    NAME='2'            

# run.py
from experiment1 import Config
cfg = Config()
...
cfg.NAME = 'ABC'            # possible runtime over-writing

# Now I would like to save `cfg` at this moment

I'd like to save this configuration and restore later. The member functions must be out of concern when restoring.

1. When I tried pickle:

import pickle
with open('cfg.pk', 'rb') as f: cfg = pickle.load(f)

##--> AttributeError: Can't get attribute 'Config' on <module '__main__'>

I saw a solution using class_def of Config, but I wish I can restore the configuration without knowing the class definition (eg, export to dict and save as JSON)

2. I tried to convert class to dict (so that I can export as JSON)

cfg.__dict__     # {'NAME': 'ABC'}
vars(cfg)        # {'NAME': 'ABC'}

In both cases, it was difficult to access attributes. Is it possible?

Improve this question
6
  • Try importing Config into the file where your are trying to unpickle it. Commented May 30, 2018 at 21:41
  • I don't understand what you're asking here. Is your goal= to switch from Pickle to some kind of JSON-based storage, or is that just something you're trying to do to get around the problem that your class isn't pickleable out of the box? If you actually did want to use pickle, have you skimmed Pickling Class Instances in the docs? Or tried just dropping in dill in place of pickle to see if that works? Commented May 30, 2018 at 21:44
  • Anyway, cfg.__dict__ doesn't include attributes from the class(es). The dir function, or various functions in inspect, can get them. But then you need to filter out all kinds of things like methods and special attributes like __module__ and so on, and… that's all doable if you understand the guts of Python classes well enough to come up with and then implement a suitable rule, but not usually a path worth going down for serialization. Commented May 30, 2018 at 21:46
  • And really, you shouldn't need to store those attributes. When you unpickle the instance later, that creates a new Config instance, which is missing the same instance attributes, and therefore inherits the class attributes in the same way as the original, so it all just works—unless you're doing something uncommon and complicated, in which case you probably need to explain what that something is. Commented May 30, 2018 at 21:48
  • Do you want to convert a class to a dictionary, or an object? Commented May 30, 2018 at 21:48

2 Answers 2

Reset to default
4

The question's title is "how to convert python class to dict", but I suspect you are really just looking for an easy way to represent (hyper)parameters.

By far the easiest solution is to not use classes for this. I've seen it happen on some machine learning tutorials, but I consider it a pretty ugly hack. It breaks some semantics about classes vs objects, and the difficulty pickling is a result from that. How about you use a simple class like this one:

class Params(dict):
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

    def __getstate__(self):
        return self

    def __setstate__(self, state):
        self.update(state)

    def copy(self, **extra_params):
        return Params(**self, **extra_params)

It can do everything the class approach can. Predefined configs are then just objects you should copy before editing, as follows:

config = Params(
    GPU_COUNT = 2,
    NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4

Or alternatively in one step:

other_config = config.copy(
    GPU_COUNT = 4
)

Works fine with pickle (although you will need to have the Params class somewhere in your source), and you could also easily write load and save methods for the Params class if you want to use JSON.

In short, do not use a class for something that really is just an object.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Great!! Thanks. It looks so good. One question! When I tried config.copy(), an error returns: __class__ assignment only supported for heap types or ModuleType subclasses. How can I deal with this?
0

Thankfully, @evertheylen's answer was great to me. However, the code returns error when p.__class__ = Params, so I slightly changed as below. I think it works in the same way.

class Params(dict):
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

    def __getstate__(self):
        return self

    def __setstate__(self, state):
        self.update(state)

    def copy(self, **extra_params):
        lhs = Params()
        lhs.update(self)
        lhs.update(extra_params)
        return lhs

and you can do 

config = Params(
    GPU_COUNT = 2,
    NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4
Improve this answer

1 Comment

Good that it works for you :) I was about to do something similar: def copy(self, **extra_params): return Params(**self, **extra_params)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.