How do I save and restore multiple variables in python?

Question

I need to save about a dozen objects to a file and then restore them later. I've tried to use a for loop with pickle and shelve but it didn't work right.

Edit.
All of the objects that I was trying to save were in the same class (I should have mentioned this before), and I didn't realize that I could just save the whole class like this:

import pickle
def saveLoad(opt):
    global calc
    if opt == "save":
        f = file(filename, 'wb')
        pickle.dump(calc, f, 2)
        f.close
        print 'data saved'
    elif opt == "load":
        f = file(filename, 'rb')
        calc = pickle.load(f)
    else:
        print 'Invalid saveLoad option'

Answer 1

If you need to save multiple objects, you can simply put them in a single list, or tuple, for instance:

import pickle

# obj0, obj1, obj2 are created here...

# Saving the objects:
with open('objs.pkl', 'w') as f:  # Python 3: open(..., 'wb')
    pickle.dump([obj0, obj1, obj2], f)

# Getting back the objects:
with open('objs.pkl') as f:  # Python 3: open(..., 'rb')
    obj0, obj1, obj2 = pickle.load(f)

If you have a lot of data, you can reduce the file size by passing protocol=-1 to dump(); pickle will then use the best available protocol instead of the default historical (and more backward-compatible) protocol. In this case, the file must be opened in binary mode (wb and rb, respectively).

The binary mode should also be used with Python 3, as its default protocol produces binary (i.e. non-text) data (writing mode 'wb' and reading mode 'rb').

Answer 2

There is a built-in library called pickle. Using pickle you can dump objects to a file and load them later.

import pickle

f = open('store.pckl', 'wb')
pickle.dump(obj, f)
f.close()

f = open('store.pckl', 'rb')
obj = pickle.load(f)
f.close()

Answer 3

Another approach to saving multiple variables to a pickle file is:

import pickle

a = 3; b = [11,223,435];
pickle.dump([a,b], open("trial.p", "wb"))

c,d = pickle.load(open("trial.p","rb"))

print(c,d) ## To verify

Answer 4

You should look at the shelve and pickle modules. If you need to store a lot of data it may be better to use a database

Answer 5

The following approach seems simple and can be used with variables of different size:

import hickle as hkl
# write variables to filename [a,b,c can be of any size]
hkl.dump([a,b,c], filename)

# load variables from filename
a,b,c = hkl.load(filename)

Answer 6

You could use klepto, which provides persistent caching to memory, disk, or database.

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('foo.txt')
>>> db['1'] = 1
>>> db['max'] = max
>>> squared = lambda x: x**2
>>> db['squared'] = squared
>>> def add(x,y):
...   return x+y
... 
>>> db['add'] = add
>>> class Foo(object):
...   y = 1
...   def bar(self, x):
...     return self.y + x
... 
>>> db['Foo'] = Foo
>>> f = Foo()
>>> db['f'] = f  
>>> db.dump()
>>> 

Then, after interpreter restart...

dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from klepto.archives import file_archive
>>> db = file_archive('foo.txt')
>>> db
file_archive('foo.txt', {}, cached=True)
>>> db.load()
>>> db
file_archive('foo.txt', {'1': 1, 'add': <function add at 0x10610a0c8>, 'f': <__main__.Foo object at 0x10510ced0>, 'max': <built-in function max>, 'Foo': <class '__main__.Foo'>, 'squared': <function <lambda> at 0x10610a1b8>}, cached=True)
>>> db['add'](2,3)
5
>>> db['squared'](3)
9
>>> db['f'].bar(4)
5
>>> 

Get the code here: https://github.com/uqfoundation

Answer 7

Here is a solution using dill (https://dill.readthedocs.io/en/latest/dill.html) package. Pickle should work similarly.

""" Some objects to save """

import numpy as np
a = 6
b = 3.5
c = np.linspace(2,5,11)

""" Test the dump part """

import dill
file_name = 'dill_dump_test_file.dil'
list_of_variable_names = ('a', 'b', 'c')

with open(file_name,'wb') as file:
    dill.dump(list_of_variable_names, file)  # Store all the names first
    for variable_name in list_of_variable_names:
        dill.dump(eval(variable_name), file) # Store the objects themselves
        
#   Clear all the variables here     

""" Test the load part """

import dill
file_name = 'dill_dump_test_file.dil'
g = globals()

with open(file_name,'rb') as file:
    list_of_variable_names = dill.load(file)  # Get the names of stored objects
    for variable_name in list_of_variable_names:
        g[variable_name] = dill.load(file)    # Get the objects themselves
        

Answer 8

Preserve variable names

Most other answers rely on the list of variables during saving to be consistent with their list during loading. That might not be the case due to a bug/typo in that list or due to loading a file that was created with an older version of the code before that list changed.

A safer solution is to save the variable names as well. That way, the saved variable names are used instead of assuming what the variable names are.

import pickle
import inspect

def save_vars(filename, var_names):
    '''
    Save several variables to a pickle file
    such that their variable names can be recovered.
    Usage: `save_vars('variables.pkl', ['a', 'b'])`
    '''
    caller_vars = inspect.stack()[1].frame.f_locals
    saved_vars = {var_name: caller_vars[var_name] for var_name in var_names} # to skip missing ones, add `if var_name in caller_vars`
    with open(filename, 'wb') as f:
        pickle.dump(saved_vars, f)

def load_vars(filename):
    caller_vars = inspect.stack()[1].frame.f_locals
    with open(filename, 'rb') as f:
        saved_vars = pickle.load(f)
    caller_vars.update(saved_vars)

Example usage:

a, b = 1, 2
save_vars('variables.pkl', ['a', 'b'])
del a, b
load_vars('variables.pkl')

This can be extended to globals() variables if needed.

load_vars() can optionally be modified such that it loads only a user-specified subset of the variables or renames them.