Update YAML file programmatically

Question

I've a Python dict that comes from reading a YAML file with the usual

yaml.load(stream)

I'd like to update the YAML file programmatically given a path to be updated like:

group1,option1,option11,value

and save the resulting dict again as a yaml file. I'm facing the problem of updating a dicntionary, taking into account that the path is dynamic (let's say a user is able to enter the path through a simple CLI I've created using Cmd).

Any ideas?

thanks!

UPDATE Let me be more specific on the question: The issue is with updating part of a dictionary where I do not know in advance the structure. I'm working on a project where all the configuration is stored on YAML files, and I want to add a CLI to avoid having to edit them by hand. This a sample YAML file, loaded to a dictionary (config-dict) using PyYaml:

config:
 a-function: enable
 b-function: disable
 firewall:
  NET:
   A:
    uplink: enable
    downlink: enable
   B:
    uplink: enable
    downlink: enable
  subscriber-filter:
   cancellation-timer: 180
 service:
  copy:
   DS: enable
  remark:
   header-remark:
    DSC: enable
    remark-table:
 port:
  linkup-debounce: 300
  p0:
   mode: amode
  p1:
   mode: bmode
  p2:
   mode: amode
  p3:
   mode: bmode

I've created the CLI with Cmd, and it's working great even with autocompletion. The user may provide a line like:

config port p1 mode amode

So, I need to edit:

config-dict['config']['port']['p1']['mode'] and set it to 'amode'. Then, use yaml.dump() to create the file again. Another possible line would be:

config a-function enable

So config-dict['config']['a-function'] has to be set to 'enable'.

My problem is when updating the dictionary. If Python passed values as a reference would be easy: Just iterate through the dict until the right value is found and save it. Actually this is what I'm doing for the Cmd autocomplete. But I don't know how to do the update.

Hope I explained myself better now!

Thanks in advance.

Answer 1

In fact the solution follows simple patter: load - modify - dump:

Before playing, be sure you have pyyaml installed:

$ pip install pyyaml

testyaml.py

import yaml
fname = "data.yaml"

dct = {"Jan": {"score": 3, "city": "Karvina"}, "David": {"score": 33, "city": "Brno"}}

with open(fname, "w") as f:
    yaml.dump(dct, f)

with open(fname) as f:
    newdct = yaml.load(f)

print newdct
newdct["Pipi"] = {"score": 1000000, "city": "Stockholm"}

with open(fname, "w") as f:
    yaml.dump(newdct, f)

Resulting data.yaml

$ cat data.yaml
David: {city: Brno, score: 33}
Jan: {city: Karvina, score: 3}
Pipi: {city: Stockholm, score: 1000000}

Answer 2

It is very simple to do it with python-benedict, a solid python dict subclass that support IO operations with many formats, including yaml.

Installation: pip install python-benedict

You can initialize it directly from the yaml file:

from benedict import benedict

f = 'data.yaml'
d = benedict.from_yaml(f)
d['Pipi'] = {'score': 1000000, 'city': 'Stockholm'}

# benedict supports keypath (dot syntax by default),
# so it's possible to update nested values easily:
d['Pipi.score'] = 2000000
print(d['Pipi']) # -> {'score': 2000000, 'city': 'Stockholm'}

d.to_yaml(filepath=f)

Here the library repository and the documentation: https://github.com/fabiocaccamo/python-benedict

Note: I am the author of this project

Answer 3

Updating seems to the one place where pyyaml falls short. You cannot even use yaml.load on a file that was opened in (a)ppend mode without an exception. Now this may be a bit tedious for complex dictionaries but if each added item represents a separate case or document you could handle it as if it were any other text file.

newinfo = {"Pipi": {"score": 100000, "city": "Stockholm"}}
with open(fname, "a") as f:
     sep = "\n" # For distinct documents use "\n...\n" as separator

     # Pay attention to where you put the separator. 
     # if the file exists and is in traditional format place at 
     # beginning of string. else place at the end.

     infostring = "{}".format(newinfo)
     f.write(infostring + sep)

While this doesn't necessarily help with value updating, it does allow for file updating. You might also look into using json.dump on the file. I know it is in YAML but the formats are largely compatible unless you are using the python-object storage feature in YAML.

For a OS agnostic approach to carriage character assignment remember to use os.linesep.

Best of luck. Hope this helps.

Answer 4

Try out this method, i am using for updating yaml or json files. def update_dictionary_recursively(dictionary, key, value, key_separator="."): """Update givendictionarywith the givenkeyandvalue`.

if dictionary contains value as dict E.g. {key1:value1, key2:{key3, {key4:value4}}} and you have to
update key4 then `key` should be given as `key2.key3.key4`.

If dictionary contains value as list E.g. {key1:{key2:[{key3:valie3}, {key4:value4}]}} and you have to update
key4 then `key` should be given as `key1.key2[1].key4`.

:param dictionary: Dictionary that is to be updated.
:type dictionary: dict
:param key: Key with which the dictionary is to be updated.
:type key: str
:param value: The value which will be used to update the key in dictionary.
:type value: object
:param key_separator: Separator with which key is separated.
:type key_separator str
:return: Return updated dictionary.
:rtype: dict
"""
index = key.find(key_separator)
if index != -1:
    current_key = key[0:index]
    key = key[index + 1:]
    try:
        if '[' in current_key:
            key_index = current_key.split('[')
            current_key = key_index[0]
            list_index = int(key_index[1].strip(']'))
            dictionary[current_key][list_index] = update_dictionary_recursively(
                dictionary[current_key][list_index], key, value, key_separator)
        else:
            dictionary[current_key] = update_dictionary_recursively(dictionary[current_key],
                                                                                    key, value, key_separator)
    except (KeyError, IndexError):
        return dictionary
else:
    if '[' in key:
        key_index = key.split('[')
        list_index = int(key_index[1].strip(']'))
        if list_index > len(dictionary) - 1:
            return dictionary
        dictionary[list_index] = value
    else:
        if key not in dictionary:
            return dictionary
        dictionary[key] = value
return dictionary

`