Removing nulls and empty objects of mixed data types from a dictionary with a nested structure

Question

Off the back of a similar question I asked, how would one go about cleaning a dictionary containing a variety of datatypes: nulls, empty lists, empty dicts etc, at varying levels of nesting E.g.

{
  "key":"value",
  "key1": {},
  "key2": [],
  "key3": True,
  "key4": False,
  "key5": None,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue", 
      "subdictkey1": {},
      "subdictkey2": [],
      "subdictkey3": None
    }
  }
}

Becomes:

{
  "key":"value",
  "key3": True,
  "key4": False,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue"
    }
  }
}

The solution should be good for n levels of nesting (not just 1 level). Obviously I want to avoid nested loops (particularly as n could equal 3 or 4), is the only solution flattening the structure? Is there a more elegant way of going about it?

Edit: Building on @Ch3steR answer and accounting for the issue I encountered with a list containing a null, this is the final working function:

def recur(n_dict,new_d={}):
    global counter
    for key,val in n_dict.items():
        if val or isinstance(val,bool) or (isinstance(val,list) and any(elem is not None for elem in val)):
                if (isinstance(val,list) and any(elem is None for elem in val)):
                   counter=counter+1
                else:
                    new_d={**new_d,**{key:val}}
                    if isinstance(val,dict):
                        new_d[key]=recur(val)

    return new_d

You asked the same question yesterday. stackoverflow.com/questions/60381477/… Just apply the same logic using recursion. — Ch3steR
– Ch3steR, Commented Feb 25, 2020 at 12:24
Does this answer your question? Removing nulls and empty objects of mixed data types from a dictionary — Ch3steR
– Ch3steR, Commented Feb 25, 2020 at 12:25
I thought the addition of the structural component was a substantial enough change that it warranted another question, should I just update the previous one? — AJG
– AJG, Commented Feb 25, 2020 at 12:29
It would be better to incorporate the solutions from the previous question into answering your current question. When you have n-level nested dictionary use Recursion. — Ch3steR
– Ch3steR, Commented Feb 25, 2020 at 12:32
The answer you accepted in that question is not pythonic. And to check if list,tuple,dict is empty use if data_structure not if len(data_structure)==0 like proposed in the accepted answer of your previous. — Ch3steR
– Ch3steR, Commented Feb 25, 2020 at 12:34

Ch3steR · Accepted Answer · 2020-02-25 13:37:26Z

2

You can use Recursion when you are dealing with an arbitrarily nested dictionary.

Try this.

def recur(n_dict,new_d={}):
    for key,val in n_dict.items():
        if val or isinstance(val,bool):
            new_d={**new_d,**{key:val}}
            if isinstance(val,dict):
                new_d[key]=recur(val)
    return new_d

a={
  "key":"value",
  "key1": {},
  "key2": [],
  "key3": True,
  "key4": False,
  "key5": None,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue", 
      "subdictkey1": {},
      "subdictkey2": [],
      "subdictkey3": None
    }
  }
}

print(recur(a))
{'key': 'value',
 'key3': True,
 'key4': False,
 'key6': [1, 2, 3],
 'key7': {'subkey': 'subvalue'},
 'key8': {'subdict': {'subdictkey': 'subdictvalue'}}}

recur(n_dict,new_d={}) uses mutable default argument.

Note:

Never mutate new_d in-place or you will encounter this problem.

One of the way to check if your default argument is changed is use __defaults__

>>>recur(a)
>>>recur.__defaults__
({},) 
>>>recur(a)
>>>recur.__defaults__
({},)

edited Feb 25, 2020 at 13:37

answered Feb 25, 2020 at 13:31

Ch3steR

20.8k4 gold badges32 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

AJG Over a year ago

This is great thanks. Would you mind explaining what the 4th line of your function is doing? Also how would one add in an extra condition to remove lists that contain a single null e.g. [None]. I tried adding an extra condition on the end of line 3 of your function or (isinstance(val,list) and any(elem is not None for elem in val) with no luck

Ch3steR Over a year ago

@AJG You need to tweak it a little bit. Now in the present function you recur when val is dictionary. Now you need to check if the val is an Iterable but not a string. Implement this into the above function. If fail to implement this comment below. I'll add the answer again.

Ch3steR Over a year ago

@AJG It's really hard to read code from comments. Did it work?

AJG Over a year ago

I updated the question with the final working code, if you have any final pointers?

Collectives™ on Stack Overflow

Removing nulls and empty objects of mixed data types from a dictionary with a nested structure

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related