ast.literal_eval(...)

If we can remove your calls to ast.literal_eval(...) we should see a nice reduction in the run time of your loops. But, why can we remove this you ask? Consider:

    m = '[0, 1, 2, ... , 9,999]' # a str representation of list w/ 10k elements, 0-9999
    n = '[0, 1, 2]'

    x = ast.literal.eval(m)
    y = ast.literal.eval(n)

    x == range(10000) # true

As you can see from the snippet above, ast.literal_eval(...) will parse and evaluate whatever string you pass it, and return a literal representation of that string (assuming of course that the string represents a valid literal). Clearly, it is more efficient to compare m and n than it is to compare x and y. Also, it doesn't appear that you are concerned with whether or not val or val_2 is a valid python literal because under the scenario that ast.literal_eval(...)throws an exception, you default to just comparing the strings returned by getter(val) and getter(val_2). Long story short you can remove the try: / except: and just use the statements you have under the except clause.

`for key in val_2.keys()` -- The above loop occurs as the inner-most loop of both loops `1` and `2`. With each iteration you check that `key` isn't equivalent to 7 other possible key values. 6 of these key values occur in the data you've presented and the 7th (`condition`) doesn't. It should be more efficient to replace: for key in val_2.keys(): if(key != "block" and key != "username" and key != "setid" and key != "setid_hash" and key != "predecessor" and key != "time_string" and key != "condition"): val[key]=val_2[key] with: # put this at the top of the test function x_keys = set(['block', 'username', 'setid', 'setid_hash', 'predecessor', 'time_string', 'condition']) # ... for key in set(val_2.keys()) - x_keys: val[key] = val_2[key]