Python deep get

Question

I'm implementing deep_get functionality to look inside arbitrarily nested Python 2.7 objects. Primarily for further logging.

This turned out to have surprising amount of quirks. Here's what I ended up with, would appreciate the feedback as I probably missed a few more things.

# coding=utf-8
from __future__ import unicode_literals
import collections

_default_stub = object()


def deep_get(obj, path, default=_default_stub, separator='.'):
    """Gets arbitrarily nested attribute or item value.

    Args:
        obj: Object to search in.
        path (str, hashable, iterable of hashables): Arbitrarily nested path in obj hierarchy.
        default: Default value. When provided it is returned if the path doesn't exist.
            Otherwise the call raises a LookupError.
        separator: String to split path by.

    Returns:
        Value at path.

    Raises:
        LookupError: If object at path doesn't exist.

    Examples:
        >>> deep_get({'a': 1}, 'a')
        1

        >>> deep_get({'a': 1}, 'b')
        LookupError: {'a': 1} has no element at 'b'

        >>> deep_get(['a', 'b', 'c'], -1)
        'c'

        >>> deep_get({'a': [{'b': [1, 2, 3]}, 'some string']}, 'a.0.b')
        [1, 2, 3]

        >>> class A(object):
        >>>     def __init__(self):
        >>>         self.x = self
        >>>         self.y = {'a': 10}
        >>>
        >>> deep_get(A(), 'x.x.x.x.x.x.y.a')
        10

        >>> deep_get({'a.b': {'c': 1}}, 'a.b.c')
        LookupError: {'a.b': {'c': 1}} has no element at 'a'

        >>> deep_get({'a.b': {'Привет': 1}}, ['a.b', 'Привет'])
        1

        >>> deep_get({'a.b': {'Привет': 1}}, 'a.b/Привет', separator='/')
        1

    """
    if isinstance(path, basestring):
        attributes = path.split(separator)
    elif isinstance(path, collections.Iterable):
        attributes = path
    else:
        attributes = [path]

    for i in attributes:
        try:
            success = False
            # 1. access as attr
            try:
                obj = getattr(obj, i)
                success = True
            except (AttributeError, TypeError, UnicodeEncodeError):
                # 2. access as dict index
                try:
                    obj = obj[i]
                    success = True
                except (TypeError, AttributeError, IndexError, KeyError):
                    # 3. access as list index
                    try:
                        obj = obj[int(i)]
                        success = True
                    except (TypeError, AttributeError, IndexError, KeyError,
                            UnicodeEncodeError, ValueError):
                        pass

            if not success:
                msg = "{obj} has no element at '{i}'".format(obj=obj, i=i)
                raise LookupError(msg.encode('utf8'))

        except Exception:
            if _default_stub != default:
                return default
            raise

    return obj

Update: Revised version with history https://gist.github.com/groz/f1838404d48971cc145609c226fdc6a2

Answer 1

Docstring

Your docstring is nice and descriptive. However, you have examples that look like doctests but they fail as doctests. One problem is that outputs contain Unicode strings, which must be written as u'abc'. Another problem is that expected exceptions have to be indicated like this. Also, the definition of class A needs to be written using ... continuation lines.

Implementation

Flag variables are usually a bad idea. You don't need success at all here.

You want to try three lookup methods until one succeeds. It could be done more elegantly using iteration rather than nesting.

# coding=utf-8
from __future__ import unicode_literals
import collections
import operator

_default_stub = object()


def deep_get(obj, path, default=_default_stub, separator='.'):
    """Gets arbitrarily nested attribute or item value.

    Args:
        obj: Object to search in.
        path (str, hashable, iterable of hashables): Arbitrarily nested path in obj hierarchy.
        default: Default value. When provided it is returned if the path doesn't exist.
            Otherwise the call raises a LookupError.
        separator: String to split path by.

    Returns:
        Value at path.

    Raises:
        LookupError: If object at path doesn't exist.

    Examples:
        >>> deep_get({'a': 1}, 'a')
        1

        >>> deep_get({'a': 1}, 'b')
        Traceback (most recent call last):
            ...
        LookupError: {u'a': 1} has no element at 'b'

        >>> deep_get(['a', 'b', 'c'], -1)
        u'c'

        >>> deep_get({'a': [{'b': [1, 2, 3]}, 'some string']}, 'a.0.b')
        [1, 2, 3]

        >>> class A(object):
        ...     def __init__(self):
        ...         self.x = self
        ...         self.y = {'a': 10}
        ...
        >>> deep_get(A(), 'x.x.x.x.x.x.y.a')
        10

        >>> deep_get({'a.b': {'c': 1}}, 'a.b.c')
        Traceback (most recent call last):
            ...
        LookupError: {u'a.b': {u'c': 1}} has no element at 'a'

        >>> deep_get({'a.b': {'Привет': 1}}, ['a.b', 'Привет'])
        1

        >>> deep_get({'a.b': {'Привет': 1}}, 'a.b/Привет', separator='/')
        1

    """
    if isinstance(path, basestring):
        attributes = path.split(separator)
    elif isinstance(path, collections.Iterable):
        attributes = path
    else:
        attributes = [path]

    LOOKUPS = [getattr, operator.getitem, lambda obj, i: obj[int(i)]]
    try:
        for i in attributes:
            for lookup in LOOKUPS:
                try:
                    obj = lookup(obj, i)
                    break
                except (TypeError, AttributeError, IndexError, KeyError,
                        UnicodeEncodeError, ValueError):
                    pass
            else:
                msg = "{obj} has no element at '{i}'".format(obj=obj, i=i)
                raise LookupError(msg.encode('utf8'))
    except Exception:
        if _default_stub != default:
            return default
        raise
    return obj

Consider requiring path to be an iterable that is already split by the caller.

Answer 2

Recursion. It would make the code smaller, easier to understand, way easier to implement as well. What you are trying to do is one of the tasks recursion is very good at.

How can I build a recursive function in python?

Basics of Recursion in Python