Inplace operations for a class with numpy-style docs

Question

In pandas lot's of methods have the keyword argument inplace. This means if inplace=True, the called function will be performed on the object itself, and returns None, on the other hand if inplace=False the original object will be untouched, and the method is performed on the returned new instance. I've implemented it in my projects too, but I ended up with too much repetitive coding, so I decided to make a decorator which gets the job done. It also appends to the method's docstring, if it's formatting follows numpy-style docs. The only thing that needs to be added is return self to every method's end. I think that step also can be skipped with metaclasses, but for the time being it's not implemented.

import re
from functools import wraps
from copy import copy


_inplace_doc = """\n\tinplace : bool, optional
            Whether to apply the operation on the dataset in an "inplace" manner.
            This means if inplace is True it will apply the changes directly on
            the current object and returns None. If inplace is False, it will
            leave the current object untouched, but returns a copy of it, and
            the operation will be performed on the copy. It's useful when
            chaining operations on a dataset. See fluent interfacing and
            method cascading.\n\n\t"""


def _has_parameter_section(method):
    try:
        return "Parameters" in method.__doc__
    except TypeError:
        return False  # There's no docstring provided


def _build_doc(method, appendix):

    # finding sections (single words above 4+ slashes)
    patt = r"(\w+(?=\s*[-]{4,}[^/]))"

    splitted_doc = re.split(patt, method.__doc__)

    try:
        # We try to append to section below Parameters.
        target = splitted_doc.index("Parameters") + 1
    except ValueError:
        return method.__doc__

    splitted_doc[target] = splitted_doc[target].rstrip() + appendix

    return ''.join(_ for _ in splitted_doc if _ is not None)
   

def _update_doc(method, doc):
    if _has_parameter_section(method):
        newdoc = _build_doc(method, doc)
        method.__doc__ = newdoc
    else:
        newdoc = """\n\tParameters
        ----------""" + _inplace_doc

        nodoc_head = (f"Docstring automatically created for {method.__name__}. "
                      "Parameter list may not be complete.\n")
        if method.__doc__ is not None:
            method.__doc__ += newdoc
        else:
            method.__doc__ = nodoc_head + newdoc


def inplacify(method):
    """
    Decorator used to allow a function in a class to be called
    as `inplace`.
    """
    _update_doc(method, _inplace_doc)

    @wraps(method)
    def wrapper(self, *args, **kwds):
        inplace = kwds.pop("inplace", True)
        if inplace:
            method(self, *args, **kwds)
        else:
            return method(copy(self), *args, **kwds)
    return wrapper

Test cases:

class Dummy:
    def __init__(self, x):
        self.x = x

    @inplacify
    def increment(self, value):
        self.x += value
        return self

    @inplacify
    def multiply(self, value):
        """
        This is some numpy-style doc.

        Parameters
        ----------
        value : float
            The value ``x`` will be multiplied with it.

        Returns
        -------
        None..
        Just another section to see if it's working.
        """
        self.x *= value
        return self
    
    @inplacify
    def subtract(self, value):
        """
        This is some numpy-style doc. Without any
        sections after Parameters.

        Parameters
        ----------
        value : float
            The value to subtract from ``x``.
        """
        self.x -= value
        return self
    
if __name__ == "__main__":
    a = Dummy(1)
    a.increment(1)
    assert a.x == 2
    b = a.increment(2, inplace=False)
    assert a.x == 2
    assert b.x == 4
    print(help(a.increment))


    a.multiply(1)
    assert a.x == 2
    b = a.multiply(2, inplace=False)
    assert a.x == 2
    assert b.x == 4
    print(help(a.multiply))
    
    a.subtract(1)
    assert a.x == 1
    b = a.subtract(2, inplace=False)
    assert a.x == 1
    assert b.x == -1
    print(help(a.subtract))

I know there are few missing things: no type hints, no documentation for internal methods. Any criticism/improvement is appreciated.

Carcigenicate · Accepted Answer · 2020-09-07 15:27:17Z

Minor, but here:

''.join(_ for _ in splitted_doc if _ is not None)

_ should be reserved for when you need to bind a object to a name, but don't need the variable. Here is a good resource on the topic. You are in fact using _ though as both the final result and in the check, so I'd give it a proper name.

If none of the valid strings are falsey (empty), you can also use filter here with None as the function:

''.join(filter(None, splitted_doc))

This will remove all falsey elements; not just Nones. That shouldn't affect anything though since the "".join will effectively get rid of any empty strings anyway.

Also, not directly related to the code, but "splitted" doesn't sound idiomatic. I'd call it just split_doc.

I'd be careful about repeated concatenation of strings using +. Each concatenation requires making a copy of the accumulated string. Performance likely won't be a concern here, but I would consider changing this so it instead appends each piece to a list, then joins the list in the end to create the string all at once. This is Python's equivalent of the StringBuilder that some other languages have.

You can see from some quick benchmarking in Python 3.8 that it can make a fair difference:

from timeit import timeit

def repeated_concat(n):
    acc = ""
    for _ in range(n):
        acc = acc + "a"
    return acc

def repeated_concat_pe(n):
    acc = ""
    for _ in range(n):
        acc += "b"
    return acc

def join_concat_li(n):
    return "".join(["c" for _ in range(n)])

def join_concat_ge(n):
    return "".join("d" for _ in range(n))

N = int(3e5)
TRIALS = int(1e3)

print(timeit(lambda: repeated_concat(N), number=TRIALS))
print(timeit(lambda: repeated_concat_pe(N), number=TRIALS))
print(timeit(lambda: join_concat_li(N), number=TRIALS))
print(timeit(lambda: join_concat_ge(N), number=TRIALS))

40.2907047
40.597058800000006
13.207587099999998
19.380548700000006

Also, apparently list comprehensions can perform better than a generator expression when joining? I'm going to have to test that more.

I wasn't aware of the str concatenation performance, thank you for pointing that out. — Péter Leéh
– Péter Leéh, Commented Sep 7, 2020 at 14:53
@PéterLeéh Just for completeness, I added some quick timings for comparison. You'd want to test your exact scenario though to be sure. — Carcigenicate
– Carcigenicate, Commented Sep 7, 2020 at 15:30
I think that the difference between list comprehension and generator expression is somehow due to the fact that str.join puts it into a list right away if it is a generator. But I never quite got why this means such a drastic performance difference, especially since you include the list comprehension in the timing. — Graipher
– Graipher, Commented Sep 7, 2020 at 16:39
@Graipher Ahh, that could be it. I didn't expect join to force the iterable into a list first. It might skip that step if the argument is already a list, which means that if you use a gen expression, there's the overhead of the generator + the creation of a list. — Carcigenicate
– Carcigenicate, Commented Sep 7, 2020 at 16:45
@Graipher That does appear to be it. join first passes the iterable to a helper, and that helper does an immediate return if the iterable is a list, otherwise it forces it into a list — Carcigenicate
– Carcigenicate, Commented Sep 7, 2020 at 16:54

Stack Exchange Network

Inplace operations for a class with numpy-style docs

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Inplace operations for a class with numpy-style docs

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions