pdoc3 references internal symbols instead of public ones #228

CarstenLeue · 2020-07-01T10:52:34Z

I am keeping my python implementation details in a package called _internal and export the methods, classes and variables that are supposed to be the public API by re-exporting them in the __init__.py of the parent package, e.g. like so:

from ._internal.interface import MyInterface
from ._internal.sample import MY_VAR, my_function

__all__ = ['MyInterface', 'MY_VAR', 'my_function']

the signature e.g. of my_function is:

def my_function(iface: MyInterface) -> str:
    """My function doc

    Args:
        iface: interface parameter doc

    Returns:
        return value doc
    """    

   ...

Expected Behavior

all three entities from __all__ to be visible in the documentation
the documentation of my_function to reference the public type MyInterface
the documentation of the arguments of my_function to show type information
ideally a link from the type MyInterface in the signature of my_function to the documentation of MyInterface

Actual Behavior

the variable MY_VAR is not documented at all
the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str
the documentation of the arguments and the return value of my_function does not show any type hint

Steps to Reproduce

I have create a small repo to reproduce at:
https://github.com/Carsten-Leue/pdoc-test

Generated documentation showing the issue is here:
https://carsten-leue.github.io/pdoc-test/pdoc_test/

Additional info

pdoc version:
0.8.3

The text was updated successfully, but these errors were encountered:

kernc · 2020-07-04T00:53:38Z

Thanks for these issues!

the variable MY_VAR is not documented at all

MY_VAR is neither function nor class and it doesn't have a PEP 224 variable docstring, so the branch skips it. I guess we might amend the branch to check for inclusion in __all__ too, albeit sans any docstring.

pdoc/pdoc/__init__.py

Lines 622 to 628 in 04960e4

    
           for name, obj in public_objs: 
        
               if _is_function(obj): 
        
                   self.doc[name] = Function(name, self, obj) 
        
               elif inspect.isclass(obj): 
        
                   self.doc[name] = Class(name, self, obj) 
        
               elif name in var_docstrings: 
        
                   self.doc[name] = Variable(name, self, var_docstrings[name], obj=obj)

the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str

Duplicate of #231. Not quite sure how to tackle this one.

he documentation of the arguments and the return value of my_function does not show any type hint

Duplicate of #112. Interpolating literal docstring strings with inferred types will be clumsy at best. If someone wants to try to tackle it, sure, but it likely won't be me. 😅

Carsten-Leue · 2020-07-05T12:18:28Z

MY_VAR is neither function nor class and it doesn't have a PEP 224 variable docstring, so the branch skips it. I guess we might amend the branch to check for inclusion in __all__ too, albeit sans any docstring.

The source of MY_VAR is:

MY_VAR = 10
"""This is my variable"""

Isn't this string a PEP224 docstring for this variable?

kernc · 2020-07-05T12:22:51Z

It is. But that's in the module _internal, I suppose, not in the module where __all__ is defined and MY_VAR exported ...

Carsten-Leue · 2020-07-05T12:33:36Z

It is. But that's in the module _internal, I suppose, not in the module where all is defined and MY_VAR exported ...

True, but the same is true for the classes and functions in the sample as well. And those are being documented. Is there a conceptual reason why variables are handled differently?

Carsten-Leue · 2020-07-05T12:41:43Z

the signature of my_function shows the internal type name not the public one (it shows def my_function (iface: pdoc_test._internal.interface.MyInterface) ‑> str) instead of def my_function(iface: pdoc_test.MyInterface) ‑> str
Duplicate of #231. Not quite sure how to tackle this one.

If I change the source to this:

def my_function(iface: MyInterface) -> str:
    """My function doc



    Args:
        iface (MyInterface): interface parameter doc

    Returns:
        return value doc
    """    

    return ''

e.g. include type information into the argument explicitly (iface (MyInterface)) then I find that the type in the function declaration is being documented using the internal type, but the type in the docstring is being documented using the API level type, even including the correct hyperlink to the documentation of MyInterface.
Why this difference, wouldn't it be possible to use the "docstring" behaviour both for the type in the declaration and in the docstring?

Example of the generated HTML:

<dl>
<dt id="pdoc_test.my_function"><code class="name flex">
<span>def <span class="ident">my_function</span></span>(<span>iface:&nbsp;pdoc_test._internal.interface.MyInterface) ‑&gt;&nbsp;str</span>
</code></dt>
<dd>
<div class="desc"><p>My function doc</p>
<h2 id="args">Args</h2>
<dl>
<dt><strong><code>iface</code></strong> : <code><a title="pdoc_test.MyInterface" href="#pdoc_test.MyInterface">MyInterface</a></code></dt>
<dd>interface parameter doc</dd>
</dl>
<h2 id="returns">Returns</h2>
<p>return value doc</p></div>
<details class="source">
<summary>
<span>Expand source code</span>
</summary>
<pre><code class="python hljs"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">my_function</span><span class="hljs-params">(iface: MyInterface)</span> -&gt; str:</span>
    <span class="hljs-string">"""My function doc



    Args:
        iface (MyInterface): interface parameter doc

    Returns:
        return value doc
    """</span>    

    <span class="hljs-keyword">return</span> <span class="hljs-string">''</span></code></pre>
</details>
</dd>
</dl>

<a title="pdoc_test.MyInterface" href="#pdoc_test.MyInterface">MyInterface</a>

kernc · 2020-07-05T13:07:24Z

Is there a conceptual reason why variables are handled differently?

Yeah, there's no way to access PEP224 docstrings at runtime (i.e. via __doc__), so we parse source code ASTs instead.

pdoc/pdoc/__init__.py

Lines 220 to 230 in 04960e4

    
           def _pep224_docstrings(doc_obj: Union['Module', 'Class'], *, 
        
                                  _init_tree=None) -> Tuple[Dict[str, str], 
        
                                                            Dict[str, str]]: 
        
               """ 
        
               Extracts PEP-224 docstrings for variables of `doc_obj` 
        
               (either a `pdoc.Module` or `pdoc.Class`). 
        
               Returns a tuple of two dicts mapping variable names to their docstrings. 
        
               The second dict contains instance variables and is non-empty only in case 
        
               `doc_obj` is a `pdoc.Class` which has `__init__` method. 
        
               """

Likewise with the type annotations in functions parameters vs. annotations in docstrings — two different code paths. Type strings in docstrings are not modified, just queried, thus making MyInterface resolve to the available module-local definition.

Not at all saying it's all good and perfect, but it's what we currently have.

Carsten-Leue · 2020-07-05T19:42:35Z

Type strings in docstrings are not modified, just queried, thus making MyInterface resolve to the available module-local definition.

I do not fully understand this comment. MyInterface defined in pdoc_test\_internal\sample.py and formally referenced as the same string in the docstring and the argument list. There is no "module-local" definition.

kernc · 2020-07-06T02:33:35Z

Should have rather said the module-local variable, the one present in the module, inferred from __all__, and available in this module's .doc.

pdoc/pdoc/__init__.py

Lines 591 to 592 in 04960e4

    
           self.doc = {}  # type: Dict[str, Union[Module, Class, Function, Variable]] 
        
           """A mapping from identifier name to a documentation object."""

Function params with linked annotations are constructed in one way:

pdoc/pdoc/__init__.py

Lines 1312 to 1318 in 04960e4

    
           if p.annotation is not EMPTY: 
        
               annotation = inspect.formatannotation(p.annotation).replace(' ', '\N{NBSP}') 
        
               # "Eval" forward-declarations (typing string literals) 
        
               if isinstance(p.annotation, str): 
        
                   annotation = annotation.strip("'") 
        
               if link: 
        
                   annotation = re.sub(r'[\w\.]+', _linkify, annotation)

and types in docstrings (any backtick-quoted identifier references, really; not only Args section types) in quite another:

pdoc/pdoc/html_helpers.py

Lines 465 to 477 in 04960e4

    
           if module and link: 
        
               # Hyperlink markdown code spans not within markdown hyperlinks. 
        
               # E.g. `code` yes, but not [`code`](...). RE adapted from: 
        
               # https://github.com/Python-Markdown/markdown/blob/ada40c66/markdown/inlinepatterns.py#L106 
        
               # Also avoid linking triple-backticked arg names in deflists. 
        
               linkify = partial(_linkify, link=link, module=module, wrap_code=True) 
        
               text = re.sub(r'(?P<inside_link>\[[^\]]*?)?' 
        
                             r'(?:(?<!\\)(?:\\{2})+(?=`)|(?<!\\)(?P<fence>`+)' 
        
                             r'(?P<code>.+?)(?<!`)' 
        
                             r'(?P=fence)(?!`))', 
        
                             lambda m: (m.group() 
        
                                        if m.group('inside_link') or len(m.group('fence')) > 2 
        
                                        else linkify(m)), text)

Better ideas welcome.

CarstenLeue changed the title ~~pdoc references internal symbols instead of public ones~~ pdoc3 references internal symbols instead of public ones Jul 1, 2020

kernc added bug good first issue labels Jul 4, 2020

kernc mentioned this issue Aug 21, 2020

Offline search with lunr.js #209

Merged

kernc removed the good first issue label Sep 4, 2020

kernc mentioned this issue Sep 10, 2020

Document an object in a place other than where it was defined #259

Closed

Nov	DEC	Jan
	22
2019	2020	2021

pdoc3 / pdoc

pdoc3 references internal symbols instead of public ones #228

pdoc3 references internal symbols instead of public ones #228

CarstenLeue commented Jul 1, 2020

kernc commented Jul 4, 2020

Carsten-Leue commented Jul 5, 2020 •

edited by kernc

kernc commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

kernc commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

kernc commented Jul 6, 2020

pdoc3 / pdoc

Sponsor pdoc3/pdoc

pdoc3 references internal symbols instead of public ones #228

pdoc3 references internal symbols instead of public ones #228

Comments

CarstenLeue commented Jul 1, 2020

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional info

kernc commented Jul 4, 2020

Carsten-Leue commented Jul 5, 2020 • edited by kernc

kernc commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

kernc commented Jul 5, 2020

Carsten-Leue commented Jul 5, 2020

kernc commented Jul 6, 2020

Carsten-Leue commented Jul 5, 2020 •

edited by kernc