Skip to content

Possible regression in Python3.12 tokenizer #105013

Closed
@arkamar

Description

@arkamar

Bug report

I have encountered a possible regression while I was testing pyrsistent with python3.12_beta1, see tobgu/pyrsistent#275. Tests fail with following error:

/usr/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.12/site-packages/_pytest/pathlib.py:564: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1293: in _gcd_import
    ???
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/hypothesis_vector_test.py:48: in <module>
    PVectorAndLists = st.lists(st.builds(RefCountTracker)).map(
/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py:347: in map
    if is_identity_function(pack):
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:638: in is_identity_function
    return bool(re.fullmatch(r"lambda (\w+): \1", get_pretty_function_description(f)))
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:432: in get_pretty_function_description
    return extract_lambda_source(f)
/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:312: in extract_lambda_source
    source = inspect.getsource(f)
/usr/lib/python3.12/inspect.py:1274: in getsource
    lines, lnum = getsourcelines(object)
/usr/lib/python3.12/inspect.py:1266: in getsourcelines
    return getblock(lines[lnum:]), lnum + 1
/usr/lib/python3.12/inspect.py:1241: in getblock
    for _token in tokens:
/usr/lib/python3.12/tokenize.py:451: in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
/usr/lib/python3.12/tokenize.py:542: in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
E     File "<string>", line 1
E       lambda l: (l, pvector(l)))
E                                ^
E   SyntaxError: unmatched ')'

It is triggered for this code (see https://github.com/tobgu/pyrsistent/blob/cc90f3e2b339653fde0df422aaf3ccdb3fc1225d/tests/hypothesis_vector_test.py#L48-L49 )

PVectorAndLists = st.lists(st.builds(RefCountTracker)).map(
    lambda l: (l, pvector(l)))

if I concatenate it to one line, everything works. I also tried to backport fix for #104866 as it sounds relevant, but it didn't solved the issue.

Your environment

  • Gentoo ~amd64
  • Python 3.12.0b1

Linked PRs

Metadata

Metadata

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)stdlibPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or error

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions