Compatibility: pycode parser: adjust dedent-token handling to maintain py3.12+ compatibility

[jayaddison]

Feature or Bugfix

• Bugfix

Purpose

• Resolves test failures in Python code parsing (particularly for named definitions within indented code blocks).

Detail

• From version 3.12.0a7 of Python, the line positions included on DEDENT tokens (encountered during Sphinx's Python code parsing logic, to determine the line-positioning of definitions) are always within the bounds of the number of lines in the source file.
• This means that we should not decrement that line position when we encounter it at the end-of-file for version 3.12+ of Python.

Relates

• Resolves Test failures: Python 3.12-dev: pycode.parser.DefinitionFinder line numbering assertion failures #11436.

Thank you, @mgmacias95 for guidance on the fix.

[jayaddison]

The first of these two conditional if blocks handles implicit dedent tokens that occur when the end-of-file is reached (because any open code blocks should be closed at that point).

I'm less confident in explaining what the second conditional is intended to handle - and it may be possible to simplify the logic used for it.

[jayaddison]

The second condition catches dedent tokens that are contained on the same line as some code. For example, similar to the 'compact' test code coverage added:

class A:
    def b():
        pass
def c():
    pass

... we want the definition of b to omit the line that c is on. It's a non-empty line, and it emits a dedent token, and it contains a name token.

[picnixz]

I think the issue is now solved with python/cpython#105030. At least my latest PR does not fail on 3.12.

[jayaddison]

Thanks @picnixz, glad to hear that the tests are passing again.

I had a read of that thread, was slightly puzzled because it affected a different platform to our test failures, and then through a linked pycodestyle discussion, landed at python/cpython#104980 - which I think was likely the resolution (I could be mistaken.. but from the description and the way that relates to my understanding of the problem here, it fits).

Either way; closing this as unnecessary.

[picnixz]

Do you want to leave the issue open? (maybe for confirmation?)

[jayaddison]

I think we can close it, unless it begins occurring again. Asking for confirmation seems like a good sanity-check though 👍