[pylint] Re-implement unreachable (PLW0101)

[augustelalande]

Summary

This PR re-introduces the control-flow graph implementation which was first introduced in #5384, and then removed in #9463 due to not being feature complete. Mainly, it lacked the ability to process try-except blocks, along with some more minor bugs.

Closes #8958 and #8959 and #14881.

Overview of Changes

I will now highlight the major changes implemented in this PR, in order of implementation.

1. Introduced a post-processing step in loop handling to find any continue or break statements within the loop body and redirect them appropriately.
2. Introduced a loop-continue block which is always placed at the end of loop blocks, and ensures proper looping regardless of the internal logic of the block. This resolves Control-flow graph fails to connect body-to-loop when loop is the last statement #8958.
3. Implemented try processing with the following logic (resolves Complete try-except handling in control-flow graph #8959):
   1. In the example below the cfg first encounters a conditional ExceptionRaised forking if an exception was (or will be) raised in the try block. This is not possible to know (except for trivial cases) so we assume both paths can be taken unconditionally.
   2. Going down the try path the cfg goes try->else->finally unconditionally.
   3. Going down the except path the cfg will meet several conditional ExceptionCaught which fork depending on the nature of the exception caught. Again there's no way to know which exceptions may be raised so both paths are assumed to be taken unconditionally.
   4. If none of the exception blocks catch the exception then the cfg terminates by raising a new exception.
   5. A post-processing step is also implemented to redirect any raises or returns within the blocks appropriately.

def func():
    try:
        print("try")
    except Exception:
        print("Exception")
    except OtherException as e:
        print("OtherException")
    else:
        print("else")
    finally:
        print("finally")

flowchart TD
  start(("Start"))
  return(("End"))
  block0[["`*(empty)*`"]]
  block1["print(#quot;finally#quot;)\n"]
  block2["print(#quot;else#quot;)\n"]
  block3["print(#quot;try#quot;)\n"]
  block4[["Exception raised"]]
  block5["print(#quot;OtherException#quot;)\n"]
  block6["try:
        print(#quot;try#quot;)
    except Exception:
        print(#quot;Exception#quot;)
    except OtherException as e:
        print(#quot;OtherException#quot;)
    else:
        print(#quot;else#quot;)
    finally:
        print(#quot;finally#quot;)\n"]
  block7["print(#quot;Exception#quot;)\n"]
  block8["try:
        print(#quot;try#quot;)
    except Exception:
        print(#quot;Exception#quot;)
    except OtherException as e:
        print(#quot;OtherException#quot;)
    else:
        print(#quot;else#quot;)
    finally:
        print(#quot;finally#quot;)\n"]
  block9["try:
        print(#quot;try#quot;)
    except Exception:
        print(#quot;Exception#quot;)
    except OtherException as e:
        print(#quot;OtherException#quot;)
    else:
        print(#quot;else#quot;)
    finally:
        print(#quot;finally#quot;)\n"]

  start --> block9
  block9 -- "Exception raised" --> block8
  block9 -- "else" --> block3
  block8 -- "Exception" --> block7
  block8 -- "else" --> block6
  block7 --> block1
  block6 -- "OtherException" --> block5
  block6 -- "else" --> block4
  block5 --> block1
  block4 --> return
  block3 --> block2
  block2 --> block1
  block1 --> block0
  block0 --> return

Loading

6. Implemented with processing with the following logic:
   1. with statements have no conditional execution (apart from the hidden logic handling the enter and exit), so the block is assumed to execute unconditionally.
   2. The one exception is that exceptions raised within the block may result in control flow resuming at the end of the block. Since it is not possible know if an exception will be raised, or if it will be handled by the context manager, we assume that execution always continues after with blocks even if the blocks contain raise or return statements. This is handled in a post-processing step.

Test Plan

Additional test fixtures and control-flow fixtures were added.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+21 -0 violations, +0 -0 fixes in 6 projects; 49 projects unchanged)

apache/airflow (+14 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

+ airflow/triggers/base.py:102:9: PLW0101 Unreachable code in `run`
+ airflow/triggers/testing.py:52:13: PLW0101 Unreachable code in `run`
+ dev/example_dags/update_example_dags_paths.py:50:5: PLW0101 Unreachable code in `check_if_url_exists`
+ tests/models/test_baseoperator.py:754:21: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:1065:21: PLW0101 Unreachable code in `other_setup`
+ tests/models/test_mappedoperator.py:1081:21: PLW0101 Unreachable code in `my_setup`
+ tests/models/test_mappedoperator.py:1107:21: PLW0101 Unreachable code in `other_setup`
+ tests/models/test_mappedoperator.py:1280:21: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:1297:21: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:1599:17: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:1637:25: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:1684:25: PLW0101 Unreachable code in `my_work`
+ tests/models/test_mappedoperator.py:994:21: PLW0101 Unreachable code in `my_setup`
+ tests/models/test_taskinstance.py:3454:13: PLW0101 Unreachable code in `on_finish_callable`

freedomofpress/securedrop (+1 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ admin/securedrop_admin/__init__.py:78:5: PLW0101 Unreachable code in `openssh_version`

latchbio/latch (+1 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ src/latch_cli/snakemake/single_task_snakemake.py:236:5: PLW0101 Unreachable code in `empty_generator`

pandas-dev/pandas (+1 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ pandas/io/parsers/python_parser.py:1316:25: PLW0101 Unreachable code in `_get_lines`

pytest-dev/pytest (+3 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ src/_pytest/config/__init__.py:1842:9: PLW0101 Unreachable code in `_assertion_supported`
+ testing/code/test_code.py:151:17: PLW0101 Unreachable code in `test_bad_getsource`
+ testing/code/test_code.py:167:17: PLW0101 Unreachable code in `test_getsource`

astropy/astropy (+1 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

+ astropy/table/tests/test_bst.py:18:5: PLW0101 Unreachable code in `tree`

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
PLW0101	21	21	0	0	0

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[charliermarsh]

Nice, I'm a fan of this.

[augustelalande]

No promises, just started taking a look at it.

[augustelalande]

Can you give me some background on why this was never activated? Was there just too many false positives?

[charliermarsh]

No, I think it's fairly reliable, but I believe try-except handling wasn't quite finished: #8959. There's also at least one bug to fix: #8958.

[codspeed-hq]

CodSpeed Performance Report

Merging #10891 will not alter performance

_{Comparing augustelalande:unreachable (d13ff6a) with main (2355472)}

Summary

✅ 32 untouched benchmarks

[augustelalande]

This is probably not completely ready for merging, but considering the size it's probably worth getting some feedback now. All the ecosystem checks seem like true positives, there might still be some false negatives, but those will be harder to find.

[augustelalande]

@charliermarsh when you get the time please have a look at this

[MichaReiser]

Sorry for the emberassing long wait. I make this a priority for next week.

The best way to review this change is probably to compare all changes starting after the "Revert" commit. If you have any other review recommendations, let me know :)

[augustelalande]

@MichaReiser no worries. Ya that's a good strategy, but also take a look at my PR summary at the top, I tried to give an outline of the changes I made.

[MichaReiser]

I rebased the commit onto main

[dylwil3]

@augustelalande Upon further reflection, I've decided to merge this in as-is. The various remaining tweaks do not really affect the core functionality of this rule, and could be tackled in follow-up PRs.

This is an outstanding contribution to Ruff, and your hard work deserves to be part of the codebase!

Thank you once again for your patience and persistence!

[augustelalande]

Thanks @dylwil3