Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to format notebook: source contains syntax errors: ParseError #8204

Closed
Jonas1312 opened this issue Oct 25, 2023 · 7 comments · Fixed by #8207
Closed

Failed to format notebook: source contains syntax errors: ParseError #8204

Jonas1312 opened this issue Oct 25, 2023 · 7 comments · Fixed by #8207
Assignees
@dhruvmanila
Labels
bug Something isn't working formatter Related to the formatter
Milestone
Formatter: Stable

Comments

@Jonas1312
Copy link

Hi,

I tried the new ruff formatter on my notebooks through a pre-commit config:

  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.1.2
    hooks:
      - id: ruff
        types_or: [python, pyi, jupyter]
        args:
            - --fix-only 
            - --exit-non-zero-on-fix
      - id: ruff-format
        types_or: [python, pyi, jupyter]

But I get these errors:

error: Failed to format recipes/basic_project_setup.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/counterfactual_data_augmentation.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 18, source_path: "<filename>" }
error: Failed to format recipes/export_a_kili_project.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_dinov2.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_openai.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_openai_summaries.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Exclamation, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/frame_dicom_data.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/geojson.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/import_text_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_assets_and_metadata.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_coco.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_labels.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_pdf_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_video_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/inference_labels.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/label_parsing.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/medical_imaging.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/ner_pre_annotations_openai.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/ocr_pre_annotations.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/pixel_level_masks.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_development.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_example.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_example_document.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/set_up_workflows.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/tagtog_to_kili.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/webhooks_example.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }

Here is the pyproject.toml:

[tool.ruff]
select = ["ALL"]
src = ["src"]
line-length = 100
target-version = "py38"

[tool.ruff.pydocstyle]
convention = "google"

[tool.ruff.format]
line-ending = "lf"

[tool.ruff.isort]
known-first-party = ["src", "tests"]

And the notebooks: https://github.com/kili-technology/kili-python-sdk/tree/main/recipes

Thanks a lot!
@MichaReiser
Copy link
Member

Can you try adding the notebook extensions to extend-include (doc)

[tool.ruff]
extend-include = ["*.ipynb"]

@MichaReiser MichaReiser added question Asking for support or clarification formatter Related to the formatter labels Oct 25, 2023
@Jonas1312
Copy link
Author

Can you try adding the notebook extensions to extend-include (doc)

[tool.ruff]
extend-include = ["*.ipynb"]
ruff.....................................................................Passed
ruff-format..............................................................Failed
- hook id: ruff-format
- exit code: 2

warning: The following rules may cause conflicts when used with the formatter: 'COM819', 'D206', 'ISC001', 'Q000', 'Q001', 'Q002', 'Q003', 'W191'. To avoid unexpected behavior, we recommend disabling these rules, either by removing them from the `select` or `extend-select` configuration, or adding then to the `ignore` configuration.
warning: The following rules may cause conflicts when used with the formatter: 'COM819', 'D206', 'ISC001', 'Q000', 'Q001', 'Q002', 'Q003', 'W191'. To avoid unexpected behavior, we recommend disabling these rules, either by removing them from the `select` or `extend-select` configuration, or adding then to the `ignore` configuration.
error: Failed to format recipes/basic_project_setup.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/counterfactual_data_augmentation.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 18, source_path: "<filename>" }
error: Failed to format recipes/export_a_kili_project.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_dinov2.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_openai.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/finetuning_openai_summaries.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Exclamation, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/frame_dicom_data.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/geojson.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/import_text_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_assets_and_metadata.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_coco.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_labels.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_pdf_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/importing_video_assets.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/inference_labels.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/label_parsing.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/medical_imaging.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/ner_pre_annotations_openai.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/ocr_pre_annotations.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/pixel_level_masks.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_development.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_example.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/plugins_example_document.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/set_up_workflows.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/tagtog_to_kili.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
error: Failed to format recipes/webhooks_example.ipynb: source contains syntax errors: ParseError { error: UnrecognizedToken(Percent, None), offset: 0, source_path: "<filename>" }
401 files left unchanged

@MichaReiser
Copy link
Member

Interesting. @dhruvmanila any ideas? Could it be that we don't correctly set the parsing mode?

@dhruvmanila
Copy link
Member

Yeah, that's correct. Let me send in a patch quickly.

@dhruvmanila dhruvmanila self-assigned this Oct 25, 2023
@MichaReiser MichaReiser added bug Something isn't working question Asking for support or clarification and removed question Asking for support or clarification labels Oct 25, 2023
@MichaReiser MichaReiser added this to the Formatter: Stable milestone Oct 25, 2023
@dhruvmanila dhruvmanila mentioned this issue Oct 25, 2023
Merged
@dhruvmanila dhruvmanila removed the question Asking for support or clarification label Oct 25, 2023
@dhruvmanila dhruvmanila mentioned this issue Oct 25, 2023
Merged
dhruvmanila added a commit that referenced this issue Oct 25, 2023
@dhruvmanila
Formatter parentheses support for IpyEscapeCommand (#8207)
dbd84c9 
## Summary

This PR removes the `todo!()` around `IpyEscapeCommand` in the
formatter.

The `NeedsParentheses` trait needs to be implemented which always return
`Never`. The reason being that if an escape command is parenthesized,
then that's not parsed as an escape command. IOW, the parentheses
shouldn't be present around an escape command.

In the similar way, the `CanSkipOptionalParenthesesVisitor` will skip
this node.

## Test Plan

Updated the `unformatted.ipynb` fixture with new cells containing
IPython escape commands and the corresponding snapshot was verified.
Also, tested it out in a few open source repositories containing
notebooks (`openai/openai-cookbook`, `huggingface/notebooks`).

#### New cells in `unformatted.ipynb`

**Cell 2**
```markdown
A markdown cell
```

**Cell 3**
```python
def some_function(foo, bar):
    pass
%matplotlib inline
```

**Cell 4**
```python
foo = %pwd
def some_function(foo,bar,):
	foo = %pwd
    print(foo
	)
```

fixes: #8204
@dhruvmanila
Copy link
Member

Thanks for opening this issue! It's fixed and will be available in the next release.

@harupy
Copy link
Contributor

harupy commented Oct 26, 2023

@dhruvmanila Thanks for fixing this! I'm curious when the the next release will be available.

@charliermarsh
Copy link
Member

I can probably cut a release tomorrow.

@dhruvmanila dhruvmanila mentioned this issue Oct 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhruvmanila dhruvmanila

Labels
bug Something isn't working formatter Related to the formatter
Projects
None yet
Milestone
Formatter: Stable
Development

Successfully merging a pull request may close this issue.

Formatter parentheses support for IpyEscapeCommand astral-sh/ruff
5 participants
@MichaReiser @charliermarsh @Jonas1312 @harupy @dhruvmanila