Highlight specific lines of code using comments

[heitorlessa]

Context

As of today, you can highlight lines using hl_lines= syntax when adding a code block. This is great when you start, or when you rarely update the code block.

def bubble_sort(items):
    for i in range(len(items)):
        for j in range(len(items) - 1 - i):
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]

As we grow way beyond 500 code snippets, we run into issues where sometimes a new import, or code snippet improvements break highlight. Two years ago we also added formatting and linting of all code snippets for quality and consistency, but it aggravated the issue at scale.

For example, if we make a minor update to the code snippet above it will break the highlight:

from __future__ import annotations

def bubble_sort(items: list):
    for i in range(len(items)):
        for j in range(len(items) - 1 - i):
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]

Description

It'd be great to be able to highlight a specific code line using comments. For example:

def bubble_sort(items):
    for i in range(len(items)):  # hl
        for j in range(len(items) - 1 - i):  # hl
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]

This removes the need to manually fix each code snippet hl_lines whenever there's a change in the code that shifts the line number where the highlight is. This is similar to the great code annotation feature today, whereas instead of # (1) we'd use another term to highlight it.

Related links

• Requesting a change
• Code highlight
• Discussion that triggered a feature request

Use Cases

In our case, we want to add from __future__ import annotations to all code snippets for older Python versions (e.g., Python 3.8), along with other enhancements. This however would turn into days of work to change, format, lint, verify, and fix each code highlighted line we had before / after the change.

Off the top of my head, this would benefit people:

• supporting multiple versions of a given programming language, where code is largely the same with syntax/idioms differences
• large documentation with code snippets being refactored
• any documentation size where code snippets are updated on a regular basis
• documentation websites that use formatting, linting, and testing tools at CI to ensure code snippets are testable

One of the examples that would break should we make that change: https://docs.powertools.aws.dev/lambda/python/latest/utilities/batch/#recommended

Visuals

Before submitting

• I have read and followed the change request guidelines.
• I have verified that my idea is a change request and not a bug report.
• I have ensured that, to the best of my knowledge, my idea will benefit the entire community.
• I have included relevant links to the documentation, related issues, and discussions to underline the need for my idea.

[squidfunk]

Thanks for suggesting! Definitely a sound use case, and yes, the current situation is not ideal.

Line highlighting is implemented in pymdownx.highlight. I'd recommend taking this upstream to Python Markdown Extensions, as it's the package that provides the original highlighting functionality. Code annotations are mounted in the browser (with non-JS fallback), but highlighting needs to be implemented in Python a we would have no fallback, with no lines being highlighted without JS. It might be an extension to the current parser or some custom SuperFences logic.

I'm taking this back with me to the drawing board where I'm currently at, thinking deeply about the UX/DX/AX of Material for MkDocs. For now, we'd consider this upstream, but we'll look into improving the highlighting capabilities.

[squidfunk]

Just another idea: as mentioned, you could write a simple custom fence that pre-processes each code block looking for # hl or even # hl:start and # hl:end comments to support multi-line highlighting. This fence would then extract and remove those from the code block and configure pygments with the appropriate hl_lines. If that proves to be viable, we can think about abstracting this further and bring it to the Material for MkDocs code base.

[heitorlessa]

Got any quick link on how to write and package one as part of the build? I might be able to do that over next weekend :)

…

On Wed, 26 Jun 2024 at 11:11, Martin Donath ***@***.***> wrote: Just another idea: as mentioned, you could write a simple custom fence that pre-processes each code block looking for # hl or even # hl:start and # hl:end comments to support multi-line highlighting. This fence would then extract and remove those from the code block and configure pygments with the appropriate hl_lines. If that proves to be viable, we can think about abstracting this further and bring it to the Material for MkDocs code base. — Reply to this email directly, view it on GitHub <#7296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBAKQS2RFGWBZOXOSM3ZJKAS5AVCNFSM6AAAAABJ5JB77KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGIYDENJVHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alexvoss]

The documentation for custom fences should contain all you need. One thing I am not sure about is how to do some processing and then allow the standard processing to take place afterwards. Perhaps @facelessuser can comment on that. Would it be possible to modify the code block in a validator and return False from it to pass through to the next validator/formatter?

[squidfunk]

Hmm, I searched but no, not really, except for the official documentation on custom fences. Maybe @facelessuser has a quick snippet where a custom fence extends / pre-configures the functionality of another fence? As TL;DR, turning this:

``` py
def bubble_sort(items):
    for i in range(len(items)):  # hl
        for j in range(len(items) - 1 - i):  # hl
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j] #hl
```

Into this:

``` py hl_lines="2-3,5"
def bubble_sort(items):
    for i in range(len(items)):
        for j in range(len(items) - 1 - i): 
            if items[j] > items[j + 1]:
                items[j], items[j + 1] = items[j + 1], items[j]
```

... and then feeding it into Pygments.

[squidfunk]

@alexvoss is faster than Lucky Luke 🤠 Sorry for double-tagging, Issac.

[leandrodamascena]

Hello everyone! I came with the same question that @heitorlessa has about how we can generate comment highlights in the code instead of having to do it in the .md file. We have over 600 snippets in our documentation and we are doing a major refactoring on most of them - aws-powertools/powertools-lambda-python#4959. Changing the .md files to have the correct highlight will take days/weeks of work and we may still run into the error of not having the highlight correctly.

I tried some solutions that were given here to create a custom_fences, but without much progress because I don't know how to make it render what I want.

Custom fence configuration:

Custom fence code:

def pseudo_highlight(src="", language="", class_name=None, options=None, md="", **kwargs):
    """Formatter wrapper."""
    return """
           ```python hl_lines="1" 
                import json
           ``` 
           """

Result

Is there any chance to reopen this issue and you include this support in the project?

Thanks

[squidfunk]

I agree with this being a useful feature/addition. However, this is a feature request that must be raised either to Python Markdown Extensions or Pygments (?). We can't do anything about it in Material for MkDocs.

[leandrodamascena]

I agree with this being a useful feature/addition. However, this is a feature request that must be raised either to Python Markdown Extensions or Pygments (?). We can't do anything about it in Material for MkDocs.

Thanks for the reply @squidfunk! I don't have much knowledge in either project, but I'll read the code for both and try to understand where it fits best. If you have any suggestions/guidance, I'd love to hear them.

By the way, after I open the issue with my use case, does it make sense for you to try to influence the decision by saying that this adds value to Material for the Mkdocs project and can benefit customers? Since this project is widely used, your opinion carries a lot of weight. But it's just an idea.

Thanks again.

[squidfunk]

So from my experience, there are some people that value my opinion and there are some that don't. That's just about how it is with every opinion, you'll find some that share it and some that don't ☺️ I think this issue might have already been discussed before, so it's a good idea to use the search here and in Python Markdown Extensions.

[facelessuser]

I'm the author of Pymdown Extensions, which provides SuperFences. I'm going to be honest, I'm not a huge fan of using comments in code blocks to highlight lines.

1. Code blocks are supposed to return the content entirely preserved, they aren't supposed to care about the content.
2. Whatever syntax I come up with could be a legitimate comment, so there could be false positives. Sure I can potentially reduce the false positives, but it is always a possibility. Now, this may not be a huge deal as linters use comments on a per-line basis, so it can be navigated.
3. Material already uses annotations this way, and as mentioned so do linters and other tools. And I freely admit Material's annotations are great. But how would we ensure we don't step on each other's toes? Now we have to be mindful of how to parse comments for what we want and preserve comments for things we don't want. This is not an impossible task, but it adds more complexity.
4. Different languages use different comments, some highlighted languages might not even have comments. This means we need something that can identify comments for us, so a preprocessor is out of the question.
5. The best way to utilize comments is the way Material does for annotations, use the highlighter to identify the comments, then parse the HTML afterward to find the tags denoted as comments. At this point, we've already highlighted the content. You have to either double-process the original content or go back and modify the existing HTML.
6. I have no desire to use such a feature, so the motivation to invest the time to add and then maintain is low.

While not impossible, I'm not sure it is something I'm willing to invest time in for the above reasons. Those willing to do so can utilize custom fences to do this work if they desire. If there are questions on how to do so, I'm willing to answer such questions as I have time. Feel free to open a discussion in https://github.com/facelessuser/pymdown-extensions/discussions.

[leandrodamascena]

Thank you both for maintaining amazing open source projects that help customers immensely, that is priceless.

And thanks for providing a space for discussion @facelessuser, but I don't see much point in opening a discussion for a decision that seems to have already been made. I'll wait for the community to get involved (or not) in some similar demand.

[facelessuser]

It wasn't about opening a discussion on a decision already made, but about custom fences:

Those willing to do so can utilize custom fences to do this work if they desire. If there are questions on how to do so, I'm willing to answer such questions as I have time. Feel free to open a discussion in https://github.com/facelessuser/pymdown-extensions/discussions.

[leandrodamascena]

It wasn't about opening a discussion on a decision already made, but about custom fences:

Aaa ok @facelessuser! Sorry for my misunderstanding. I'll open a discussion to see how I can make it work with custom fences. I tried, but I seem to be failing on some things.