Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add source code deck #2038

Merged
merged 9 commits into from
Dec 15, 2023
Merged

Conversation

jasonlai1218
Copy link
Contributor

@jasonlai1218 jasonlai1218 commented Dec 11, 2023

Tracking issue

Why are the changes needed?

Because Data Engineers sometimes open the console (UI) with stakeholders to discuss business logic in the data pipeline, at this time, in addition to the expression of data graphics, they can also present source code to discuss program architecture design if necessary.
This is the idea from @pingsutw, I also agree with this idea very much, so I implemented this function.

What changes were proposed in this pull request?

  • Add pygments to the list of plugin requirements in setup.py
  • Add pygments import in renderer.py
  • Add CSS styling to the rendered HTML code in renderer.py
  • Update the to_html method to include the CSS styling in the returned HTML
  • Add a new static method _get_css in renderer.py to extract CSS from the provided HtmlFormatter instance and replace a specific color code

How was this patch tested?

  • Add a new test method test_get_css in test_renderer.py to test the _get_css method in renderer.py

Setup process

  1. write a sample code
import inspect

import flytekit
import pandas as pd
from flytekitplugins.deck.renderer import SourceCodeRenderer

from flytekit import task, workflow


@task(disable_deck=False)
def frame_renderer() -> None:
    df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]})
    # Get the path of the source code file
    file_path = inspect.getsourcefile(frame_renderer.__wrapped__)
    with open(file_path, 'r') as f:
        source_code = f.read()
    flytekit.Deck("source code", SourceCodeRenderer().to_html(source_code))


@workflow
def wf() -> None:
    frame_renderer()


if __name__ == "__main__":
    print(wf())

  1. execute cmd:FLYTE_SDK_LOGGING_LEVEL=20 pyflyte run deck.py wf

Screenshots

Screenshot 2023-12-11 at 4 40 15 PM
Screenshot 2023-12-11 at 4 40 03 PM

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

flytesnacks: flyteorg/flytesnacks#1319

Docs link

- Add `pygments` to the list of plugin requirements in `setup.py`
- Add `pygments` import in `renderer.py`

Signed-off-by: jason.lai <[email protected]>
- Add CSS styling to the rendered HTML code in `renderer.py`
- Update the `to_html` method to include the CSS styling in the returned HTML

Signed-off-by: jason.lai <[email protected]>
- Add a new class `CustomColorfulStyle` that extends `ColorfulStyle` from `pygments.styles.colorful`
- Remove the styling for `Generic.Error` in `CustomColorfulStyle`
- Comment out the line that initializes `formatter` with the `HtmlFormatter` class and instead initialize it with `CustomColorfulStyle`
- Update the `to_html` method to use the updated `formatter`

Signed-off-by: jason.lai <[email protected]>
- Remove `CustomColorfulStyle` class
- Update `to_html` method to use default `HtmlFormatter` style and fix css color code

Signed-off-by: jason.lai <[email protected]>
…nvolve updating the codebase's configuration files (`renderer.py` and `setup.py`) and do not fix a bug or add a new feature.: update code style and dependencies

- Update the `to_html` method in `renderer.py` to use double quotes for the `HtmlFormatter` style parameter
- Rearrange the order of the `plugin_requires` list in `setup.py`

Signed-off-by: jason.lai <[email protected]>
- Modify the `to_html` method in `renderer.py` to include a docstring explaining its functionality
- Update the `css` variable in the `to_html` method to use a separate method `_get_css` in `renderer.py`
- Add a new static method `_get_css` in `renderer.py` to extract CSS from the provided `HtmlFormatter` instance and replace a specific color code
- Add a new test method `test_get_css` in `test_renderer.py` to test the `_get_css` method in `renderer.py`

Signed-off-by: jason.lai <[email protected]>
Copy link

codecov bot commented Dec 11, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6cd0a60) 85.98% compared to head (b2dc2bb) 64.87%.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2038       +/-   ##
===========================================
- Coverage   85.98%   64.87%   -21.12%     
===========================================
  Files         308      308               
  Lines       22946    22965       +19     
  Branches     3468     3468               
===========================================
- Hits        19731    14898     -4833     
- Misses       2615     7653     +5038     
+ Partials      600      414      -186     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jasonlai1218 jasonlai1218 changed the title Add source code deck feat: add source code deck Dec 11, 2023
- Remove unused imports and code from `renderer.py`
- Refactor `_get_css` method to be a static method of `SourceCodeRenderer`
- Add a test for `SourceCodeRenderer.to_html` method in `test_renderer.py`
- Update assertions in `test_source_code_renderer` to check the result of `renderer.to_html` method

Signed-off-by: jason.lai <[email protected]>
pingsutw
pingsutw previously approved these changes Dec 13, 2023
- Remove the import of `pytest`
- Add the import of `tempfile`
- Remove the import of `pandas`
- Add the import of `markdown`
- Remove the import of `ImageRenderer`
- Change the order of imports

Signed-off-by: jason.lai <[email protected]>
@pingsutw pingsutw merged commit 67470a3 into flyteorg:master Dec 15, 2023
73 of 75 checks passed
RRap0so pushed a commit to RRap0so/flytekit that referenced this pull request Dec 15, 2023
* chore: add pygments as a plugin requirement and import in renderer

- Add `pygments` to the list of plugin requirements in `setup.py`
- Add `pygments` import in `renderer.py`

Signed-off-by: jason.lai <[email protected]>

* style: improve HTML rendering with CSS styling

- Add CSS styling to the rendered HTML code in `renderer.py`
- Update the `to_html` method to include the CSS styling in the returned HTML

Signed-off-by: jason.lai <[email protected]>

* feat: refactor HTML formatting to use custom styling

- Add a new class `CustomColorfulStyle` that extends `ColorfulStyle` from `pygments.styles.colorful`
- Remove the styling for `Generic.Error` in `CustomColorfulStyle`
- Comment out the line that initializes `formatter` with the `HtmlFormatter` class and instead initialize it with `CustomColorfulStyle`
- Update the `to_html` method to use the updated `formatter`

Signed-off-by: jason.lai <[email protected]>

* refactor: refactor code for better HTML formatting

- Remove `CustomColorfulStyle` class
- Update `to_html` method to use default `HtmlFormatter` style and fix css color code

Signed-off-by: jason.lai <[email protected]>

* The label that best describes this change is "chore". These changes involve updating the codebase's configuration files (`renderer.py` and `setup.py`) and do not fix a bug or add a new feature.: update code style and dependencies

- Update the `to_html` method in `renderer.py` to use double quotes for the `HtmlFormatter` style parameter
- Rearrange the order of the `plugin_requires` list in `setup.py`

Signed-off-by: jason.lai <[email protected]>

* feat: refactor `renderer.py` to improve documentation and CSS extraction

- Modify the `to_html` method in `renderer.py` to include a docstring explaining its functionality
- Update the `css` variable in the `to_html` method to use a separate method `_get_css` in `renderer.py`
- Add a new static method `_get_css` in `renderer.py` to extract CSS from the provided `HtmlFormatter` instance and replace a specific color code
- Add a new test method `test_get_css` in `test_renderer.py` to test the `_get_css` method in `renderer.py`

Signed-off-by: jason.lai <[email protected]>

* refactor: refactor SourceCodeRenderer and update test assertions

- Remove unused imports and code from `renderer.py`
- Refactor `_get_css` method to be a static method of `SourceCodeRenderer`
- Add a test for `SourceCodeRenderer.to_html` method in `test_renderer.py`
- Update assertions in `test_source_code_renderer` to check the result of `renderer.to_html` method

Signed-off-by: jason.lai <[email protected]>

* chore: reorganize imports in files for better readability

- Remove the import of `pytest`
- Add the import of `tempfile`
- Remove the import of `pandas`
- Add the import of `markdown`
- Remove the import of `ImageRenderer`
- Change the order of imports

Signed-off-by: jason.lai <[email protected]>

---------

Signed-off-by: jason.lai <[email protected]>
Signed-off-by: Rafael Raposo <[email protected]>
@fg91
Copy link
Member

fg91 commented Jan 10, 2024

Nice feature @jasonlai1218 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants