Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve resume pipeline suggestion for SequentialRunner #1795

Merged
merged 38 commits into from
Sep 2, 2022
Merged

Improve resume pipeline suggestion for SequentialRunner #1795

merged 38 commits into from
Sep 2, 2022

Commits on Aug 19, 2022

  1. Add _find_first_persistent_ancestors and stubs for supporting functions. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    426824a View commit details
    Browse the repository at this point in the history

  2. Add body to _enumerate_parents. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    f90daf7 View commit details
    Browse the repository at this point in the history

  3. Add function to check persistence of node outputs. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    e1cb2e3 View commit details
    Browse the repository at this point in the history

  4. Modify _suggest_resume_scenario to use _find_first_persistent_ancestors 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    18a6105 View commit details
    Browse the repository at this point in the history

  5. Pass catalog to self._suggest_resume_scenario 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    7753486 View commit details
    Browse the repository at this point in the history

  6. Track and return all ancestor nodes that must be re-run during DFS. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    a402aa7 View commit details
    Browse the repository at this point in the history

  7. Integrate DFS with original _suggest_resume_scenario. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    699a9f5 View commit details
    Browse the repository at this point in the history

  8. Implement backwards-DFS strategy on all boundary nodes. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    a49a6f7 View commit details
    Browse the repository at this point in the history

  9. Switch to multi-node start BFS approach to finding persistent ancestors. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    7955d0d View commit details
    Browse the repository at this point in the history

  10. Add a useful error message if no nodes ran. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    68764f7 View commit details
    Browse the repository at this point in the history

  11. Add docstrings to new functions. 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    74c60f7 View commit details
    Browse the repository at this point in the history

  12. Add catalog argument to self._suggest_resume_scenario 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    958fb91 View commit details
    Browse the repository at this point in the history

  13. Modify exception_fn to allow it to take multiple arguments 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    d61a19b View commit details
    Browse the repository at this point in the history

  14. Add test for AbstractRunner._suggest_resume_scenario 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    8724923 View commit details
    Browse the repository at this point in the history

  15. Add docstring for _suggest_resume_scenario 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    f57c431 View commit details
    Browse the repository at this point in the history

  16. Improve formatting 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    9fda4c0 View commit details
    Browse the repository at this point in the history

  17. Move new functions out of AbstractRunner 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    3a79059 View commit details
    Browse the repository at this point in the history

  18. Remove bare except 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    13063dd View commit details
    Browse the repository at this point in the history

  19. Fix broad except clause 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    f29bbf5 View commit details
    Browse the repository at this point in the history

  20. Access datasets __dict__ using vars() 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    01d5ab0 View commit details
    Browse the repository at this point in the history

  21. Sort imports 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    1dae5e7 View commit details
    Browse the repository at this point in the history

  22. Improve resume message 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    d572896 View commit details
    Browse the repository at this point in the history

  23. Add a space to resume suggestion message 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    af405ed View commit details
    Browse the repository at this point in the history

  24. Merge branch 'main' into feat/improve-resume-scenario-suggestion

    @jmholzer
    jmholzer authored Aug 19, 2022
    Configuration menu
    Copy the full SHA
    d1b6693 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2022

  1. Modify DFS logic to eliminate possible queue duplicates 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    836d583 View commit details
    Browse the repository at this point in the history

  2. Modify catalog.datasets to catalog._data_sets w/ disabled linter warning 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    7c7e795 View commit details
    Browse the repository at this point in the history

  3. Move all pytest fixtures to conftest.py 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    069a6fe View commit details
    Browse the repository at this point in the history

  4. Modify all instances of Pipeline to pipeline 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    e4a6525 View commit details
    Browse the repository at this point in the history

  5. Fix typo in the name of TestSequentialRunnerBranchedPipeline 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    32aa7a2 View commit details
    Browse the repository at this point in the history

  6. Remove spurious assert in save of persistent_dataset_catalog 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    e92a756 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. Replace instantiations of Pipeline with pipeline 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    ffb4074 View commit details
    Browse the repository at this point in the history

  2. Modify test_suggest_resume_scenario fixture to use node names 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    1bf5f1d View commit details
    Browse the repository at this point in the history

  3. Add disable=unused-argument to _save 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    c2481db View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into feat/improve-resume-scenario-suggestion

    @jmholzer
    jmholzer authored Aug 31, 2022
    Configuration menu
    Copy the full SHA
    35ad81f View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2022

  1. Remove resume suggestion for ParallelRunner 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    99e0bb2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feat/improve-resume-scenario-suggestion

    @jmholzer
    jmholzer authored Sep 2, 2022
    Configuration menu
    Copy the full SHA
    85f7609 View commit details
    Browse the repository at this point in the history

  3. Remove spurious try / except 

    Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    a74fef6 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'feat/improve-resume-scenario-suggestion' of github.com:… 

    …kedro-org/kedro into feat/improve-resume-scenario-suggestion

Signed-off-by: Jannic Holzer <[email protected]>
    @jmholzer
    jmholzer committed Sep 2, 2022
    Configuration menu
    Copy the full SHA
    74d0cec View commit details
    Browse the repository at this point in the history