[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier · 2024-11-27T15:38:59Z

Tracking issue

Why are the changes needed?

To enhance user visibility into Flyte Decks at different stages of workflow execution (running, failing, and succeeding), enabling better debugging and analysis.

What changes were proposed in this pull request?

Concept:

propeller will turn node info to NodeExecutionEvent, and send it to admin.

flyte/flytepropeller/pkg/controller/nodes/executor.go

Lines 1251 to 1261 in b3330ba

    
           nev, err := ToNodeExecutionEvent( 
        
           	nCtx.NodeExecutionMetadata().GetNodeExecutionID(), 
        
           	p, 
        
           	nCtx.InputReader().GetInputPath().String(), 
        
           	nCtx.NodeStatus(), 
        
           	nCtx.ExecutionContext().GetEventVersion(), 
        
           	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(), 
        
           	c.clusterID, 
        
           	nCtx.NodeStateReader().GetDynamicNodeState().Phase, 
        
           	c.eventConfig, 
        
           	targetEntity)

Life Cycle:

propeller keep adding DeckURI when the task is running (whether we enable flyte deck or not)
propeller will put DeckURI to node info, and turn it to NodeExecutionEvent to flyte admin.
flyte admin will add DeckURI to Closure

flyte console will get DeckURI by sending request to admin.

flyte/flyteadmin/dataproxy/service.go

Lines 175 to 189 in b3330ba

    
           		nativeURL = node.GetClosure().GetDeckUri() 
        
           	} 
        
           } else { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource())) 
        
           } 
        
           if len(nativeURL) == 0 { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req) 
        
           } 
        
           ref := storage.DataReference(nativeURL) 
        
           meta, err := s.dataStore.Head(ctx, ref) 
        
           if err != nil { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err) 
        
           }

if flyte console can't get the DeckURI from the node Closure, it will not show the Flyte Deck button.

How was this patch tested?

unit test and remote execution.

Setup process

single binary.

flyte: this branch
flytekit: flyteorg/flytekit#2779
flyteconsole: flyteorg/flyteconsole#890

Screenshots

RUNNING AND SUCCEED

Screen.Recording.2024-11-28.at.10.04.09.AM.mov

FAILED

Screen.Recording.2024-11-28.at.10.06.21.AM.mov

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

follow up questions

should we support Abort phase for the streaming deck?

should we support EPhaseAbort in this file?

https://github.com/flyteorg/flyte/blob/b3330ba4430538f91ae9fc7d868a29a2e96db8bd/flytepropeller/pkg/controller/nodes/handler/transition_info.go

how can we support the auto-refresh UX?

Signed-off-by: Future-Outlier <[email protected]>

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Yi Cheng <[email protected]> Co-authored-by: pingsutw <[email protected]>

codecov · 2024-11-27T15:45:04Z

Codecov Report

Attention: Patch coverage is 51.02041% with 24 lines in your changes missing coverage. Please review.

Project coverage is 37.10%. Comparing base (ab04192) to head (dd774cb).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...lytepropeller/pkg/controller/nodes/task/handler.go	50.00%	19 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6053      +/-   ##
==========================================
+ Coverage   37.08%   37.10%   +0.01%     
==========================================
  Files        1318     1318              
  Lines      132284   132349      +65     
==========================================
+ Hits        49062    49102      +40     
- Misses      78950    78972      +22     
- Partials     4272     4275       +3

Flag	Coverage Δ
unittests-datacatalog	`51.58% <ø> (ø)`
unittests-flyteadmin	`54.07% <100.00%> (-0.03%)`	⬇️
unittests-flytecopilot	`30.99% <ø> (ø)`
unittests-flytectl	`62.29% <ø> (-0.05%)`	⬇️
unittests-flyteidl	`7.23% <ø> (ø)`
unittests-flyteplugins	`53.82% <ø> (+0.09%)`	⬆️
unittests-flytepropeller	`42.62% <50.00%> (-0.01%)`	⬇️
unittests-flytestdlib	`57.59% <ø> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Future-Outlier <[email protected]>

hamersaw · 2024-11-27T16:52:38Z

flytepropeller/pkg/controller/nodes/task/handler.go

 	switch pluginTrns.pInfo.Phase() {
 	case pluginCore.PhaseSuccess:
+		// This is to prevent the console from potentially checking the deck URI that does not exist if in final phase(PhaseSuccess).
+		err = pluginTrns.RemoveNonexistentDeckURI(ctx, tCtx)


Does this do a head call on the deck URI for every task that succeeds? Two thoughts here:
(1) does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?
(2) this is incurring a 20-30ms performance degredation to every task execution

will take a look tmr, thank you!!!

Does this do a head call on the deck URI for every task that succeeds?

yes it will do a head call by RemoteFileOutputReader

flyte/flyteplugins/go/tasks/pluginmachinery/ioutils/remote_file_output_reader.go

Lines 306 to 313 in b3330ba

func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {

md, err := r.store.Head(ctx, r.outPath.GetDeckPath())

if err != nil {

return false, err

}

return md.Exists(), nil

}

how do you know the performance degradation?
did you use grafana or other performance tools?

does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?

flyteadmin will set the deckURI in the execution metadata to nil if the propeller removes it.

cc @hamersaw

Future-Outlier · 2024-11-27T17:02:52Z

How to test it?

start a new sandbox

flytectl demo start --image futureoutlier/sandbox:deck-1205-1138 --force

checkout streaming deck flytekit branch

cd flytekit
gh pr checkout 2779

run a failure task (show deck after it failed)

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "473ae1119af6f86c26c0790dee0affa3eb29be64"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(5):
        Deck.publish()
        # # raise Exception("This is an exception")
        time.sleep(3)

@workflow
def wf():
    t_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    # result = runner.invoke(pyflyte.main,
    #                        ["run", path, "wf"])
    # print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    # "--remote"
    print("Remote Execution: ", result.output)

EngHabu · 2024-11-28T00:38:55Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Future-Outlier · 2024-11-28T01:52:29Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Yes no problem

Future-Outlier · 2024-11-28T02:10:23Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

its provided!
#6053 (comment)

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier and others added 2 commits November 27, 2024 23:36

add tests from Yi-Cheng

54aa165

Signed-off-by: Future-Outlier <[email protected]>

helped by Kevin and Yi-Cheng

9ed6b6e

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Yi Cheng <[email protected]> Co-authored-by: pingsutw <[email protected]>

lint

4b4f6bd

Signed-off-by: Future-Outlier <[email protected]>

hamersaw reviewed Nov 27, 2024

View reviewed changes

nit

dd774cb

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier requested a review from hamersaw December 12, 2024 13:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

hamersaw Nov 27, 2024

Future-Outlier Nov 27, 2024

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier commented Nov 27, 2024 •

edited

Loading

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

	nev, err := ToNodeExecutionEvent(
	nCtx.NodeExecutionMetadata().GetNodeExecutionID(),
	p,
	nCtx.InputReader().GetInputPath().String(),
	nCtx.NodeStatus(),
	nCtx.ExecutionContext().GetEventVersion(),
	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(),
	c.clusterID,
	nCtx.NodeStateReader().GetDynamicNodeState().Phase,
	c.eventConfig,
	targetEntity)

	nativeURL = node.GetClosure().GetDeckUri()
	}
	} else {
	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource()))
	}

	if len(nativeURL) == 0 {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req)
	}

	ref := storage.DataReference(nativeURL)
	meta, err := s.dataStore.Head(ctx, ref)
	if err != nil {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err)
	}

	func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {
	md, err := r.store.Head(ctx, r.outPath.GetDeckPath())
	if err != nil {
	return false, err
	}

	return md.Exists(), nil
	}

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Are you sure you want to change the base?

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Conversation

Future-Outlier commented Nov 27, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

RUNNING AND SUCCEED

FAILED

Check all the applicable boxes

Related PRs

follow up questions

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

hamersaw Nov 27, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 27, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier commented Nov 27, 2024 • edited Loading

How to test it?

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier commented Nov 27, 2024 •

edited

Loading