Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in_process_executor use-case to Kubernetes deploy documentation. #20815

Conversation

czroth
Copy link
Contributor

@czroth czroth commented Mar 28, 2024

Summary & Motivation

We have a partitioned job running on our Kubernetes cluster which benefited greatly by switching from the default executor to the in_process_executor.

Other Kubernetes deployed users may benefit from the same setup and this in not covered in the kubernetes deploy docs.

In our case, we went from this being typical:

image

to this:

image

Resulting in our jobs running in a third of the time, and using fewer resources.

Steps like the following were completely avoided by switching to the in_process_executor:

image

How I Tested These Changes

By deploying the user code to our production server with only the single change from the default executor to the in_process_executor.

@czroth czroth requested a review from erinkcochran87 as a code owner March 28, 2024 23:14
@garethbrickman garethbrickman added area: docs Related to documentation in general deployment: k8s Related to deploying Dagster to Kubernetes labels Mar 29, 2024
@czroth
Copy link
Contributor Author

czroth commented Apr 1, 2024

@erinkcochran87, thanks for picking this up to review. If you need any additional details from me, please ask.

@czroth
Copy link
Contributor Author

czroth commented Apr 3, 2024

h/t to @sebaap for helpful discussion related to this PR on Dagster's slack.

Copy link
Contributor

@erinkcochran87 erinkcochran87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! I'm looping in @gibsondan and @mlarose to also give this a once-over.

Could you also please run make mdx-format in the /docs directory and commit any changes? The Markdown files in our docs have be formatted a specific way to pass our tests - this command will do the formatting for you.

@czroth
Copy link
Contributor Author

czroth commented Apr 17, 2024

@mlarose , I believe I've addressed your comments. Let me know if I missed anything.
@erinkcochran87 , let me know if it reads better with less repetition on my part.

In the last commit, I combine two paragraphs that start with the same sentence and come on either side of the heading Kubernetes configuration on every step in a run. I think this reads better than having them separate, but if you disagree, let me know and I'll revert the change.

@czroth
Copy link
Contributor Author

czroth commented Apr 17, 2024

And I re-ran yarn mdx-format before the last commit. But I've never used mdx or yarn before, so I'm uncertain as to whether I've got everything correct here, including the PyObject module links. Someone will need to double check that.

Improved readability and hyperlinks.
Commit suggestion made by @erinkcochran87.

Co-authored-by: Erin Cochran <[email protected]>
@czroth czroth requested a review from erinkcochran87 April 17, 2024 19:32
@erinkcochran87 erinkcochran87 merged commit 957e752 into dagster-io:master Apr 24, 2024
1 of 2 checks passed
@czroth czroth deleted the kubernetes-deploy-single-pod-single-processor-job branch April 24, 2024 15:09
nikomancy pushed a commit to nikomancy/dagster that referenced this pull request May 1, 2024
…agster-io#20815)

## Summary & Motivation

We have a partitioned job running on our Kubernetes cluster which
benefited greatly by switching from the default executor to the
`in_process_executor`.

Other Kubernetes deployed users may benefit from the same setup and this
in not covered in the kubernetes deploy docs.

In our case, we went from this being typical:


![image](https://github.com/dagster-io/dagster/assets/95671524/98118f90-189d-4880-90f4-83517eb304e7)

to this:


![image](https://github.com/dagster-io/dagster/assets/95671524/32cac97d-581e-4859-a9e8-474d8c9a2315)

Resulting in our jobs running in a third of the time, and using fewer
resources.

Steps like the following were completely avoided by switching to the
`in_process_executor`:


![image](https://github.com/dagster-io/dagster/assets/95671524/28d0171c-ffcb-4f52-8941-fadd48f63874)

## How I Tested These Changes

By deploying the user code to our production server with only the single
change from the default executor to the `in_process_executor`.

---------

Co-authored-by: Erin Cochran <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: docs Related to documentation in general deployment: k8s Related to deploying Dagster to Kubernetes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants