-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add in_process_executor use-case to Kubernetes deploy documentation. #20815
Add in_process_executor use-case to Kubernetes deploy documentation. #20815
Conversation
@erinkcochran87, thanks for picking this up to review. If you need any additional details from me, please ask. |
h/t to @sebaap for helpful discussion related to this PR on Dagster's slack. |
…ngle-processor-job
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this! I'm looping in @gibsondan and @mlarose to also give this a once-over.
Could you also please run make mdx-format
in the /docs
directory and commit any changes? The Markdown files in our docs have be formatted a specific way to pass our tests - this command will do the formatting for you.
docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx
Outdated
Show resolved
Hide resolved
docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx
Outdated
Show resolved
Hide resolved
docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx
Outdated
Show resolved
Hide resolved
docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx
Outdated
Show resolved
Hide resolved
@mlarose , I believe I've addressed your comments. Let me know if I missed anything. In the last commit, I combine two paragraphs that start with the same sentence and come on either side of the heading Kubernetes configuration on every step in a run. I think this reads better than having them separate, but if you disagree, let me know and I'll revert the change. |
And I re-ran |
docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx
Outdated
Show resolved
Hide resolved
Improved readability and hyperlinks. Commit suggestion made by @erinkcochran87. Co-authored-by: Erin Cochran <[email protected]>
…agster-io#20815) ## Summary & Motivation We have a partitioned job running on our Kubernetes cluster which benefited greatly by switching from the default executor to the `in_process_executor`. Other Kubernetes deployed users may benefit from the same setup and this in not covered in the kubernetes deploy docs. In our case, we went from this being typical: ![image](https://github.com/dagster-io/dagster/assets/95671524/98118f90-189d-4880-90f4-83517eb304e7) to this: ![image](https://github.com/dagster-io/dagster/assets/95671524/32cac97d-581e-4859-a9e8-474d8c9a2315) Resulting in our jobs running in a third of the time, and using fewer resources. Steps like the following were completely avoided by switching to the `in_process_executor`: ![image](https://github.com/dagster-io/dagster/assets/95671524/28d0171c-ffcb-4f52-8941-fadd48f63874) ## How I Tested These Changes By deploying the user code to our production server with only the single change from the default executor to the `in_process_executor`. --------- Co-authored-by: Erin Cochran <[email protected]>
Summary & Motivation
We have a partitioned job running on our Kubernetes cluster which benefited greatly by switching from the default executor to the
in_process_executor
.Other Kubernetes deployed users may benefit from the same setup and this in not covered in the kubernetes deploy docs.
In our case, we went from this being typical:
to this:
Resulting in our jobs running in a third of the time, and using fewer resources.
Steps like the following were completely avoided by switching to the
in_process_executor
:How I Tested These Changes
By deploying the user code to our production server with only the single change from the default executor to the
in_process_executor
.