Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example of dynamic directive with input file #4545

Merged
merged 6 commits into from
Feb 18, 2025

Conversation

bentsherman
Copy link
Member

See #4493 (comment)

I don't really like this particular example because it is probably an anti-pattern, i.e. many tools are designed to stream the input data so that they don't have to load entire datasets in memory. But I want to explicitly show that input files can also be used to configure directives.

Copy link

netlify bot commented Nov 27, 2023

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 8691b04
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/67b4997e30252f0008387967
😎 Deploy Preview https://deploy-preview-4545--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@bentsherman bentsherman mentioned this pull request Nov 27, 2023
@drpatelh
Copy link
Contributor

drpatelh commented Dec 5, 2023

There are some other real-world examples in this talk:
https://youtu.be/oAdSpdUrJag?t=16928

Would be awesome to have something like this to tune resources.

@bentsherman
Copy link
Member Author

@christopher-hakkaart for your consideration. It would be good to have an example of setting the memory based on the task inputs, but ideally would be something that is proven to work in the wild. Instead of using data.size(), which I just made up, you might find a more concrete example in the talk that Harshil linked.

Even something like this could be better:

// allocate 1 GB for every million reads, up to 32 GB
memory { meta.num_reads / 1_000_000 * 1.GB }

@bentsherman bentsherman linked an issue Feb 15, 2025 that may be closed by this pull request
Signed-off-by: Ben Sherman <[email protected]>
@bentsherman
Copy link
Member Author

@christopher-hakkaart can you give this one a review when you have a chance

Copy link
Collaborator

@christopher-hakkaart christopher-hakkaart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor suggestions.

Co-authored-by: Chris Hakkaart <[email protected]>
Signed-off-by: Ben Sherman <[email protected]>
Copy link
Collaborator

@christopher-hakkaart christopher-hakkaart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with the language. The 1e9 notation could be removed or a note could be added to explain. It's not blocking for me.

@bentsherman bentsherman merged commit 2ebeb34 into master Feb 18, 2025
10 checks passed
@bentsherman bentsherman deleted the docs-dynamic-directive-input-file branch February 18, 2025 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

docs: more examples of dynamic resource calculations
4 participants