Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate eko.tar and fastkernel tables in the server #2081

Closed
scarlehoff opened this issue May 15, 2024 · 3 comments · Fixed by #2087
Closed

Separate eko.tar and fastkernel tables in the server #2081

scarlehoff opened this issue May 15, 2024 · 3 comments · Fixed by #2087
Labels
enhancement New feature or request

Comments

@scarlehoff
Copy link
Member

scarlehoff commented May 15, 2024

Given that the tables for our current NNLO theory are about ~700 MB and the eko for the postfit evolution instead ~1 GB, I think it would be very convenient to separate both.

I think this would bring only advantages:

  1. ekos are not needed for validphys, so no need to have them in your computer to make plots
  2. If you need to copy stuff to your nodes when doing a fit, you halve the size of what you are copying for the fit itself.
  3. We have many theories without an eko, where the eko is actually shared (e.g. theory 200, 400, 700, 708, 600, 40000000). Making it a separate entity means you can have just one 1GB object in the server instead of many.

I would not change anything of the architecture once they are downloaded since they get used in the same manner. I would just create another resource which then gets downloaded to the right theory id and evolven3fit will depend on it.

@scarlehoff scarlehoff added the enhancement New feature or request label May 15, 2024
@RoyStegeman
Copy link
Member

So would there be two download scripts as well? The advantages seem very minor to me (memory is very cheap and bandwidth when downloading is hardly an issue) so I'm not really sure this would be worth the effort and increased complexity of the theory management (in particular due to point 3)

@scarlehoff
Copy link
Member Author

scarlehoff commented May 16, 2024

So would there be two download scripts as well?

No, it is the same resource loader.

memory is very cheap

Just wait until you get your mac.
Jokes aside, I'm getting my jobs throttled in the cluster due to this.

(and we had to ask CERN for extra space because we exhausted the disk space that was allocated to us)

bandwidth when downloading is hardly an issue

At home in Germany I usually need a few attempts because getting 2GB of data in one go is too much for Vodafone.

so I'm not really sure this would be worth the effort and increased complexity of the theory management

There are quite a few theories without an eko inside. And right now it might not be complex but it is somewhat broken (#2003). Also, I'd say doing it in this way would reduce complexity, not increase it.

@scarlehoff scarlehoff mentioned this issue May 20, 2024
34 tasks
@scarlehoff
Copy link
Member Author

After talking a bit with @comane, this gives us yet another advantage.
If the eko is not included in the theory (and given that most of the weight of the theories is now the eko) we can try to use the normal theories for the tests. The fktables will be heavier that theories like 399 but it also allows us to test exactly the same thing we are running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants