Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize file copying during build #74

Closed
focustense opened this issue Jul 25, 2021 · 0 comments
Closed

Parallelize file copying during build #74

focustense opened this issue Jul 25, 2021 · 0 comments
Labels
easynpc Issues/requests relating the EasyNPC app enhancement New feature or request
Milestone
v0.90-beta2

Comments

@focustense
Copy link
Owner

Although the copy itself (or write to BSA, as in #73) is going to be strictly I/O bound, I believe that for files in BSAs, there is some non-trivial overhead and that parallel reads could lead to some CPU efficiency gains.

It has to be tested, but one user mentioned preferring loose files for a different tool because the build ran 20-30% faster that way. I think BSAs can probably be just as fast - there's really no reason they shouldn't be since it's just reading the same number of bytes or even fewer bytes (if compressed) from an I/O stream, and there's less overhead opening and closing the files. So any additional overhead coming from BSA extraction must be in things like decompression, which can be done concurrently on most modern rigs.
@focustense focustense added enhancement New feature or request easynpc Issues/requests relating the EasyNPC app labels Jul 25, 2021
focustense added a commit that referenced this issue Aug 16, 2021
@focustense
Initial rewrite of the build system.
bf31400 
Intent is to have a loosely-coupled arrangement of tasks, with a pipeline system to handle all the scheduling and dependencies. This will allow us to take some of the extremely complex tasks (like the former `MutagenMergedPluginBuilder` and `MergedFolder`) and turn them into manageable units of work, and maybe improve parallelism.

Currently not tested at all, there are probably several bugs in here, but as yet none of this is linked to the main app. The initial graph does assemble with Autofac.

#26 #37 #64 #74
@focustense focustense added this to the v0.9-beta2 milestone Aug 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
easynpc Issues/requests relating the EasyNPC app enhancement New feature or request
Projects
None yet
Milestone
v0.90-beta2
Development

No branches or pull requests
1 participant
@focustense