Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modularize build process #1017

Merged
merged 3 commits into from
Jan 13, 2022
Merged

Conversation

bmcdonald3
Copy link
Contributor

@bmcdonald3 bmcdonald3 commented Jan 7, 2022

Summary

The approach taken in this prototype was adding a map that maps from command strings to first class functions and removed the majority of commands from the main select cmd statement so that they would not be required with every Arkouda build. A ServerModules.cfg configuration file was added that takes the name of modules to include in the build, one per line, and generates import statements for those modules and adds their messaging functions to the command map only when in the configuration file to avoid compiling files that are not desired in a given Arkouda build.

Using the configuration file

Running make on this branch compiles Arkouda just like today with all modules included, and that is because the ServerModules.cfg file has all of the module names included by default. To build Arkouda without a module, all that is needed to do is comment out the module that you would like to exclude from that file and rerun make.

Additionally, in the case where a user has multiple configuration files, for example, the user can specify which configuration file will be used to generate the import statements by setting the environment variable ARKOUDA_CONFIG_FILE

Adding a module/converting an existing module

Assuming a module is already created that has its messaging functions, all that the user would need to do to add that module into a build is
(1) add a registerMe() (name up for discussion...) function in their file adding each function to the command map that looks like this:

proc registerMe() {
  use CommandMap;
  registerFunction("my-cmd", myCmdMsg);
  registerFunction("my-cmd2", myCmdMsg2);
}

(2) add the module name to the ServerModules.cfg file
(3) add a client-side function to handle the call

Remaining work

In the future, we would like to add the ability to add modules from any directory into this build to allow a user to easily plug in their own modules. Currently, this PR is limited to modules in the src/ directory.

@glitch glitch linked an issue Jan 7, 2022 that may be closed by this pull request
@bmcdonald3 bmcdonald3 force-pushed the modularize-build branch 2 times, most recently from eb38c08 to 142827b Compare January 8, 2022 00:23
Copy link
Collaborator

@glitch glitch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good to me. I pulled it locally and ran a few builds / test runs, etc.

Obviously if you comment modules out tests will fail/hang etc. but since this is meant for development use there is an expectation that you know what you're doing.

In the future we may need to figure out some type of module dependency management and dynamic inclusion/exclusion for unit tests (maybe through the use of pytest marks), but that's another story for another time ;)

Nice work!

@glitch
Copy link
Collaborator

glitch commented Jan 12, 2022

@bmcdonald3 The pydoc looks good to me 👍

@mhmerrill mhmerrill self-requested a review January 12, 2022 20:03
Copy link
Contributor

@mhmerrill mhmerrill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks ok to me

@mhmerrill mhmerrill merged commit e622dc5 into Bears-R-Us:master Jan 13, 2022
@bmcdonald3 bmcdonald3 deleted the modularize-build branch January 27, 2022 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modularization of the Arkouda build process
3 participants