Modularize the Arkouda build process

[bmcdonald3]

This draft PR is a prototype of an approach to modularizing the Arkouda build process aimed at promoting design discussion for next steps in this effort.

Summary

The approach taken in this prototype was adding a map that maps from command strings to first class functions and removed the majority of commands from the main select cmd statement so that they would not be required with every Arkouda build. A ServerModules.cfg configuration file was added that takes the name of modules to include in the build, one per line, and generates import statements for those modules and adds their messaging functions to the command map only when in the configuration file to avoid compiling files that are not desired in a given Arkouda build.

Using the configuration file

Running make on this branch compiles Arkouda just like today with all modules included, and that is because the ServerModules.cfg file has all of the module names included by default. To build Arkouda without a module, all that is needed to do is comment out the module that you would like to exclude from that file and rerun make.

Additionally, in the case where a user has multiple configuration files, for example, the user can specify which configuration file will be used to generate the import statements by setting the environment variable ARKOUDA_CONFIG_FILE

Adding a module/converting an existing module

Assuming a module is already created that has its messaging functions, all that the user would need to do to add that module into a build is
(1) add a registerMe() (name up for discussion...) function in their file adding each function to the command map that looks like this:

proc registerMe() {
  use CommandMap;
  registerFunction("my-cmd", myCmdMsg);
  registerFunction("my-cmd2", myCmdMsg2);
}

(2) add the module name to the ServerModules.cfg file
(3) add a client-side function to handle the call

[bmcdonald3]

Adding a few additional comments:

Backbone modules

The approach taken in this initial prototype for determining which modules were considered "backbone" modules (modules that are included by default in every build) was moving all modules that were used in the MsgProcessing.chpl file to optional modules and leaving the rest as they stand in arkouda_server.chpl. I would be interested in hearing other's opinions on that if they have a different idea for what should be included in the "backbone".

Adding a function to the client side

In the case where a user would like to add their own module with this modular approach, the challenge still remains of how they would make those functions accessible on the client side. One approach that was talked about (thanks for the suggestions Kyle/Elliot) was to have a script to call the new server function like this:

import arkouda as ak
from arkouda.client import generic_msg

def test_command():
    rep_msg = generic_msg(cmd='test-command')

ak.__dict__["test_command"] = test_command

And then import that script in their python environment to add the command to the client.

[bradcray]

@mhmerrill : This is Ben's draft of modularizing Arkouda using a separate configuration file and Python script as you requested when he demo'd it the other week. We'll be curious for your feedback when you get the chance to take a look at this.

[mhmerrill]

@bmcdonald3 @bradcray @Bears-R-Us/arkouda-core-dev i've finally got a chance to glance at this and I am happy with it so far. I do have several ideas I think we should discuss.

• defining a set of core/backbone operations which are always included
• ability to include content from another directory instead of just the arkouda/src directory
• ability to include more client code maybe by exporting/importing some basic arkouda client python messaging functionality
• mason package inclusion but maybe that is a bridge too far...
• maybe having a way of having multiple cfg files which you can point at without editing each time you want to change cfg

[mhmerrill]

wow, great minds think alike I just went back and read the description about including client side code ;-)

[glitch]

I think this looks good. The only thing I think should be done is moving Parquet support into the optional configuration since it is arguably the most optional module we have ;)

EDIT: With Ben's explanation for why Parquet needs to remain in, I'm good with this.

[bmcdonald3]

Closing as I have opened #1017 instead