Automated command registration (#3371)

* remove binary commandMap by including binary-payload field in MsgTuple. Fix misleading indentation in serverDaemon. Remove repeated error-handling code from ServerDaemon.

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove commandMapArray by adding payload field to MessageArgs

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix messageArgs serialization bug

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove unecessary string.doFormat, replacing with string.format

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove symbols duplicated across IOCompat versions

Signed-off-by: Jeremiah Corrado <[email protected]>

* start refactor of ParameterObj. Simplified argument parsing API to include 'toScalar', 'toScalarTuple', 'toScalarList', 'toScalarArray'. Other methods were removed or "deprecated".  Added 'this()' indexing to msgArgs

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove objType enum from Message.chpl (and message.py) - resolves #2522. Only 'efunc2Msg' was using this functionality, so put a temporary fix in place until it can be refactored

Signed-off-by: Jeremiah Corrado <[email protected]>

* add this() accessor to SymbolTable, adjust ManipulationMsg to use it. Fold error handling into 'executeCommand'

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix dep modules wrt IOCompat changes

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix build errors originating in server deamon error message metrics

Signed-off-by: Jeremiah Corrado <[email protected]>

* fold symbol creation response code into the symbol table. Update ManipulationMsg to use the new API

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove objType from message.py tests

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix typo in merge-conflict resolution

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix build failures and warnings for older versions of Chapel

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix merge conflict resolution errors:

Signed-off-by: Jeremiah Corrado <[email protected]>

* add script to support registerCommand and registerAndInstantiate annotations for fully and partially automated command registration

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix frontend parsing of multi-array responses generated from 'MsgTuple.fromResponses'

Signed-off-by: Jeremiah Corrado <[email protected]>

* add register-commnds to makefile for calling register_commands.py script. Rename server-config.json to registration-config.json (update old parseServerConfig script accordingly). Make some minor qol improvements to the new register_commands script

Signed-off-by: Jeremiah Corrado <[email protected]>

* finish converting ManipulationMsg to use new registration script

Signed-off-by: Jeremiah Corrado <[email protected]>

* update 'test_client_get_server_commands' to not include 'array' (this command no longer requires special handling in the serverDaemon)

Signed-off-by: Jeremiah Corrado <[email protected]>

* adjust array_api/manipulation_functions to use new commands. Move type checking into frontend for concat and stack (implement type promotion that was missing)

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix type annotation

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix type annotation

Signed-off-by: Jeremiah Corrado <[email protected]>

* start translating statsMsg to use new registration framework

Signed-off-by: Jeremiah Corrado <[email protected]>

* finish switching statsMsg to new registration framework

Signed-off-by: Jeremiah Corrado <[email protected]>

* generate code with spaces instead of tabs in register_commands.py

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix aryUtilCompat modules

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix compilation errors in stats

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix compilation errors for older versions of chapel and for multidimensional builds

Signed-off-by: Jeremiah Corrado <[email protected]>

* use python dtype names in MsgTuple.fromScalar (instead of Chapel type names)

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix corr and cov computations

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix some array-api test failures

Signed-off-by: Jeremiah Corrado <[email protected]>

* don't include modules in Commands.chpl if they don't contain any annotated procedures

Signed-off-by: Jeremiah Corrado <[email protected]>

* apply black formatting to array api modules

Signed-off-by: Jeremiah Corrado <[email protected]>

* fix failing array api tests

Signed-off-by: Jeremiah Corrado <[email protected]>

* remove duplicate line from ServerDaemon

Signed-off-by: Jeremiah Corrado <[email protected]>

---------

Signed-off-by: Jeremiah Corrado <[email protected]>

Makefile

@@ -3,6 +3,7 @@ ARKOUDA_PROJECT_DIR := $(dir $(realpath $(firstword $(MAKEFILE_LIST))))
 
 PROJECT_NAME := arkouda
 ARKOUDA_SOURCE_DIR := $(ARKOUDA_PROJECT_DIR)/src
+ARKOUDA_REGISTRY_DIR=$(ARKOUDA_SOURCE_DIR)/registry
 ARKOUDA_MAIN_MODULE := arkouda_server
 ARKOUDA_MAKEFILES := Makefile Makefile.paths
 
@@ -85,6 +86,10 @@ ifndef ARKOUDA_CONFIG_FILE
 ARKOUDA_CONFIG_FILE := $(ARKOUDA_PROJECT_DIR)/ServerModules.cfg
 endif
 
+ifndef ARKOUDA_REGISTRATION_CONFIG
+ARKOUDA_REGISTRATION_CONFIG := $(ARKOUDA_PROJECT_DIR)/registration-config.json
+endif
+
 CHPL_FLAGS += -lhdf5 -lhdf5_hl -lzmq -liconv -lidn2 -lparquet -larrow
 
 ARROW_READ_FILE_NAME += $(ARKOUDA_SOURCE_DIR)/parquet/ReadParquet
@@ -380,9 +385,9 @@ endif
 SERVER_CONFIG_SCRIPT=$(ARKOUDA_SOURCE_DIR)/parseServerConfig.py
 # This is the main compilation statement section
 $(ARKOUDA_MAIN_MODULE): check-deps $(ARROW_UTIL_O) $(ARROW_READ_O) $(ARROW_WRITE_O) $(ARKOUDA_SOURCES) $(ARKOUDA_MAKEFILES)
-	$(eval MOD_GEN_OUT=$(shell python3 $(SERVER_CONFIG_SCRIPT) $(ARKOUDA_CONFIG_FILE) $(ARKOUDA_SOURCE_DIR)))
+	$(eval MOD_GEN_OUT=$(shell python3 $(SERVER_CONFIG_SCRIPT) $(ARKOUDA_CONFIG_FILE) $(ARKOUDA_REGISTRATION_CONFIG) $(ARKOUDA_SOURCE_DIR)))
 
-	$(CHPL) $(CHPL_DEBUG_FLAGS) $(PRINT_PASSES_FLAGS) $(REGEX_MAX_CAPTURES_FLAG) $(OPTIONAL_SERVER_FLAGS) $(CHPL_FLAGS_WITH_VERSION) $(CHPL_COMPAT_FLAGS) $(ARKOUDA_MAIN_SOURCE) $(ARKOUDA_COMPAT_MODULES) $(ARKOUDA_SERVER_USER_MODULES) $(MOD_GEN_OUT) $(ARKOUDA_RW_DEFAULT_FLAG) -I$(ARKOUDA_SOURCE_DIR)/parquet -o $@
+	$(CHPL) $(CHPL_DEBUG_FLAGS) $(PRINT_PASSES_FLAGS) $(REGEX_MAX_CAPTURES_FLAG) $(OPTIONAL_SERVER_FLAGS) $(CHPL_FLAGS_WITH_VERSION) $(CHPL_COMPAT_FLAGS) $(ARKOUDA_MAIN_SOURCE) $(ARKOUDA_COMPAT_MODULES) $(ARKOUDA_SERVER_USER_MODULES) $(MOD_GEN_OUT) $(ARKOUDA_RW_DEFAULT_FLAG) $(ARKOUDA_REGISTRY_DIR)/Commands.chpl -I$(ARKOUDA_SOURCE_DIR)/parquet -o $@
 
 CLEAN_TARGETS += arkouda-clean
 .PHONY: arkouda-clean
@@ -568,6 +573,14 @@ benchmark:
 version:
 	@echo $(VERSION);
 
+
+#####################
+#### register.mk ####
+#####################
+
+register-commands:
+	python3 $(ARKOUDA_REGISTRY_DIR)/register_commands.py $(ARKOUDA_REGISTRATION_CONFIG) $(ARKOUDA_CONFIG_FILE) $(ARKOUDA_SOURCE_DIR)
+
 #####################
 #### Epilogue.mk ####
 #####################

PROTO_tests/tests/client_test.py

@@ -137,7 +137,7 @@ def test_client_get_server_commands(self):
         sample of commands.
         """
         cmds = ak.client.get_server_commands()
-        for cmd in ["connect", "array", "create1D", "tondarray1D", "info", "str"]:
+        for cmd in ["connect", "create1D", "tondarray1D", "info", "str"]:
             assert cmd in cmds
 
     def test_client_array_dim_cmd_error(self):

arkouda/array_api/_typing.py

@@ -44,11 +44,9 @@
 
 
 class NestedSequence(Protocol[_T_co]):
-    def __getitem__(self, key: int, /) -> _T_co | NestedSequence[_T_co]:
-        ...
+    def __getitem__(self, key: int, /) -> _T_co | NestedSequence[_T_co]: ...
 
-    def __len__(self, /) -> int:
-        ...
+    def __len__(self, /) -> int: ...
 
 
 Device = Literal["cpu"]
@@ -75,5 +73,4 @@ def __len__(self, /) -> int:
 
 
 class SupportsDLPack(Protocol):
-    def __dlpack__(self, /, *, stream: None = ...) -> PyCapsule:
-        ...
+    def __dlpack__(self, /, *, stream: None = ...) -> PyCapsule: ...

arkouda/array_api/array_object.py

@@ -122,7 +122,9 @@ def item(self):
         if self._has_single_elem():
             return self._array[0]
         else:
-            raise ValueError("Can only convert an array with one element to a Python scalar")
+            raise ValueError(
+                "Can only convert an array with one element to a Python scalar"
+            )
 
     def transpose(self, axes: Optional[Tuple[int, ...]] = None):
         """
@@ -131,6 +133,7 @@ def transpose(self, axes: Optional[Tuple[int, ...]] = None):
         For axes=None, reverse all the dimensions of the array.
         """
         from .manipulation_functions import permute_dims
+
         if axes is None:
             _axes = tuple(range(self.ndim - 1, -1, -1))
         else:
@@ -185,7 +188,9 @@ def chunk_info(self: Array, /) -> List[List[int]]:
         """
         import json
 
-        return json.loads(ak.generic_msg(cmd=f"chunkInfo{self.ndim}D", args={"array": self._array}))
+        return json.loads(
+            ak.generic_msg(cmd=f"chunkInfo{self.ndim}D", args={"array": self._array})
+        )
 
     def __array__(self, dtype: None | np.dtype[Any] = None):
         """
@@ -459,7 +464,7 @@ def __getitem__(
                 if isinstance(kt, Array):
                     if kt.size == 1 or kt.shape == ():
                         k.append(kt._array[0])
-                    else :
+                    else:
                         k.append(kt._array)
                 elif isinstance(kt, np.ndarray):
                     k.append(asarray(kt)._array)
@@ -504,7 +509,9 @@ def __index__(self: Array, /) -> int:
             if isinstance(s, int):
                 return s
             else:
-                raise TypeError("Only integer arrays can be converted to a Python integer")
+                raise TypeError(
+                    "Only integer arrays can be converted to a Python integer"
+                )
         else:
             raise ValueError("cannot convert non-scalar array to int")
 
@@ -607,9 +614,9 @@ def __pow__(self: Array, other: Union[int, float, Array], /) -> Array:
         Compute the power of this array by another array or scalar.
         """
         if isinstance(other, (int, float)):
-            return Array._new(self._array ** other)
+            return Array._new(self._array**other)
         else:
-            return Array._new(self._array ** other._array)
+            return Array._new(self._array**other._array)
 
     def __rshift__(self: Array, other: Union[int, Array], /) -> Array:
         """
@@ -842,9 +849,9 @@ def __rpow__(self: Array, other: Union[int, float, Array], /) -> Array:
         Compute the power of another array or scalar by this array.
         """
         if isinstance(other, (int, float)):
-            return Array._new(other ** self._array)
+            return Array._new(other**self._array)
         else:
-            return Array._new(other._array ** self._array)
+            return Array._new(other._array**self._array)
 
     def __irshift__(self: Array, other: Union[int, Array], /) -> Array:
         """
@@ -972,7 +979,9 @@ def T(self) -> Array:
 
 def implements_numpy(numpy_function):
     """Register an __array_function__ implementation for MyArray objects."""
+
     def decorator(func):
         HANDLED_FUNCTIONS[numpy_function] = func
         return func
+
     return decorator

arkouda/array_api/creation_functions.py

@@ -82,7 +82,9 @@ def asarray(
     elif isinstance(obj, np.ndarray):
         return Array._new(_to_pdarray(obj, dt=dtype))
     else:
-        raise ValueError("asarray not implemented for 'NestedSequence' or 'SupportsBufferProtocol'")
+        raise ValueError(
+            "asarray not implemented for 'NestedSequence' or 'SupportsBufferProtocol'"
+        )
 
 
 def arange(