Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make arkouda server configurable w.r.t. supported types #2876

Merged

Conversation

jeremiah-corrado
Copy link
Contributor

@jeremiah-corrado jeremiah-corrado commented Dec 7, 2023

Adds a new serverConfig.json file that contains options for specifying maximum array dimensionality (from #2829), as well as some new options to specify which scalar types should be supported by the server.

With the new options in place, Arkouda will be able to improve its array_api compliance, where all of numpy's scalar data types are to be supported. This will also allow us to improve the "Arkouda as a numpy drop-in replacement" story when the server is compiled to support all the scalar types. When support for a given type is not desired, it will be ignored at compile time (using some config param values), leading to faster compilation and a smaller binary.

Other improvements:

  • the randIntMsg, castMsg, createMsg, and setMsg procedures will now support all the numpy scalar types. This was implement as a proof-of-concept, with the intention of incrementally expanding type-configurability support in later PRs
  • command dispatching now detects when an unrecognized command was not in the table because (1) the relevant command doesn't support ND arrays, or (2) the server was not compiled with a high enough maximum array dimensionality to support the command. In either of these cases, a more specific error message is returned to the client.

@jeremiah-corrado jeremiah-corrado marked this pull request as ready for review December 13, 2023 18:45
Copy link
Member

@stress-tess stress-tess left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Sorry for the delay in reviewing

Do you mind providing your compilation times on master and with this PR just as a quick check that compilation times don't seem to go up

src/CastMsg.chpl Outdated Show resolved Hide resolved
src/ServerConfig.chpl Outdated Show resolved Hide resolved
@jeremiah-corrado
Copy link
Contributor Author

@pierce314159, I had to make some changes to castMsg to make it more obvious that code paths from unsupported types should not be explored (i.e., check that the type is supported before calling into castGenSymEntry, instead of doing the check in castGenSymEntry).

With that change, the compilation times look about the same on my machine when compared with the master branch for the llvm and clang backends:

master this branch
llvm 515.970 515.502
clang 463.867 466.128
llvm - master...
Pass               Name                   Main    Check    Clean     Time    %     Accum    %
---- ---------------------------------  -------  -------  -------  ------- -----  ------- -----
     startup                              0.000    0.000    0.000    0.000   0.0    0.000   0.0
     init                                 0.285    0.000    0.000    0.285   0.1    0.285   0.1
   1 parseAndConvertUast                  9.868    0.000    0.015    9.883   1.9   10.168   2.0
   2 checkUast                            0.126    0.003    0.013    0.141   0.0   10.309   2.0
   3 readExternC                          0.000    0.000    0.013    0.013   0.0   10.322   2.0
   4 expandExternArrayCalls               0.001    0.000    0.013    0.014   0.0   10.336   2.0
   5 cleanup                              0.140    0.000    0.013    0.153   0.0   10.489   2.0
   6 scopeResolve                         1.584    0.001    0.015    1.600   0.3   12.089   2.3
   7 flattenClasses                       0.000    0.000    0.013    0.013   0.0   12.103   2.3
   8 normalize                            1.610    0.000    0.036    1.646   0.3   13.748   2.7
   9 checkNormalized                      0.006    0.000    0.033    0.039   0.0   13.787   2.7
  10 buildDefaultFunctions                0.417    0.000    0.048    0.465   0.1   14.252   2.8
  11 createTaskFunctions                  0.036    0.006    0.047    0.089   0.0   14.341   2.8
  12 resolve                            145.141    0.089    0.774  146.004  28.3  160.344  31.1
  13 resolveIntents                       0.120    0.010    0.155    0.285   0.1  160.630  31.1
  14 checkResolved                        0.208    0.000    0.154    0.362   0.1  160.992  31.2
  15 replaceArrayAccessesWithRefTemps     0.000    0.010    0.154    0.165   0.0  161.157  31.2
  16 flattenFunctions                     0.288    0.010    0.154    0.452   0.1  161.609  31.3
  17 cullOverReferences                   0.533    0.112    0.157    0.802   0.2  162.411  31.5
  18 lowerErrorHandling                   0.396    0.000    0.162    0.558   0.1  162.968  31.6
  19 callDestructors                     35.042    0.000    0.203   35.244   6.8  198.213  38.4
  20 lowerIterators                       8.883    0.044    0.491    9.418   1.8  207.631  40.2
  21 parallel                             6.510    0.042    0.396    6.949   1.3  214.579  41.6
  22 prune                                1.088    0.044    0.608    1.740   0.3  216.319  41.9
  23 bulkCopyRecords                      0.006    0.030    0.260    0.296   0.1  216.615  42.0
  24 removeUnnecessaryAutoCopyCalls       0.029    0.030    0.261    0.320   0.1  216.935  42.0
  25 inlineFunctions                     14.292    0.074    0.827   15.193   2.9  232.128  45.0
  26 scalarReplace                        3.499    0.062    0.603    4.164   0.8  236.292  45.8
  27 refPropagation                       4.155    0.060    0.550    4.764   0.9  241.056  46.7
  28 copyPropagation                     31.005    0.049    0.472   31.525   6.1  272.581  52.8
  29 deadCodeElimination                  4.232    0.038    0.245    4.515   0.9  277.096  53.7
  30 removeEmptyRecords                   0.000    0.039    0.236    0.275   0.1  277.371  53.8
  31 localizeGlobals                      0.500    0.038    0.238    0.775   0.2  278.146  53.9
  32 loopInvariantCodeMotion              7.996    0.039    0.242    8.276   1.6  286.423  55.5
  33 prune2                               0.841    0.039    0.262    1.142   0.2  287.565  55.7
  34 returnStarTuplesByRefArgs            0.190    0.040    0.246    0.476   0.1  288.040  55.8
  35 insertWideReferences                 0.231    0.040    0.240    0.510   0.1  288.550  55.9
  36 optimizeOnClauses                    1.422    0.039    0.240    1.701   0.3  290.251  56.3
  37 addInitCalls                         0.004    0.040    0.239    0.283   0.1  290.534  56.3
  38 insertLineNumbers                    0.721    0.039    0.239    0.999   0.2  291.533  56.5
  39 denormalize                          6.027    0.000    0.296    6.323   1.2  297.856  57.7
  40 codegen                            127.130    0.000    0.227  127.357  24.7  425.213  82.4
  41 makeBinary                          89.589    0.000    1.167   90.756  17.6  515.970 100.0
     driverCleanup                        0.000    0.000    0.000    0.000   0.0  515.970 100.0

     total time                         504.149    1.068   10.753  515.970

total time (front end) : 214.141 seconds
total time (middle end) :  83.419 seconds
total time (back end) : 218.409 seconds
total time : 515.970 seconds
clang - master...
Pass               Name                   Main    Check    Clean     Time    %     Accum    %
---- ---------------------------------  -------  -------  -------  ------- -----  ------- -----
     startup                              0.000    0.000    0.000    0.000   0.0    0.000   0.0
     init                                 0.287    0.000    0.000    0.287   0.1    0.287   0.1
   1 parseAndConvertUast                  9.607    0.000    0.013    9.620   2.1    9.908   2.1
   2 checkUast                            0.121    0.003    0.012    0.136   0.0   10.044   2.2
   3 readExternC                          0.000    0.000    0.012    0.012   0.0   10.056   2.2
   4 expandExternArrayCalls               0.001    0.000    0.012    0.013   0.0   10.069   2.2
   5 cleanup                              0.138    0.000    0.012    0.150   0.0   10.219   2.2
   6 scopeResolve                         1.586    0.001    0.015    1.602   0.3   11.822   2.5
   7 flattenClasses                       0.000    0.000    0.013    0.013   0.0   11.834   2.6
   8 normalize                            1.597    0.000    0.034    1.632   0.4   13.466   2.9
   9 checkNormalized                      0.005    0.000    0.031    0.037   0.0   13.503   2.9
  10 buildDefaultFunctions                0.416    0.000    0.046    0.461   0.1   13.964   3.0
  11 createTaskFunctions                  0.036    0.006    0.045    0.086   0.0   14.050   3.0
  12 resolve                            145.702    0.085    0.761  146.548  31.6  160.599  34.6
  13 resolveIntents                       0.118    0.010    0.154    0.282   0.1  160.881  34.7
  14 checkResolved                        0.208    0.000    0.153    0.361   0.1  161.242  34.8
  15 replaceArrayAccessesWithRefTemps     0.000    0.011    0.153    0.164   0.0  161.406  34.8
  16 flattenFunctions                     0.285    0.010    0.153    0.448   0.1  161.855  34.9
  17 cullOverReferences                   0.535    0.112    0.156    0.802   0.2  162.657  35.1
  18 lowerErrorHandling                   0.394    0.000    0.162    0.555   0.1  163.212  35.2
  19 callDestructors                     35.115    0.000    0.201   35.316   7.6  198.528  42.8
  20 lowerIterators                       8.793    0.042    0.492    9.328   2.0  207.856  44.8
  21 parallel                             6.475    0.041    0.395    6.911   1.5  214.767  46.3
  22 prune                                1.085    0.042    0.616    1.743   0.4  216.510  46.7
  23 bulkCopyRecords                      0.007    0.029    0.260    0.295   0.1  216.805  46.7
  24 removeUnnecessaryAutoCopyCalls       0.029    0.028    0.261    0.318   0.1  217.122  46.8
  25 inlineFunctions                     14.315    0.074    0.823   15.212   3.3  232.334  50.1
  26 scalarReplace                        3.490    0.065    0.596    4.150   0.9  236.484  51.0
  27 refPropagation                       4.103    0.055    0.544    4.703   1.0  241.187  52.0
  28 copyPropagation                     31.020    0.049    0.473   31.542   6.8  272.729  58.8
  29 deadCodeElimination                  4.238    0.039    0.245    4.521   1.0  277.250  59.8
  30 removeEmptyRecords                   0.000    0.039    0.236    0.275   0.1  277.525  59.8
  31 localizeGlobals                      0.505    0.038    0.238    0.781   0.2  278.306  60.0
  32 loopInvariantCodeMotion              8.038    0.039    0.242    8.319   1.8  286.625  61.8
  33 prune2                               0.845    0.039    0.258    1.142   0.2  287.767  62.0
  34 returnStarTuplesByRefArgs            0.194    0.039    0.242    0.476   0.1  288.243  62.1
  35 insertWideReferences                 0.225    0.039    0.235    0.500   0.1  288.742  62.2
  36 optimizeOnClauses                    1.403    0.039    0.236    1.677   0.4  290.420  62.6
  37 addInitCalls                         0.004    0.040    0.236    0.280   0.1  290.700  62.7
  38 insertLineNumbers                    0.724    0.039    0.238    1.001   0.2  291.701  62.9
  39 denormalize                          6.037    0.000    0.295    6.331   1.4  298.032  64.2
  40 codegen                             11.323    0.000    0.218   11.541   2.5  309.573  66.7
  41 makeBinary                         152.992    0.000    1.302  154.294  33.3  463.867 100.0
     driverCleanup                        0.000    0.000    0.000    0.000   0.0  463.867 100.0

     total time                         451.997    1.053   10.817  463.867

total time (front end) : 214.331 seconds
total time (middle end) :  83.407 seconds
total time (back end) : 166.130 seconds
total time : 463.867 seconds
llvm - this branch...
Pass               Name                   Main    Check    Clean     Time    %     Accum    %
---- ---------------------------------  -------  -------  -------  ------- -----  ------- -----
     startup                              0.000    0.000    0.000    0.000   0.0    0.000   0.0
     init                                 0.274    0.000    0.000    0.274   0.1    0.274   0.1
   1 parseAndConvertUast                  9.845    0.000    0.015    9.859   1.9   10.133   2.0
   2 checkUast                            0.126    0.003    0.012    0.141   0.0   10.274   2.0
   3 readExternC                          0.000    0.000    0.012    0.012   0.0   10.287   2.0
   4 expandExternArrayCalls               0.001    0.000    0.013    0.014   0.0   10.301   2.0
   5 cleanup                              0.142    0.000    0.012    0.154   0.0   10.455   2.0
   6 scopeResolve                         1.640    0.001    0.015    1.657   0.3   12.112   2.3
   7 flattenClasses                       0.000    0.000    0.013    0.013   0.0   12.125   2.4
   8 normalize                            1.621    0.000    0.035    1.657   0.3   13.782   2.7
   9 checkNormalized                      0.006    0.000    0.033    0.039   0.0   13.821   2.7
  10 buildDefaultFunctions                0.417    0.000    0.048    0.465   0.1   14.286   2.8
  11 createTaskFunctions                  0.036    0.006    0.046    0.088   0.0   14.374   2.8
  12 resolve                            143.045    0.086    0.767  143.899  27.9  158.273  30.7
  13 resolveIntents                       0.120    0.010    0.155    0.285   0.1  158.558  30.8
  14 checkResolved                        0.210    0.000    0.155    0.364   0.1  158.922  30.8
  15 replaceArrayAccessesWithRefTemps     0.000    0.010    0.155    0.166   0.0  159.088  30.9
  16 flattenFunctions                     0.297    0.010    0.155    0.462   0.1  159.550  31.0
  17 cullOverReferences                   0.542    0.113    0.157    0.811   0.2  160.361  31.1
  18 lowerErrorHandling                   0.400    0.000    0.163    0.563   0.1  160.925  31.2
  19 callDestructors                     37.442    0.000    0.205   37.648   7.3  198.572  38.5
  20 lowerIterators                       8.906    0.044    0.498    9.448   1.8  208.020  40.4
  21 parallel                             6.561    0.041    0.406    7.009   1.4  215.029  41.7
  22 prune                                1.104    0.042    0.610    1.757   0.3  216.785  42.1
  23 bulkCopyRecords                      0.007    0.029    0.259    0.295   0.1  217.080  42.1
  24 removeUnnecessaryAutoCopyCalls       0.029    0.029    0.260    0.317   0.1  217.397  42.2
  25 inlineFunctions                     14.531    0.074    0.839   15.444   3.0  232.842  45.2
  26 scalarReplace                        3.750    0.063    0.607    4.419   0.9  237.261  46.0
  27 refPropagation                       4.113    0.054    0.541    4.708   0.9  241.969  46.9
  28 copyPropagation                     30.615    0.047    0.474   31.136   6.0  273.105  53.0
  29 deadCodeElimination                  4.253    0.037    0.243    4.533   0.9  277.638  53.9
  30 removeEmptyRecords                   0.000    0.037    0.233    0.270   0.1  277.908  53.9
  31 localizeGlobals                      0.501    0.037    0.235    0.772   0.1  278.680  54.1
  32 loopInvariantCodeMotion              8.104    0.038    0.239    8.381   1.6  287.061  55.7
  33 prune2                               0.834    0.038    0.254    1.126   0.2  288.187  55.9
  34 returnStarTuplesByRefArgs            0.190    0.038    0.240    0.468   0.1  288.654  56.0
  35 insertWideReferences                 0.223    0.037    0.233    0.493   0.1  289.147  56.1
  36 optimizeOnClauses                    1.408    0.037    0.232    1.678   0.3  290.825  56.4
  37 addInitCalls                         0.004    0.038    0.233    0.275   0.1  291.100  56.5
  38 insertLineNumbers                    0.720    0.037    0.235    0.993   0.2  292.093  56.7
  39 denormalize                          6.050    0.000    0.289    6.339   1.2  298.431  57.9
  40 codegen                            127.408    0.000    0.210  127.618  24.8  426.049  82.6
  41 makeBinary                          88.271    0.000    1.181   89.452  17.4  515.502 100.0
     driverCleanup                        0.000    0.000    0.000    0.000   0.0  515.502 100.0

     total time                         503.747    1.037   10.718  515.502

total time (front end) : 214.582 seconds
total time (middle end) :  83.561 seconds
total time (back end) : 217.359 seconds
total time : 515.502 seconds
clang - this branch...
new branch (clang)
Pass               Name                   Main    Check    Clean     Time    %     Accum    %
---- ---------------------------------  -------  -------  -------  ------- -----  ------- -----
     startup                              0.000    0.000    0.000    0.000   0.0    0.000   0.0
     init                                 0.302    0.000    0.000    0.302   0.1    0.302   0.1
   1 parseAndConvertUast                  9.897    0.000    0.014    9.911   2.1   10.213   2.2
   2 checkUast                            0.120    0.003    0.012    0.135   0.0   10.348   2.2
   3 readExternC                          0.000    0.000    0.012    0.012   0.0   10.360   2.2
   4 expandExternArrayCalls               0.001    0.000    0.012    0.013   0.0   10.373   2.2
   5 cleanup                              0.138    0.000    0.011    0.150   0.0   10.522   2.3
   6 scopeResolve                         1.641    0.001    0.014    1.657   0.4   12.179   2.6
   7 flattenClasses                       0.000    0.000    0.012    0.012   0.0   12.191   2.6
   8 normalize                            1.637    0.000    0.033    1.671   0.4   13.862   3.0
   9 checkNormalized                      0.005    0.000    0.030    0.035   0.0   13.897   3.0
  10 buildDefaultFunctions                0.418    0.000    0.044    0.462   0.1   14.359   3.1
  11 createTaskFunctions                  0.036    0.005    0.043    0.084   0.0   14.443   3.1
  12 resolve                            143.728    0.076    0.744  144.548  31.0  158.991  34.1
  13 resolveIntents                       0.116    0.010    0.151    0.277   0.1  159.268  34.2
  14 checkResolved                        0.208    0.000    0.154    0.362   0.1  159.630  34.2
  15 replaceArrayAccessesWithRefTemps     0.000    0.012    0.152    0.163   0.0  159.793  34.3
  16 flattenFunctions                     0.293    0.010    0.151    0.454   0.1  160.247  34.4
  17 cullOverReferences                   0.535    0.111    0.153    0.800   0.2  161.047  34.5
  18 lowerErrorHandling                   0.392    0.000    0.159    0.551   0.1  161.598  34.7
  19 callDestructors                     37.475    0.000    0.196   37.671   8.1  199.269  42.7
  20 lowerIterators                       8.780    0.039    0.487    9.305   2.0  208.574  44.7
  21 parallel                             6.573    0.036    0.393    7.002   1.5  215.576  46.2
  22 prune                                1.089    0.036    0.608    1.733   0.4  217.309  46.6
  23 bulkCopyRecords                      0.006    0.025    0.258    0.289   0.1  217.599  46.7
  24 removeUnnecessaryAutoCopyCalls       0.029    0.025    0.257    0.311   0.1  217.910  46.7
  25 inlineFunctions                     14.520    0.070    0.828   15.419   3.3  233.329  50.1
  26 scalarReplace                        3.942    0.064    0.608    4.614   1.0  237.943  51.0
  27 refPropagation                       4.129    0.051    0.536    4.717   1.0  242.660  52.1
  28 copyPropagation                     30.676    0.046    0.465   31.187   6.7  273.847  58.7
  29 deadCodeElimination                  4.254    0.036    0.241    4.531   1.0  278.379  59.7
  30 removeEmptyRecords                   0.000    0.036    0.232    0.268   0.1  278.647  59.8
  31 localizeGlobals                      0.498    0.036    0.234    0.767   0.2  279.414  59.9
  32 loopInvariantCodeMotion              8.040    0.037    0.237    8.314   1.8  287.728  61.7
  33 prune2                               0.835    0.037    0.253    1.124   0.2  288.852  62.0
  34 returnStarTuplesByRefArgs            0.189    0.037    0.237    0.463   0.1  289.315  62.1
  35 insertWideReferences                 0.221    0.037    0.231    0.489   0.1  289.804  62.2
  36 optimizeOnClauses                    1.401    0.037    0.231    1.669   0.4  291.473  62.5
  37 addInitCalls                         0.004    0.037    0.231    0.273   0.1  291.746  62.6
  38 insertLineNumbers                    0.715    0.037    0.234    0.986   0.2  292.732  62.8
  39 denormalize                          6.056    0.000    0.288    6.344   1.4  299.076  64.2
  40 codegen                             11.361    0.000    0.217   11.578   2.5  310.654  66.6
  41 makeBinary                         154.251    0.000    1.222  155.474  33.4  466.128 100.0
     driverCleanup                        0.000    0.000    0.000    0.000   0.0  466.128 100.0

     total time                         454.516    0.988   10.624  466.128

total time (front end) : 215.147 seconds
total time (middle end) :  83.640 seconds
total time (back end) : 167.340 seconds
total time : 466.128 seconds

Copy link
Contributor

@bmcdonald3 bmcdonald3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a couple of comments that you don't necessarily need to address, just things to think about.

ServerModules.cfg Show resolved Hide resolved
arkouda/dtypes.py Outdated Show resolved Hide resolved
serverConfig.json Show resolved Hide resolved
src/CastMsg.chpl Show resolved Hide resolved
@bmcdonald3 bmcdonald3 enabled auto-merge January 29, 2024 17:19
@bmcdonald3 bmcdonald3 added this pull request to the merge queue Jan 29, 2024
Merged via the queue into Bears-R-Us:master with commit eb64c40 Jan 29, 2024
12 checks passed
@jeremiah-corrado jeremiah-corrado deleted the module-config-improvements branch February 13, 2024 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants