Optimize remote array copies by creating a local copy of the domain when possible (off by default)

[bradcray]

This PR adds an optimization that localizes an array's domain when a copy of the array is made across locales. For example, given a pattern like:

var D = {1..100, 1..100};
var A: [D] real;
on Locales[1] {
  var B = A;
  B[i] = …;
}

B will typically share A's domain so that if D is reassigned (resized), B will reflect that. This has the downside that operations on 'A' may need to refer back to 'D' in order to get its size, bounds, etc.

However, if D is declared const, it cannot be re-assigned, so rather than referring back to the original domain, a local copy of D could be created and B could be declared over that, permitting operations on 'B' to be done without referring to a remote locale. Effectively, the compiler could translate the code into:

const D = {1..100, 1..100};
var A: [D] real;
on Locales[1] {
  const localD = D;
  var B:[D] A.eltType = A;
  B[i] = …;
}

Similarly, if 'B' is const, the same optimization can be done because we say that the domain of a const array cannot be re-assigned while the array is alive.

This PR implements this optimization in module code and adds a few tests of it. It also updates one existing test that locked in the old behavior, by testing it with and without the optimization.

The optimization is currently off by default because we are close to the 2.1 deadline and want some more time to live with it and understand the implications before merging. That said, it passed a full paratest GASNet run. I've forked off #25238 to capture the intention to enable this by default and some of the things we'd discussed checking before doing so.

Thanks to @jabraham17 for the conversation that led to the insight that this might be a simple optimization (it was) after years of having people hit this issue and be surprised by it (see issue #13213, e.g.). And to @e-kayrakli and others on the @chapel-lang/perf-team for feedback on the initial draft.

[e-kayrakli]

Just in case it helps save some time: I have bumped into different cases where .locale ended up to be a performance-killer. It has a very complex implementation including some dynamic dispatch if I am not remembering wrong. If you do time measurements or this is intended to optimize quick-running idioms, you might want to use _wide_get_locale primitive. I am doing that here: https://github.com/chapel-lang/chapel/pull/24390/files#diff-52d2477eb48a33d9e81b64d64d74a50145a1bacdd880825e267b5bd5549559afR2481-R2486. I think we should consider:

1. Improving .locale/here performance (there's discussion about this under some issues, but I can't find anything quickly. Note that here can also hurt performance)
2. Having some helpers like isLocal at least internally for cases like that where we just use primitives for fast computation.

[bradcray]

Interestingly, I ran back into this need while working on sparse matrix-matrix multiplication algorithms this past week. Specifically, in copying a remote locale's sparse array, I wanted to localize the domain as well for efficient access and realized that some minor tweaks to this PR would be a start in the right direction (though more work is required to get it working for sparse arrays…).

[bradcray]

disable the optimization if the domain is already privatized

This turns out to fall out naturally due to the check for _value, which will use the local class by default.

[bradcray]

Capturing this PR's OP here:

After hitting #13213 this afternoon and talking to Jade about it, I started wondering aloud whether it might be very easy to implement an optimization that would localize the domain of an array variable being created from scratch if the original was a remote, const domain. This quick sketch suggests maybe it is, where some remaining TODOs are:

• disable the optimization if the domain is already privatized
• make sure that the array element copy is done in bulk
• figure out what's going on with the (incorrect?) compiler warnings
• make sure it doesn't break other tests
• file an issue + future about myArr.domain.locale generating a warning? Is the answer wrong too? (seemed like it wasn't...)
• impact on:
  • sparse
  • associative
• creating copy would thwart "identical domain" check—does that matter?
  • impact on swap, say?
  • impact on array-of-arrays?
• which ops go through domain with --fast?
  • and what fields could we re-store in the array to avoid comm
• relationship with Engin's notion of optimizing arrays with anonymous domains to avoid reference counting statically

Since this is going in off-by-default, I've forked the items in this list that seem important to check before turning it on into #25238

[jabraham17]

Does it make sense to add compiledForSingleLocale() here as well? This is a param and could avoid a branch when COMM=none. Arguably not important for now and you could probably just log it as future work in #25238

[bradcray]

That's a good idea. We could even potentially have both param and non-param aspects of this const split apart and to have the conditional be something like:

if paramLocalize && localize {
  …
} else {
  …
}

s.t. if paramLocalize was false, we'd fold the conditional? I like the idea of forking this off to its own PR, primarily to keep working on things that aren't done yet and adding it to the follow-on issue if you're good with that (though of all the things on that issue, I may do it first—possibly even tomorrow if time permits).