Use fragment_stats to split chunks by cost

[ChristopherHogan]

Closes #536. Attempts to divide the cell into chunks that take a similar amount of work. The work estimate is produced by fragment_stats::cost, which was obtained from running linear regression on training data from random simulations.

The default is still to use split_by_effort, which splits the chunks evenly without taking cost into account, but setting the Simulation constructor parameter split_chunks_evenly=False will use split_by_cost. Here are some benchmark results from the following simulation (a tall skinny cell with an expensive flux plane at the top):

import argparse
import time
import meep as mp

def main(args):
    sx = 2
    sy = 30
    sz = 1
    fcen = 0.15
    df = 0.1

    src = mp.Source(mp.GaussianSource(fcen, fwidth=df), mp.Ey, mp.Vector3())

    sim = mp.Simulation(cell_size=mp.Vector3(sx, sy, sz),
                        sources=[src],
                        resolution=10,
                        split_chunks_evenly=args.split_evenly)

    flux = sim.add_flux(fcen, df, 500, mp.FluxRegion(center=mp.Vector3(0, 12), size=mp.Vector3(2, 1)))
    start = time.time()
    sim.run(until=200)
    print("bench:{},{},{}".format(args.split_evenly, mp.count_processors(), time.time() - start))

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-e', '--split-evenly', action='store_true', default=False,
                                        help='Use the old split_by_effort method to create chunks')
    args = parser.parse_args()
    main(args)

The meep::structure is now created in C++ (create_structure_and_set_materials) instead of Python. Using the geometry in structure::choose_chunkdivision would require making copies, or passing it around, both of which would require more significant changes. create_structure_and_set_materials allows the Python geometry to be converted to C++ once, then deleted when it returns.

The number of processors is divided into prime factors and the cell is split into chunks starting from the largest prime factor towards the smallest in order to achieve a more even grid split for powers of 3. Here are some examples of the improvement

Left: split_by_effort Right: split_by_cost

Cell of air with 9 processors

Cell of air with 9 processors and PML

Tall skinny cell with expensive flux plane at the top, 3 processors

(split_by_effort puts all the work on 1 processor).

Tall skinny cell with expensive flux plane at the top, 9 processors.

This can be improved by splitting in the X direction, but the gain is not above the 30% threshold so split_by_cost splits in the Y direction. Using machine learning to get the actual communication cost should improve this.

Remaining issues

1. split_by_cost doesn't work with cylindrical coordinates yet so it falls back to split_by_effort
2. split_by_cost breaks the structure::dump and structure::load features. When you run the initial simulation and dump the structure file, the chunks are split optimally based on the geometry, but when you attempt to load the file, there is no geometry passed in so the chunks are split evenly, resulting in chunk size mismatches. Need to pass a list of volumes to these functions.

[stevengj]

Seems like we should have continued to use desired_num_chunks for the split_by_effort code, rather than adjusted_num_chunks (used for the new split_by_cost code)?