refactor gradient calculations to use DerivativeSurfaceMesh

[tylerflex]

Part 1 of an effort to generalize the VJP calculations so the Geometry and Medium just need to construct a single DerivativeSurfaceMesh for every field. The DerivativeInfo then has a method that computes derivatives given a DerivativeSurfaceMesh.

[tylerflex]

Let's also get this one in. Looks like I have a good amount of merge conflicts to sort out first, but I'll work on that today and re-test

[tylerflex]

did a bit of testing of this and it seems like it still works. going to leave it unsquashed for now but should be ready to go after reviews @momchil-flex @yaugenst-flex

[tylerflex]

I just noticed an issue with the epsilon grabbing from the other PR after the rebase, so don't merge this yet as I'll fix it tomorrow morning

[tylerflex]

alright fixed it, turned out when there were inf in the PolySlab.slab_bounds or Cylinder.length, some weird things would happen after the rebase. These are handled properly now.

[yaugenst-flex]

Really nice to have this 🎉
Only some minor comments. Biggest issue for me is, this is basically a merge from develop right? I think we should do that separately, there are a lot of unrelated changes in here.

[momchil-flex]

Yeah @tylerflex I think you need to drop the first 6 commits (then maybe clean up the rest?)

[tylerflex]

ok I cleaned up the commits into just 2, addressed Yannick's comments, but now ruff is complaining (but not locally, so that's a bit weird)

[momchil-flex]

Whenever changes are made to the frontend autograd components we need to be careful about the backend too. Seems like this needs to be updated and I'm not sure how?

[tylerflex]

ah, right. I think the fix is to pass the sim_fields_keys into the make_adjoint_monitors. Since you have the optimizer, that would mean getting

sim = optimizer.design.simulation
traced_fields_sim = setup_run(simulation=sim)
sim_fields_keys = list(traced_fields_sim.keys())

[momchil-flex]

I don't think setup_run will work if this is not called inside a grad call?

[tylerflex]

not sure, guess you could just do

    return simulation.strip_traced_fields(
        include_untraced_data_arrays=False, starting_path=("structures",)
    )

which is what setup run does.

[momchil-flex]

But the point is that there are no traced fields since it's not a simulation called in grad no?

[momchil-flex]

Basically what we're trying to do there is get the gradient monitors from the static definition of the InverseDesign object.

[tylerflex]

ah ok. yea I guess we dont know. but we can effectively "fake" it then, because we know it's a Box with a CustomMedium and we need the permittivity data.

So I think one approach could just be to pass

field_keys=(f'structures/{(len(sim.structures) - 1}/medium/permittivity', )

into the make_adjoint_monitors call. Or you can just make the monitors manually since again we know it's just a box so it needs a similarly sized field and permittivity monitor

[momchil-flex]

Using the first approach, works for now, obviously should be dealt with better in the future..