Fill out more of the EFIT mapping to MDS+ #255

torrinba · 2023-06-29T20:44:51Z

This may not be 100% complete, but should be pretty close.

torrinba · 2023-06-29T20:50:08Z

I am leaving it up to you to test this out @AreWeDreaming

There several areas that need more attention still

constraints.pressure.:.position.psi needs to have an absolute value applied (some additional logic could expand the applicability further)
constraints.pf_current needs to have e-coils, f-coils, and a-coils appended (I just put slashes for now so this is most likely why the regression build fails)
constraints.mse_polarisation_angle needs to have an atan applied
constraints.chi_squared_total is an OMAS extension that is not yet part of IMAS (up to you whether it should be included now or not)
constraints.j_tor needs to be converted to OMAS units (some additional logic could expand the applicability of the position.psi as well)

Changes will be made in EFIT to make these usable

This includes nearly all of the additional EFIT variables in MDS+ that I know how to mapp to IMAS.

There were other name changes I had missed. This takes care of everything that comes out of EFIT now.

AreWeDreaming · 2023-06-30T20:32:01Z

constraints.pressure.:.position.psi needs to have an absolute value applied (some additional logic could expand the applicability further)

constraints.pf_current needs to have e-coils, f-coils, and a-coils appended (I just put slashes for now so this is most likely why the regression build fails)

constraints.mse_polarisation_angle needs to have an atan applied

constraints.chi_squared_total is an OMAS extension that is not yet part of IMAS (up to you whether it should be included now or not)

constraints.j_tor needs to be converted to OMAS units (some additional logic could expand the applicability of the position.psi as well)

I think I got this covered in my next commit (not yet pushed because it needs to be tested)

torrinba · 2023-06-30T23:40:32Z

I also forgot that the constraints.j_tor will need to be added as an OMAS extension until the new IMAS version gets released and updated. Hopefully you resolved this already, but if not I can help with that

AreWeDreaming · 2023-07-17T22:17:46Z

After some modifications of OMAS I got most of the fields to load, but the following do not have data for my test case 17408219 created with CAKE. Notably I didn't test all of these but rather these are the ones I get MDS+ exception for. The first one ZSEPS is kinda weird because RSEP is set correctly.

equilibrium.time_slice.:.boundary.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.DSEP)/100.
equilibrium.time_slice.:.boundary_separatrix.geometric_axis.z: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZOUT)/100.
equilibrium.time_slice.:.boundary_separatrix.strike_point.1.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSID/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSID/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.1.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSID/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSID/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.2.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSOD/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSOD/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.2.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSOD/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSOD/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.3.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSIU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSIU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.3.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSIU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSIU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.4.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSOU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSOU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.4.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSOU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSOU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.constraints.bpol_probe.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGMPI)
equilibrium.time_slice.:.constraints.diamagnetic_flux.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIDFLUX)
equilibrium.time_slice.:.constraints.diamagnetic_flux.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTDIA)
equilibrium.time_slice.:.constraints.flux_loop.:.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.SAISI)
equilibrium.time_slice.:.constraints.flux_loop.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGSIL)
equilibrium.time_slice.:.constraints.ip.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIPASMA)
equilibrium.time_slice.:.constraints.ip.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGPASMA)
equilibrium.time_slice.:.constraints.ip.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTPASMA)
equilibrium.time_slice.:.constraints.j_tor.:.measured: data(\{EFIT_tree}::TOP.MEASUREMENTS.VZEROJ)*data(\{EFIT_tree}::TOP.RESULTS.GEQDSK.CPASMA)/(data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.AREA)/10000.)
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.chi_squared: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.CHIECC','\{EFIT_tree}::TOP.MEASUREMENTS.CHIFCC')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pf_current.:.measured_error_upper: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.SIGECC','\{EFIT_tree}::TOP.MEASUREMENTS.SIGFCC')
equilibrium.time_slice.:.constraints.pf_current.:.reconstructed: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.CECURR','\{EFIT_tree}::TOP.MEASUREMENTS.CCBRSP')
equilibrium.time_slice.:.constraints.pf_current.:.weight: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.FWTEC','\{EFIT_tree}::TOP.MEASUREMENTS.FWTFC')
equilibrium.time_slice.:.constraints.pressure.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.RPRESS','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')

AreWeDreaming · 2023-07-17T22:19:04Z

@bechtt Could you take a look at this list and note any fields that should be set by a standard EFIT run but aren't. If none of these are expected to be set then that's fine, we'll add them on CAKE side of things.

AreWeDreaming · 2023-07-18T00:47:26Z

@orso82 Any idea why the following TDI expression would cause trouble:

"equilibrium.time_slice.:.convergence.iterations_n": {
    "eval2TDI": "py2tdi(nan_max_where,'size(\\{EFIT_tree}::TOP.MEASUREMENTS.CERROR)','\\{EFIT_tree}::TOP.MEASUREMENTS.CERROR',0)",
    "treename": "{EFIT_tree}"
 }

with

def nan_max_where(a, b, n):
    import numpy as np

    a = a.data()
    b = b.data()
    a[b == n] = np.NaN
    return np.nanmax(a)

MDS+ just returns an error with no info on what's actually wrong.

torrinba · 2023-07-18T19:41:37Z

After some modifications of OMAS I got most of the fields to load, but the following do not have data for my test case 17408219 created with CAKE. Notably I didn't test all of these but rather these are the ones I get MDS+ exception for. The first one ZSEPS is kinda weird because RSEP is set correctly.

Most of the issues from MEASUREMENTS were missing or conflicting names between what was setup in MDS+ for rtEFIT and the offline version of EFIT. I fixed the EFIT outputs to match, but you will have to rerun the case to see that.

The AEQDSK failures are harder to diagnose. It's possible there are are problems with efitloader. That's the only reason I can think RSEPS would be set but not ZSEPS (which I see in other cases as well)

omas/machine_mappings/_efit.json

omas/omas_core.py

omas/machine_mappings/_efit.json

torrinba · 2023-07-20T18:11:20Z

Here are the fields that are still broken

equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZEPLIM)/100.
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer_3,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pressure.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.RPRESS','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')

I fixed the first and third. I can confirm the quantities in the second and fourth are in MDS+ so I expect the problem is with the TDI (possibly the dimensions)

AreWeDreaming · 2023-07-20T18:13:07Z

I'll retest.

I fixed the first and third. I can confirm the quantities in the second and fourth are in MDS+ so I expect the problem is with the TDI (possibly the dimensions)
My favorite kinda error.

AreWeDreaming · 2023-07-20T20:38:00Z

There is no more broken fields now for @bechtt's test case.

AreWeDreaming · 2023-07-20T20:42:46Z

@orso82 If you could take a look at my changes to OMAS machine it would be much appreciated. The main change is that you can now load all available mappings by using a wildcard in the location.

torrinba · 2023-07-20T20:57:25Z

@AreWeDreaming do you understand why regression is failing? It passed prior to 02fef13

AreWeDreaming · 2023-07-20T20:59:53Z

I'll have to run the regression tests locally. The error before was related to OMFIT things, now it doesn't seem to give us any information at all. The lines that were changed in the hash you pointed out don't exist anymore...

orso82 · 2023-07-20T23:22:45Z

Looks good to me

I am happy to see that someone else other than me was able to generate the updated IMAS JSON data dictionary files! 👍

smithsp · 2023-07-27T18:56:24Z

I may have merged https://github.com/gafusion/OMFIT-source/pull/6708 prematurely, since this one is not passing regressions. It was passing regression on the OMFIT side...

smithsp · 2023-07-27T19:44:35Z

If I run make test on my own computer, I get

======================================================================
ERROR: test_omas_dynamic_machine (test_omas_examples.TestOmasExamples)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/smithsp/omas/omas/omas_core.py", line 1303, in __getitem__
    value = self.dynamic.__getitem__(location)
  File "/Users/smithsp/omas/omas/omas_machine.py", line 984, in __getitem__
    return self.cache[o2u(key)][key]
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  [Previous line repeated 3 more times]
  File "/Users/smithsp/omas/omas/omas_core.py", line 1314, in __getitem__
    self.__setitem__(key[0], self.same_init_ods())
  File "/Users/smithsp/omas/omas/omas_core.py", line 872, in __setitem__
    raise ValueError('`%s` has no data' % location)
ValueError: `pf_active.coil.2.element.6.geometry.geometry_type` has no data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/smithsp/omas/omas/tests/test_omas_examples.py", line 143, in test_omas_dynamic_machine
    from omas.examples import omas_dynamic_machine
  File "/Users/smithsp/omas/omas/examples/omas_dynamic_machine.py", line 30, in <module>
    ods.plot_overlay(wall=True, magnetics=True, pf_active=True)
  File "/Users/smithsp/omas/omas/omas_plot.py", line 2443, in overlay
    overlay_function(ods, ax, **overlay_kw)
  File "/Users/smithsp/omas/omas/omas_plot.py", line 2748, in pf_active_overlay
    ods['pf_active.coil'][c]['element'][e]['geometry.geometry_type'], 'pf_active', ods.imas_version
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1305, in __getitem__
    raise OmasDynamicException(f'Error dynamic fetching of `{location}` for {self.dynamic.kw}: {repr(_excp)}')
omas.omas_core.OmasDynamicException: Error dynamic fetching of `pf_active.coil.2.element.6.geometry.geometry_type` for {'machine': 'd3d', 'pulse': 168830, 'options': {'EFIT_tree': 'EFIT02'}, 'branch': '', 'user_machine_mappings': None}: ValueError('`pf_active.coil.2.element.6.geometry.geometry_type` has no data')

----------------------------------------------------------------------
Ran 42 tests in 17.585s

Note that this does not seem to be the same problem as reported in the automatic tests.

torrinba · 2023-07-27T21:57:58Z

Thanks for helping test this @smithsp. We didn't make changes to any of the DIII-D machine mappings and I don't see any changes to the IMAS schema for pf_active so I don't understand why you are seeing that error

From what I can tell _common.py only sets the geometry properties for element 0, which could be related, but when I run make test on Iris I can't reproduce the issue. Instead it seemed like the only problem was that COCOs needed to be updated (last commit). Hopefully that resolves everything and this merge can go forward!

AreWeDreaming · 2023-07-28T23:25:04Z

This shouldn't be merged yet. I just implemented the changes we do have into omas-viewer and this is what the equilibrium plot looks like:

I'll try and figure out what went wrong. I think psi is just a transposition error, but I do not understand what's going on with

equilibrium.time_slice.:.constraints.pressure.:.position.psi

AreWeDreaming · 2023-07-29T00:48:39Z

Much better:

The comments do matter...

smithsp · 2023-07-31T18:41:01Z

Need to check omas_from_imas.

AreWeDreaming · 2023-08-02T18:47:43Z

Midway of testing omas_from_imas I realized how incredibly unrelated that is to this PR. I did it anyways and found no problems.
Looking for someone that could test _efit.json for non-DIII-D machine. Looped in @jmcclena to test something NSTX related.

torrinba · 2023-08-02T20:01:50Z

Midway of testing omas_from_imas I realized how incredibly unrelated that is to this PR. I did it anyways and found no problems.

It is related to the orientation of the flux matrix loaded into OMAS from the MDS+ machine mapping. There is an issue with the g-file to OMAS conversion being resolved in this PR https://github.com/gafusion/OMFIT-source/pull/6684

I'm surprised, but happy to hear that the orientation in MDS+ must be different

omas/machine_mappings/_efit.json

torrinba · 2023-08-10T01:10:13Z

@orso82 or @smithsp could you merge this if you approve? Neither Severin or I have permission to, but both agree it is ready

Fill out more of the EFIT mapping to MDS+

bc532a0

This may not be 100% complete, but should be pretty close.

torrinba added the enhancement label Jun 29, 2023

torrinba requested a review from AreWeDreaming June 29, 2023 20:44

torrinba self-assigned this Jun 29, 2023

torrinba added 4 commits June 29, 2023 16:23

Add magnetic axis field

bfa2e50

Add variables that are in the tree, but not yet being filled

a0ba816

Changes will be made in EFIT to make these usable

Extending the EFIT mapping further

ccd53ca

This includes nearly all of the additional EFIT variables in MDS+ that I know how to mapp to IMAS.

Found even more variables in MDS+

1456c2a

There were other name changes I had missed. This takes care of everything that comes out of EFIT now.

Allow ods roots to be loaded with machine mappings

5640f7e

AreWeDreaming added 9 commits July 17, 2023 17:48

This works but needs some clean-up

8a906a3

Forgot a to commit a file

01f9b06

Removed non-IMAS conform fields

00c791b

Correction to MDS field names

ee34604

Another MDS+ vs EFIT discrepancy

fca3012

Bug fixes for stack_outer

84c90f3

removed *args from stack_outer

89065a9

Forgot changes to _efit in last commit

687d4ad

Updated error handling

a73d591

torrinba commented Jul 18, 2023

View reviewed changes

omas/machine_mappings/_efit.json Outdated Show resolved Hide resolved

torrinba commented Jul 18, 2023

View reviewed changes

omas/omas_core.py Outdated Show resolved Hide resolved

torrinba commented Jul 18, 2023

View reviewed changes

omas/machine_mappings/_efit.json Outdated Show resolved Hide resolved

torrinba commented Jul 18, 2023

View reviewed changes

omas/machine_mappings/_efit.json Outdated Show resolved Hide resolved

Removed more imas non-conform mappings

44420ed

Fixed last two broken fields

e6b2227

AreWeDreaming requested a review from orso82 July 20, 2023 20:38

Removed cruft

8814983

Update COCOs

c275b12

AreWeDreaming mentioned this pull request Jul 27, 2023

Test new efit mappings for ida #257

Merged

AreWeDreaming added 2 commits July 28, 2023 01:59

More missing COCOS

8fa7c01

Cherry pick efit mapping changes from omas viewer

a82d6d4

torrinba and others added 2 commits July 29, 2023 16:45

Fix COCOs

485e3ba

The comments do matter...

Fixed messed up indentation

eda21ec

AreWeDreaming mentioned this pull request Jul 31, 2023

Omas plot w new efit mapping #258

Merged

Added sizes for all new entries

d48dd90

AreWeDreaming approved these changes Aug 9, 2023

View reviewed changes

omas/machine_mappings/_efit.json Outdated Show resolved Hide resolved

omas/machine_mappings/_efit.json Outdated Show resolved Hide resolved

AreWeDreaming merged commit 47ae79c into master Aug 12, 2023

AreWeDreaming deleted the EFIT_mdsplus_mapping branch August 12, 2023 04:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fill out more of the EFIT mapping to MDS+ #255

Fill out more of the EFIT mapping to MDS+ #255

torrinba commented Jun 29, 2023

torrinba commented Jun 29, 2023 •

edited

Loading

AreWeDreaming commented Jun 30, 2023

torrinba commented Jun 30, 2023

AreWeDreaming commented Jul 17, 2023

AreWeDreaming commented Jul 17, 2023

AreWeDreaming commented Jul 18, 2023

torrinba commented Jul 18, 2023 •

edited

Loading

torrinba commented Jul 20, 2023 •

edited

Loading

AreWeDreaming commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

torrinba commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

orso82 commented Jul 20, 2023 •

edited

Loading

smithsp commented Jul 27, 2023

smithsp commented Jul 27, 2023

torrinba commented Jul 27, 2023

AreWeDreaming commented Jul 28, 2023

AreWeDreaming commented Jul 29, 2023

smithsp commented Jul 31, 2023

AreWeDreaming commented Aug 2, 2023

torrinba commented Aug 2, 2023

torrinba commented Aug 10, 2023

Fill out more of the EFIT mapping to MDS+ #255

Fill out more of the EFIT mapping to MDS+ #255

Conversation

torrinba commented Jun 29, 2023

torrinba commented Jun 29, 2023 • edited Loading

AreWeDreaming commented Jun 30, 2023

torrinba commented Jun 30, 2023

AreWeDreaming commented Jul 17, 2023

AreWeDreaming commented Jul 17, 2023

AreWeDreaming commented Jul 18, 2023

torrinba commented Jul 18, 2023 • edited Loading

torrinba commented Jul 20, 2023 • edited Loading

AreWeDreaming commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

torrinba commented Jul 20, 2023

AreWeDreaming commented Jul 20, 2023

orso82 commented Jul 20, 2023 • edited Loading

smithsp commented Jul 27, 2023

smithsp commented Jul 27, 2023

torrinba commented Jul 27, 2023

AreWeDreaming commented Jul 28, 2023

AreWeDreaming commented Jul 29, 2023

smithsp commented Jul 31, 2023

AreWeDreaming commented Aug 2, 2023

torrinba commented Aug 2, 2023

torrinba commented Aug 10, 2023

torrinba commented Jun 29, 2023 •

edited

Loading

torrinba commented Jul 18, 2023 •

edited

Loading

torrinba commented Jul 20, 2023 •

edited

Loading

orso82 commented Jul 20, 2023 •

edited

Loading