PhysicalProperty definition (02.04.2024 discussion)

[JosePizarro3]

@JFRudzinski @ndaelman-hu @Bernadette-Mohr

It is clear that, whatever the final design of Outputs is, we need some further abstraction to store the value, errors, bins, and other id strings for our properties.

It was proposed by @JFRudzinski to define PhysicalProperty as an ArchiveSection. An alternative way would be to inherit from Quantity instead, and define a value for setting the PhysicalProperty itself.

Both options have pros and cons:

Inheriting from ArchiveSection

• We can easily extend the quantities defining a property
• It will be used altogether with Outputs and the references defined there
• I wonder whether we can automate setting such properties and let some stuff be shared (see below)

Inheriting from Quantity

• We can automate setting a PhysicalProperty with __set__(). Not sure if this is not true for ArchiveSection tho.
• Certain shapes, dimensions, units, are shared across quantities within PhysicalProperty, e.g., the units for value and value_error should be the same.
• However, it is perhaps entering in a rabbit hole of implementation which will last for months.

---

Besides these, I wonder how the actual implementation of variables as a list of potential variables [Energy, Temperature...] could look like. I have some idea to define the shape of a PhysicalProperty as:

# `shape` is passed as an attr to the PhysicalProperty or left empty for scalars (its length is related with the order of the tensor property)
# `variables` should be a list of Quantities on the form `variables = [Energy, Temperature, ElectricField...]` which is set as an attr to PhysicalProperty
dimensions = [v.shape for v in variables]
full_shape = shape + dimensions

Now, IMO, the call for a PhysicalProperty could look like:

electronic_band_gap = PhysicalProperty(
    type=np.float64,
    shape=[],
    variables=[],
    unit='joule',
    iri='http://fairmat-nfdi.eu/taxonomy/ElectronicBandGap'
    description="""
    Energy difference between the highest occupied electronic state and the lowest unoccupied electronic state.
    """
)

...

electronic_band_gap.source = 'simulation'
electronic_band_gap.label = 'DFT'
electronic_band_gap.type_name = 'indirect'
electronic_band_gap.model_system_ref = archive.data.model_system[-1]
electronic_band_gap = 1.0 * ureg.eV
# add here bins, error info when relevant?

[ndaelman-hu]

Thank you for summarizing the takeaways of yesterday's discussion. First a couple of remarks on your suggestion, then my own.

Remarks

Firstly, the interface in your suggestion mostly aligns with Quantity, so it's clear which to extend...
I don't understand the usage of variables in PhysicalProperty, though. You give the example of [Energy, Temperature...], but these are already determined by the units... Can you elaborate?

Now, in your application, why are there these properties like electronic_band_gap.source and electronic_band_gap.label? Aren't the input settings contained in the same entry. This looks like metadata for describing the file structure itself, which isn't the point of metadata at all, rather a symptom of a design flaw in the schema. You could, in this case, use a reference again.

I do want to caution against overusing references: I have serious doubts how well NOMAD can handle massive volume of them. They're more a patch for some scenarios, not a main tool. We should really ask Area D about this or perform a benchmark. In some cases, they can be avoided by grouping together Quantitys in ArchiveSection, but steering clear from any deeper nesting.

Small note: if we have the iri, we can extract the description from there.

Lastly, the extension of Quantity is indeed tricky, as this delves into meta-programming. I have some experience there, so can give it a shot.
I'm not sure about your other pros and cons. You should develop the points there better before I can really comment...

Proposal

So, my heuristic for deciding whether something goes to Quantity or ArchiveSection is where the decision is made: in the parsers / normalizers vs schema, respectively. 2 examples:

1. The parser will extract any error bars, so this is for Quantity.
2. We decide at a schema-level if a property is taken up in a SCF step. I like @JosePizarro3 's suggestion yesterday to have the (SCF) convergence accompany the final value. However, this is as simple as having 2 separate Quantity classes in our schema. We could write an ArchiveSection that normalizes them when convergence is attained.

Regarding @JFRudzinski 's point on the complexity of error bars: likely we need an interface which can host different error definitions (their own type).
That doesn't take away that the simple case of absolute and relative errors is a great starting point.

[ndaelman-hu]

Okay, one big conceptual question that I still have: what are the actual objective of PhysicalProperty? @JFRudzinski @JosePizarro3 care to elaborate?

Pls don't get into errors, binning, time series, etc.: each of these can be tackled more succinctly, e.g. new meta-type, new Quantity.type, ArchiveSection, respectively.
Your design is instead much more open-ended than that: if I can define the name of any quantity, then this becomes de facto the new Quantity and we should deprecate the latter together with keywords.

[JosePizarro3]

@ndaelman-hu we decided to go with ArchiveSection, from the simple fact that we need still functionalities and normalizations for our properties, i.e., PhysicalProperty(ArchiveSection). We were talking that Outputs then will contain a flatlist of properties, i.e.:

class Outputs(ArchiveSection):

    model_system_ref = ...

    electronic_band_gap = SubSection(ElectronicBandGap(PhysicalProperty).m_def)

    electronic_dos = SubSection(ElectronicDOS(PhysicalProperty).m_def)

    specific_heat = SubSection(SpecificHeat(PhysicalProperty).m_def)

    ...

Still, we were talking that we should also use some methods from Quantity, mainly, setting and getting the PhysicalProperty.value similar to Quantity.

We also need more checks regarding value_unit, variables, and shape. More in detail, we were talking about separating the order of the property tensor from the binned variables to which it depends, such that the full_shape is the combination of both things (like I described above). For example, a 3d vector (i.e., shape=[3]) which depends on temperatures and pressure, will end up being of full_shape=[n_temperatures, n_pressure, 3], and this is the actual shape of PhysicalProperty.value.

We also talked about including refs within PhysicalProperty, in order to store value_per_atom, value_per_orbital, value_per_molecule kind of quantities.

---

As you see there is some functionalities need to be implemented, but now we have a more clear design idea:

1. Define PhysicalProperty(ArchiveSection) and add some extra set/get functionalities, and other checks. Focus on simpler things first (errors maybe for later)
2. Define Outputs(ArchiveSection) and all the list of properties as a flat list.
3. Define each property, e.g. ElectronicBandGap(PhysicalProperty), and their functionalities separately (when possible).

This is more or less the idea.

[ndaelman-hu]

@ndaelman-hu we decided to go with ArchiveSection

I asked for the purpose, not the implementation... I got the impression that the implementation is still open-ended: that's why my input was asked. This example appears the closest to a purpose:

For example, a 3d vector (i.e., shape=[3]) which depends on temperatures and pressure, will end up being of full_shape=[n_temperatures, n_pressure, 3], and this is the actual shape of PhysicalProperty.value`

You mean, like a thermodynamic potential (canonical, micro-canonical, macro-canonical)? Tbh, this can be done using the more conventional approaches: just have them 2 quantities in a section... Why do input and output have to be coupled in a single variable?

Still, it would be nice to have a "binned Quantity", yes. We could see to repurpose shape to also accept other quantities when type == 'bin'.

We were talking that Outputs then will contain a flatlist of properties

It's a list of sections with at least 1 deeper layer of subsections, not sure if that qualifies as "flat". Anyhow, in your example, you could also go with a flatter option:

class Outputs(ArchiveSection):

    model_system_ref = ...

    ...

class ElectronicStructure(Outputs):

   band_gap = Quantity(...)

   dos = Quantity(...)

   band_structure = Quantity(...)

   def normalize(self):
       ...

We also need more checks regarding value_unit, variables, and shape.

If those checks are generic, as your example is, Quantity can tackle them. The question is: in how far these have to be customizable?

As you see there is some functionalities need to be implemented

Since I have only a limited view on the purpose, I still can't contribute much here.
Just for clarity, though: I'm not saying that Quantity has to bear all these apparent features. ArchiveSection can help out here as well, to an extent.
I think we can only decide how to separate both, when the full use case is laid out. While abstraction is nice and all, it has to be conceptualized clearly first.

[JFRudzinski]

I appreciate the summary and the questioning...but I suggest that we discuss this more at our meeting on Mon, rather than spending a lot of energy arguing all these points in detail in writing here. Feel free to continue to discuss, but I need to formulate my ideas and let the things discussed yesterday and today sit before I am ready to discuss again