Refactor and update for ropt 0.11

[verveerpj]

Issue
ropt has been updated to version 0.11, with a few changes that affect Everest:

• ropt does not store names of controls, objectives, constraints and realizations anymore, as it does not need them
• Some functions/classes have been renamed to better reflect their function
• Event handling has been made more generic, changing its interface

These changes are intended to make ropt simpler and give more control to Everest over some aspects (such as control name handling). In addition it aims to make Everest less dependent on ropt internals.

This PR does the following:

• Implement changes needed to use ropt==0.11.

• Remove the output tables generated by ropt, since that should be done by Everest.

• Remove some unused functions in the everest data api

• Remove some data from the everest storage that is not needed

• PR title captures the intent of the changes, and is fitting for release notes.

• Added appropriate release note label

• Commit history is consistent and clean, in line with the contribution guidelines.

• Make sure unit tests pass locally after every commit (git rebase -i main --exec 'just rapid-tests')

When applicable

• When there are user facing changes: Updated documentation
• New behavior or changes to existing untested code: Ensured that unit tests are added (See Ground Rules).
• Large PR: Prepare changes in small commits for more convenient review
• Bug fix: Add regression test for the bug
• Bug fix: Create Backport PR to latest release

[codspeed-hq]

CodSpeed Performance Report

Merging #9923 will not alter performance

_{Comparing refactor-ropt-0.11 (a45fce0) with main (c6128f7)}

Summary

✅ 25 untouched benchmarks

[yngve-sk]

(this comment is also scoped to the same thing in _store_gradient_results) This ensures we select only stuff that is actually requested by Everest? Would ROPT ever generate names that are not in this list? Some kind of comment to clarify why we do this would be helpful, also own commit with descriptive comment.

[verveerpj]

names just collects the names of controls, objectives etc., so that the generated dataframe contains them, instead of an index. It might be possible/better to add the names to the dataframe afterwards, by replacing indices with names. Then we don't have to depend on ropt for that. Do you think that would be doable?

[yngve-sk]

I'm not fully sure I understand this, so having the names in the dataframe instead of an index? So adding them kind of does the same as we now do with the .reset_index()? How would we know where to add which names and so on?

[verveerpj]

I am doing it right now in an extra commit, so if that works I will push it and you will probably understand.

[yngve-sk]

I think some kind of comment somewhere on why we need to do this would be good, it is not apparent that ropt operates only with the indices, and that we have to add the names back

[verveerpj]

Added in the new _ropt_to_df method.

[yngve-sk]

Could we shorten down a bit more to something like:

        @dataclass
        class _RoptToEverestCol:
            ropt_column: str
            everest_column: polars.Series

        ropt_to_everest_cols = {
            "variable": _RoptToEverestCol(
                ropt_column="variable",
                everest_column=self.data.controls["control_name"],
            ),
            "objective": _RoptToEverestCol(
                ropt_column="objective",
                everest_column=self.data.objective_functions["objective_name"],
            ),
            "nonlinear_constraint": _RoptToEverestCol(
                ropt_column="nonlinear_constraint",
                everest_column=self.data.objective_functions["objective_name"],
            ),
            "realization": _RoptToEverestCol(
                ropt_column="realization",
                everest_column=self.data.nonlinear_constraints["constraint_name"],
            ),
        }

        for ropt_col, everest_col in ropt_to_everest_cols.items():
            if ropt_col in select:
                df = df.with_columns(
                    polars.col(ropt_col).replace_strict(dict(enumerate(everest_col)))
                )

[verveerpj]

done.

[yngve-sk]

Nice! LGTM