Improve OmegaConfigLoader performance when global/variable interpolations are involved

[ravi-kumar-pilla]

Description

Extending on the investigation in #3893 , OmegaConfigLoader lags in resolving catalog configurations when global/variable interpolations are involved.

Context

Some previous observations:
#3893 (comment)
#3893 (comment)

Steps to Reproduce

Run stress test which creates a single catalog with variable interpolations - https://github.com/kedro-org/kedro/blob/test/ocl-bottleneck/kedro_benchmarks/temp_investigate_ocl/ocl_plot_variables.py

Expected Result

Reduce the time spent on below methods (when interpolations are involved, the bottleneck seems to be OmegaConf.to_container)

• OmegaConf.load
• OmegaConf.to_container
• OmegaConf.merge

Actual Result

#3893 (comment)

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.19.9
• Python version used (python -V): 3.11
• Operating system and version: macOS and Sonoma 14.7.1

[noklam]

During the investigation, I found that there was a slow bit in _set_globals_value. I didn't spent enough time to fix it, but with a quick fix it improves roughly from 1.5s -> 0.9s, but there are probably more.

[noklam]

Particularly, there is an obvious slow path global_oc get created and destroyed for every reference of $globals

kedro/kedro/config/omegaconf_config.py

Lines 433 to 439 in 01c095b

	def _get_globals_value(self, variable: str, default_value: Any = _NO_VALUE) -> Any:
	"""Return the globals values to the resolver"""
	if variable.startswith("_"):
	raise InterpolationResolutionError(
	"Keys starting with '_' are not supported for globals."
	)
	globals_oc = OmegaConf.create(self._globals)