set_struct does not work in version 2.1.0.dev24

[jbohnslav]

Describe the bug
OmegaConf.set_struct(cfg, False) does not behave as expected.

To Reproduce

import omegaconf
print(omegaconf.__version__) # prints  '2.1.0dev24'
from omegaconf import OmegaConf
cfg = OmegaConf.create({"foo" : 10,"bar": "???"})
print(cfg.baz)

Traceback

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 352, in __getattr__
    key=key, value=None, cause=e, type_override=ConfigAttributeError
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\base.py", line 195, in _format_and_raise
    type_override=type_override,
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 701, in format_and_raise
    _raise(ex, cause)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 599, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 349, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 416, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 448, in _get_node
    raise ConfigKeyError(f"Missing key {key}")
omegaconf.errors.ConfigAttributeError: Missing key baz
    full_key: baz
    object_type=dict

OmegaConf.set_struct(cfg, False)
print(cfg.baz)

Traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 352, in __getattr__
    key=key, value=None, cause=e, type_override=ConfigAttributeError
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\base.py", line 195, in _format_and_raise
    type_override=type_override,
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 701, in format_and_raise
    _raise(ex, cause)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 599, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 349, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 416, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 448, in _get_node
    raise ConfigKeyError(f"Missing key {key}")
omegaconf.errors.ConfigAttributeError: Missing key baz
    full_key: baz
    object_type=dict

Expected behavior
After running pip install --upgrade omegaconf==2, everything works exactly as expected.

from omegaconf import OmegaConf
cfg = OmegaConf.create({"foo" : 10,"bar": "???"})
print(cfg.baz) # None
OmegaConf.set_struct(cfg, True)

result:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 319, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 522, in format_and_raise
    raise_(ex, cause)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 515, in raise_
    raise ex
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 317, in __getattr__
    return self._get_impl(key=key, default_value=DEFAULT_VALUE_MARKER)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 375, in _get_impl
    node = self._get_node(key=key)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 400, in _get_node
    self._validate_get(key)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\dictconfig.py", line 128, in _validate_get
    key=key, value=value, cause=ConfigAttributeError(msg)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 590, in format_and_raise
    raise_(ex, cause)
  File "C:\ProgramData\Anaconda3\envs\deg\lib\site-packages\omegaconf\_utils.py", line 515, in raise_
    raise ex
omegaconf.errors.ConfigAttributeError: Key 'baz' in not in struct
        full_key: baz
        reference_type=Optional[Dict[Any, Any]]
        object_type=dict

OmegaConf.set_struct(cfg, False)
print(cfg.baz) # prints None

Additional context

• OmegaConf version: 2.1.0.dev24
• Python version: 3.7.6
• Operating system : Windows 10
• Please provide a minimal repro

[omry]

This has nothing to do with set_struct and is an intentional change.

OmegaConf 2.1 will align the behavior of OmegaConf DictConfig with that of plain dict much more, enabling better drop-in-replacement support.

One of the breaking changes in OmegaConf 2.1 is that dict __getitem__ access of none-existing keys behaves like a plain dict and raises a KeyError (ConfigKeyError) or an AttributeError (ConfigAttributeError) in such cases, much like plain dicts and regular objects.

You can use one of those alternatives:

# dict style access
c1.get("aa") # None

# object style acecss
getattr(c1, "aa", None) # None

[jbohnslav]

Hmm, it seemed to be related to set_struct because the error messages were identical. I'll have to update to check for keys, rather than assume None as a default.

[omry]

Check #515 for the motivation behind this change and some migration hints.

[jbohnslav]

Is there any way to restore the previous behavior? It is now a pain to access optional, very nested parameters.

Sometimes, I have highly nested configurations, e.g.

model:
  feature_extractor:
    arch: resnet50

The model dict might not be in my configuration.

Previous code:

if cfg.model.feature_extractor.arch is not None:
  do_something(cfg.model.feature_extractor.arch)

Code now?

if 'model' in cfg and `feature_extractor` in cfg.model and `arch` in cfg.model.feature_extractor`: 
  do_something(cfg.model.feature_extractor.arch)

[omry]

There is no way to get the previous behavior with OmegaConf 2.1.

Your example is wrong though, you don't need to check every level, just the last one.

In [1]: from omegaconf import *

In [2]: __version__
Out[2]: '2.0.6'

In [3]: cfg = OmegaConf.create()

In [4]: cfg.foo.bar
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-c2e7eafeb852> in <module>
----> 1 cfg.foo.bar

AttributeError: 'NoneType' object has no attribute 'bar'

The equivalent code is:

if "arch" in cfg.model.feature_extractor: 
  do_something(cfg.model.feature_extractor.arch)

[omry]

You actually get what you thought you were getting using select:

cfg = OmegaConf.create()
print(OmegaConf.select(cfg, "foo.bar")) # None

[omry]

Closing, feel free to follow up if you have more questions.