Inconsistent behavior with constructors

[tchx84]

I noticed the following inconsistency:

#!/usr/bin/env python3

import json
import yaml

case1 = """
first: !construct &anchor
  key: original
second: *anchor
"""

case2 = """
first: !construct &anchor  
  key: original
second:
  <<: *anchor
"""

def construct(loader, node):
    return {'key': 'constructed'}
yaml.add_constructor('!construct', construct)

print(json.dumps(yaml.load(case1), indent=2))
print(json.dumps(yaml.load(case2), indent=2))

The output is:

{
  "first": {
    "key": "constructed"
  },
  "second": {
    "key": "constructed"
  }
}
{
  "first": {
    "key": "constructed"
  },
  "second": {
    "key": "original"
  }
}

Shouldn't case1 and case2 produce identical output? Looks like the combination <<: *anchor ends up with the constructor getting ignored.

Environment:

[tchx84@fedora ~]$ rpm -qa | grep yaml
libyaml-devel-0.1.6-8.fc24.x86_64
libyaml-0.1.6-8.fc24.x86_64
[tchx84@fedora ~]$ rpm -qa | grep YAML
perl-CPAN-Meta-YAML-0.018-366.fc25.noarch
python3-PyYAML-3.11-13.fc25.x86_64

[perlpunk]

I'm not sure if it's really defined what should happen in this case. A constructor might return something that is not a mapping (an object, a list, a scalar, ...). So I could imagine that pyyaml simply skips resolving tags for merged aliases.

Do you have a real world use case example for this? (note that I don't know the pyyaml implementation, but I'm writing a YAML processor in perl, and with my current plan the result would be the same. edit: actually I'm not sure about how this would be handled. the question is, if the resolving of the tag has already happened or not)

[tchx84]

I'm not sure if it's really defined what should happen in this case. A constructor might return something that is not a mapping (an object, a list, a scalar, ...). So I could imagine that pyyaml simply skips resolving tags for merged aliases.

I see, hmmm, I didn't think of that.

Do you have a real world use case example for this? (note that I don't know the pyyaml implementation, but I'm writing a YAML processor in perl, and with my current plan the result would be the same. edit: actually I'm not sure about how this would be handled. the question is, if the resolving of the tag has already happened or not)

Yes, here we are using YAML files to provide templates from our lib that are "imported" by our apps. What we try to do here is to use !tags inside our templates to allow app developers to override certain values inside the templates. The use of these !tags and constructors (as per the example) would allow us to "inject" the overridden values into the templates.

[tchx84]

After looking at constructor code, this is what happens:

Example (case2):

first: &anchor !construct  
  key: original
second:
  <<: *anchor

Right after it's loaded, looks like this:

first: &anchor !construct  
  key: original
second:
  <<: !construct
      key: original

After the root map node is contructed, looks like this:

first:
  key: constructed
second:
  <<: !construct
      key: original

NOTE: the !construct tags are processed while traversing the document and constructing the parent nodes (parent of the tagged nodes), so at this point the second !construct tag has not been processed yet.

Right after that, it reaches the merge node and the key-merge is flatten, looking like this:

first:
  key: constructed
second:
  key: original

NOTE: the flatten step occurs right before the contruct step and completely ignores the tags.

So, there seems to be 2 issues, 1) flatten step ignores the tags and/or, 2) (even if the tags were not lost during flatten) the tags are processed only when constructing the parents node, but since the parent node of the merge node has already been constructed, the resulting node of the merge is never "constructed", thus, the its !constructed would never be processed anyways.

To me, it sounds like these tags should be processed before the merge happens, but that is not how pyyaml traverses the document. I wonder if @xitology has other ideas for what to do with this case.