Support codegen by namespace

semantic-conventions/CHANGELOG.md

@@ -4,6 +4,11 @@ Please update the changelog as part of any significant pull request.
 
 ## Unreleased
 
+- Added code-generation mode that groups attributes by the root namespace and ability to wring each group into individual file.
+  [BREAKING] The `--file-per-group <pattern>` that used to create multiple directories (like `output/<pattern>/file`) now generates
+  multiple files (`output/<pattern>file`) instead.
+  ([#243](https://github.com/open-telemetry/build-tools/pull/243))
+
 ## v0.23.0
 
 - Rephrase and relax sampling-relevant description
@@ -17,7 +22,7 @@ Please update the changelog as part of any significant pull request.
   ([#205](https://github.com/open-telemetry/build-tools/pull/205))
 - Fix referencing template attributes
   ([#206](https://github.com/open-telemetry/build-tools/pull/206))
-  
+
 ## v0.21.0
 
 - Render template-type attributes from yaml files

semantic-conventions/README.md

@@ -85,8 +85,8 @@ convention that have the tag `network`.
 `<!-- semconv http.server(tag=network, full) -->` will print the constraints and attributes of both `http` and `http.server`
 semantic conventions that have the tag `network`.
 
-`<!-- semconv metric.http.server.active_requests(metric_table) -->` will print a table describing a single metric 
-`http.server.active_requests`. 
+`<!-- semconv metric.http.server.active_requests(metric_table) -->` will print a table describing a single metric
+`http.server.active_requests`.
 
 ## Code Generator
 
@@ -125,12 +125,66 @@ This way, multiple files are generated. The value of `pattern` can be one of the
 - `semconv_id`: The id of the semantic convention.
 - `prefix`: The prefix with which all attributes starts with.
 - `extends`: The id of the parent semantic convention.
+- `root_namespace`: The root namespace of attribute to group by.
 
 Finally, additional value can be passed to the template in form of `key=value` pairs separated by
 comma using the `--parameters [{key=value},]+` or `-D` flag.
 
 ### Customizing Jinja's Whitespace Control
 
-The image also supports customising
+The image also supports customizing
 [Whitespace Control in Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/#whitespace-control)
 via the additional flag `--trim-whitespace`. Providing the flag will enable both `lstrip_blocks` and `trim_blocks`.
+
+### Accessing Semantic Conventions in the templated
+
+When template is processed, it has access to a set of variables that depends on the `--file-per-group` value (or lack of it).
+You can access properties of these variables and call Jinja or Python functions defined on them.
+
+#### Single file (no `--file-per-group` pattern is provided)
+
+Processes all parsed semantic conventions
+
+- `semconvs` - the dictionary containing parsed `BaseSemanticConvention` instances with semconv `id` as a key
+- `attributes_and_templates` - the dictionary containing all attributes (including template ones) grouped by their root namespace.
+  Each element in the dictionary is a list of attributes that share the same root namespace. Attributes that don't have a namespace
+  appear under `""` key.
+- `attributes` - the list of all attributes from all parsed semantic conventions. Does not include template attributes.
+- `attribute_templates` - the list of all attribute templates from all parsed semantic conventions.
+
+#### The `root_namespace` pattern
+
+Processes a single namespace and is called for each namespace detected.
+
+- `attributes_and_templates` - the list containing all attributes (including template ones) in the given root namespace.
+- `root_namespace` - the root namespace being processed.
+
+#### Other patterns
+
+Processes a single pattern value and is called for each distinct value.
+
+- `semconv` - the instance of parsed `BaseSemanticConvention` being processed.
+
+### Filtering attributes
+
+Jinja templates has a notion of [filters](https://jinja.palletsprojects.com/en/2.11.x/templates/#list-of-builtin-filters) allowing to transform objects or filter lists.
+
+Semconvgen supports following additional filters to simplify common operations in templates.
+
+#### Attribute filters
+
+1. `is_definition` - Checks if the attribute is the original definition of the attribute and not a reference.
+2. `is_deprecated` - Checks if the attribute is deprecated. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.DEPRECATED"`
+3. `is_experimental` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.EXPERIMENTAL"`
+4. `is_stable` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.STABLE"`
+5. `is_template` - Checks if the attribute is a template attribute. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.STABLE"`
+
+#### String operations
+
+1. `first_up` - Upper-cases the first character in the string. Does not modify anything else
+2. `regex_replace(text, pattern, replace)` - Makes regex-based replace in `text` string using `pattern``
+3. `to_camelcase` - Converts a string to camel case (using `.` and `_` as words delimiter in the original string).
+   The first character of every word is upper-cased, other characters are lower-cased. E.g. `foo.bAR_baz` becomes `fooBarBaz`
+4. `to_const_name` - Converts a string to Python or Java constant name (SNAKE_CASE) replacing `.` or `-` with `_`. E.g.
+   `foo.bAR-baz` becomes `FOO_BAR_BAZ`.
+5. `to_doc_brief` - Trims whitespace and removes dot at the end. E.g. ` Hello world.\t` becomes `Hello world`

semantic-conventions/src/opentelemetry/semconv/main.py

@@ -129,7 +129,8 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--output",
         "-o",
-        help="Specify the output file for the code generation.",
+        help="Specify the output file name for the code generation. "
+        "See also `--file-per-group` on how to generate multiple files.",
         type=str,
         required=True,
     )
@@ -143,8 +144,10 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--file-per-group",
         dest="pattern",
-        help="Each Semantic Convention is processed by the template and store in a different file. PATTERN is expected "
-        "to be the name of a SemanticConvention field and is prepended as a prefix to the output argument",
+        help="Semantic conventions are processed by the template and stored in a different file. "
+        "File names start with a 'pattern' and end with the name specified in the 'output' argument. "
+        "The 'pattern' can either match 'root_namespace' to group attributes by the root namespace or "
+        "match a name of Semantic Convention property which value will be used as a file name prefix.",
         type=str,
     )
     parser.add_argument(

semantic-conventions/src/opentelemetry/semconv/model/constraints.py

@@ -15,11 +15,10 @@
 from dataclasses import dataclass, replace
 from typing import List, Tuple
 
-from ruamel.yaml.comments import CommentedSeq
-
 from opentelemetry.semconv.model.exceptions import ValidationError
 from opentelemetry.semconv.model.semantic_attribute import SemanticAttribute
 from opentelemetry.semconv.model.utils import validate_values
+from ruamel.yaml.comments import CommentedSeq
 
 
 # We cannot frozen due to later evaluation of the attributes

semantic-conventions/src/opentelemetry/semconv/model/semantic_attribute.py

@@ -18,14 +18,13 @@
 from enum import Enum
 from typing import Dict, List, Optional, Union
 
-from ruamel.yaml.comments import CommentedMap, CommentedSeq
-
 from opentelemetry.semconv.model.exceptions import ValidationError
 from opentelemetry.semconv.model.utils import (
     check_no_missing_keys,
     validate_id,
     validate_values,
 )
+from ruamel.yaml.comments import CommentedMap, CommentedSeq
 
 TEMPLATE_PREFIX = "template["
 TEMPLATE_SUFFIX = "]"
@@ -60,6 +59,7 @@ class SemanticAttribute:
     sampling_relevant: bool
     note: str
     position: List[int]
+    root_namespace: str
     inherited: bool = False
     imported: bool = False
 
@@ -203,6 +203,10 @@ def parse(
             fqn = fqn.strip()
             parsed_brief = TextWithLinks(brief.strip() if brief else "")
             parsed_note = TextWithLinks(note.strip())
+
+            namespaces = fqn.split(".")
+            root_namespace = namespaces[0] if len(namespaces) > 1 else ""
+
             attr = SemanticAttribute(
                 fqn=fqn,
                 attr_id=attr_id,
@@ -218,6 +222,7 @@ def parse(
                 sampling_relevant=sampling_relevant,
                 note=parsed_note,
                 position=position,
+                root_namespace=root_namespace,
             )
             if attr.fqn in attributes:
                 position = position_data[list(attribute)[0]]

semantic-conventions/src/opentelemetry/semconv/model/semantic_convention.py

@@ -18,8 +18,6 @@
 from enum import Enum
 from typing import Dict, Optional, Tuple, Union
 
-from ruamel.yaml import YAML
-
 from opentelemetry.semconv.model.constraints import AnyOf, Include, parse_constraints
 from opentelemetry.semconv.model.exceptions import ValidationError
 from opentelemetry.semconv.model.semantic_attribute import (
@@ -28,6 +26,7 @@
 )
 from opentelemetry.semconv.model.unit_member import UnitMember
 from opentelemetry.semconv.model.utils import ValidatableYamlNode, validate_id
+from ruamel.yaml import YAML
 
 
 class SpanKind(Enum):

semantic-conventions/src/opentelemetry/semconv/templating/code.py

@@ -19,9 +19,11 @@
 
 import mistune
 from jinja2 import Environment, FileSystemLoader, select_autoescape
-
 from opentelemetry.semconv.model.semantic_attribute import (
+    AttributeType,
     RequirementLevel,
+    SemanticAttribute,
+    StabilityLevel,
     TextWithLinks,
 )
 from opentelemetry.semconv.model.semantic_convention import SemanticConventionSet
@@ -156,6 +158,30 @@ def to_camelcase(name: str, first_upper=False) -> str:
     return first + "".join(word.capitalize() for word in rest)
 
 
+def first_up(name: str) -> str:
+    return name[0].upper() + name[1:]
+
+
+def is_stable(attribute: SemanticAttribute) -> bool:
+    return attribute.stability == StabilityLevel.STABLE
+
+
+def is_deprecated(attribute: SemanticAttribute) -> bool:
+    return attribute.stability == StabilityLevel.DEPRECATED
+
+
+def is_experimental(attribute: SemanticAttribute) -> bool:
+    return attribute.stability == StabilityLevel.EXPERIMENTAL
+
+
+def is_definition(attribute: SemanticAttribute) -> bool:
+    return attribute.is_local and attribute.ref is None
+
+
+def is_template(attribute: SemanticAttribute) -> bool:
+    return AttributeType.is_template_type(attribute.attr_type)
+
+
 class CodeRenderer:
     pattern = f"{{{ID_RE.pattern}}}"
 
@@ -188,6 +214,7 @@ def get_data_single_file(
             "semconvs": semconvset.models,
             "attributes": semconvset.attributes(),
             "attribute_templates": semconvset.attribute_templates(),
+            "attributes_and_templates": self._grouped_attribute_definitions(semconvset),
         }
         data.update(self.parameters)
         return data
@@ -206,18 +233,28 @@ def setup_environment(env: Environment, trim_whitespace: bool):
         env.filters["to_const_name"] = to_const_name
         env.filters["merge"] = merge
         env.filters["to_camelcase"] = to_camelcase
+        env.filters["first_up"] = first_up
         env.filters["to_html_links"] = to_html_links
         env.filters["regex_replace"] = regex_replace
         env.filters["render_markdown"] = render_markdown
+        env.filters["is_deprecated"] = is_deprecated
+        env.filters["is_definition"] = is_definition
+        env.filters["is_stable"] = is_stable
+        env.filters["is_experimental"] = is_experimental
+        env.filters["is_template"] = is_template
+        env.tests["is_stable"] = is_stable
+        env.tests["is_experimental"] = is_experimental
+        env.tests["is_deprecated"] = is_deprecated
+        env.tests["is_definition"] = is_definition
+        env.tests["is_template"] = is_template
         env.trim_blocks = trim_whitespace
         env.lstrip_blocks = trim_whitespace
 
     @staticmethod
-    def prefix_output_file(file_name, pattern, semconv):
+    def prefix_output_file(file_name, prefix):
         basename = os.path.basename(file_name)
         dirname = os.path.dirname(file_name)
-        value = getattr(semconv, pattern)
-        return os.path.join(dirname, to_camelcase(value, True), basename)
+        return os.path.join(dirname, to_camelcase(prefix, True) + basename)
 
     def render(
         self,
@@ -233,19 +270,78 @@ def render(
             autoescape=select_autoescape([""]),
         )
         self.setup_environment(env, self.trim_whitespace)
-        if pattern:
-            for semconv in semconvset.models.values():
-                output_name = self.prefix_output_file(output_file, pattern, semconv)
-                data = self.get_data_multiple_files(semconv, template_path)
-                template = env.get_template(file_name, globals=data)
-                template.globals["now"] = datetime.datetime.utcnow()
-                template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
-                template.globals["RequirementLevel"] = RequirementLevel
-                template.stream(data).dump(output_name)
+        if pattern == "root_namespace":
+            self._render_group_by_root_namespace(
+                semconvset, template_path, file_name, output_file, env
+            )
+        elif pattern is not None:
+            self._render_by_pattern(
+                semconvset, template_path, file_name, output_file, pattern, env
+            )
         else:
             data = self.get_data_single_file(semconvset, template_path)
             template = env.get_template(file_name, globals=data)
-            template.globals["now"] = datetime.datetime.utcnow()
-            template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
-            template.globals["RequirementLevel"] = RequirementLevel
-            template.stream(data).dump(output_file)
+            self._write_template_to_file(template, data, output_file)
+
+    def _render_by_pattern(
+        self,
+        semconvset: SemanticConventionSet,
+        template_path: str,
+        file_name: str,
+        output_file: str,
+        pattern: str,
+        env: Environment,
+    ):
+        for semconv in semconvset.models.values():
+            prefix = getattr(semconv, pattern)
+            output_name = self.prefix_output_file(output_file, prefix)
+            data = self.get_data_multiple_files(semconv, template_path)
+            template = env.get_template(file_name, globals=data)
+            self._write_template_to_file(template, data, output_name)
+
+    def _render_group_by_root_namespace(
+        self,
+        semconvset: SemanticConventionSet,
+        template_path: str,
+        file_name: str,
+        output_file: str,
+        env: Environment,
+    ):
+        root_namespaces = self._grouped_attribute_definitions(semconvset)
+        for ns in root_namespaces:
+            sanitized_ns = ns if ns != "" else "other"
+            output_name = self.prefix_output_file(output_file, sanitized_ns)
+
+            data = {
+                "template": template_path,
+                "attributes_and_templates": root_namespaces[ns],
+                "root_namespace": sanitized_ns,
+            }
+            data.update(self.parameters)
+
+            template = env.get_template(file_name, globals=data)
+            self._write_template_to_file(template, data, output_name)
+
+    def _grouped_attribute_definitions(self, semconvset):
+        root_namespaces = {}
+        for semconv in semconvset.models.values():
+            for attr in filter(
+                lambda a: is_definition(a), semconv.attributes_and_templates
+            ):
+                if attr.root_namespace not in root_namespaces:
+                    root_namespaces[attr.root_namespace] = []
+                root_namespaces[attr.root_namespace].append(attr)
+
+        for ns in root_namespaces:
+            root_namespaces[ns] = sorted(root_namespaces[ns], key=lambda a: a.fqn)
+        return root_namespaces
+
+    def _write_template_to_file(self, template, data, output_name):
+        template.globals["now"] = datetime.datetime.utcnow()
+        template.globals["version"] = os.environ.get("ARTIFACT_VERSION", "dev")
+        template.globals["RequirementLevel"] = RequirementLevel
+
+        content = template.render(data)
+        if content != "":
+            with open(output_name, "w") as f:
+                f.write(content)