Support separator option in Variables section.

This option specifies the separator to use if a scalar variable gets
more than one value. The old way to accomplish that is using a
`SEPARATOR` marker as the first value. The configuration option is
already recognized by the parser and also consistent with the
`separator` option the new `VAR` syntax (robotframework#3761) will get.

Fixes robotframework#4896.

atest/robot/variables/variable_section.robot

@@ -55,24 +55,30 @@ Invalid variable name
     Parsing Variable Should Have Failed    3    19    \${not}[[]ok]
     Parsing Variable Should Have Failed    4    20    \${not \${ok}}
 
-Scalar catenated from multile values
+Scalar catenated from multiple values
+    Check Test Case     ${TEST NAME}
+
+Scalar catenated from multiple values with 'SEPARATOR' marker
+    Check Test Case     ${TEST NAME}
+
+Scalar catenated from multiple values with 'separator' option
     Check Test Case     ${TEST NAME}
 
 Creating variable using non-existing variable fails
     Check Test Case    ${TEST NAME}
-    Creating Variable Should Have Failed    8     \${NONEX 1}     33
+    Creating Variable Should Have Failed    8     \${NONEX 1}     35
     ...    Variable '\${NON EXISTING}' not found.
-    Creating Variable Should Have Failed    9     \${NONEX 2A}    34
+    Creating Variable Should Have Failed    9     \${NONEX 2A}    36
     ...    Variable '\${NON EX}' not found.*
-    Creating Variable Should Have Failed    10    \${NONEX 2B}    35
+    Creating Variable Should Have Failed    10    \${NONEX 2B}    37
     ...    Variable '\${NONEX 2A}' not found.*
 
 Using variable created from non-existing variable in imports fails
-    Creating Variable Should Have Failed    5    \${NONEX 3}     36
+    Creating Variable Should Have Failed    5    \${NONEX 3}      38
     ...    Variable '\${NON EXISTING VARIABLE}' not found.
-    Import Should Have Failed               6    Resource        39
+    Import Should Have Failed               6    Resource         41
     ...    Variable '\${NONEX 3}' not found.*
-    Import Should Have Failed               7    Library         40
+    Import Should Have Failed               7    Library          42
     ...    Variable '\${NONEX 3}' not found.*
 
 *** Keywords ***

atest/testdata/variables/variable_section.robot

@@ -28,8 +28,10 @@ ${ASSING MARK} =  This syntax works starting from 1.8
 @{ASSIGN MARK LIST}=   This syntax works    starting    from    ${1.8}
 ${THREE DOTS}     ...
 @{3DOTS LIST}     ...   ...
-${CATENATED}      I    am    a    scalar     catenated     from    many     items
-${CATENATED W/ SEP}    SEPARATOR=-    I    can    haz    custom    separator
+${CATENATED}            By    default    values    are    joined    with    a    space
+${SEPARATOR VALUE}      SEPARATOR=-    Special    SEPARATOR    marker    as    ${1}    st    value
+${SEPARATOR OPTION}     Explicit    separator    option    works    since    RF    ${7.0}    separator=-
+${BOTH SEPARATORS}      SEPARATOR=marker    has    lower    precedence    than    option    separator=:
 ${NONEX 1}        Creating variable based on ${NON EXISTING} variable fails.
 ${NONEX 2A}       This ${NON EX} is used for creating another variable.
 ${NONEX 2B}       ${NONEX 2A}
@@ -125,9 +127,15 @@ Three dots on the same line should be interpreted as string
     ${sos} =    Catenate    SEPARATOR=---    @{3DOTS LIST}
     Should Be Equal    ${sos}    ...---...
 
-Scalar catenated from multile values
-    Should Be Equal    ${CATENATED}      I am a scalar catenated from many items
-    Should Be Equal    ${CATENATED W/ SEP}    I-can-haz-custom-separator
+Scalar catenated from multiple values
+    Should Be Equal    ${CATENATED}           By default values are joined with a space
+
+Scalar catenated from multiple values with 'SEPARATOR' marker
+    Should Be Equal    ${SEPARATOR VALUE}     Special-SEPARATOR-marker-as-1-st-value
+
+Scalar catenated from multiple values with 'separator' option
+    Should Be Equal    ${SEPARATOR OPTION}    Explicit-separator-option-works-since-RF-7.0
+    Should Be Equal    ${BOTH SEPARATORS}     SEPARATOR=marker:has:lower:precedence:than:option
 
 Creating variable using non-existing variable fails
     Variable Should Not Exist    ${NONEX 1}

doc/userguide/src/CreatingTestData/Variables.rst

@@ -495,18 +495,35 @@ variables slightly more explicit.
 
 If a scalar variable has a long value, it can be `split into multiple rows`__
 by using the `...` syntax. By default rows are concatenated together using
-a space, but this can be changed by having `SEPARATOR=<sep>` as the first item.
+a space, but this can be changed by using a having `separator` configuration
+option after the last value:
 
 .. sourcecode:: robotframework
 
    *** Variables ***
    ${EXAMPLE}      This value is joined
    ...             together with a space.
+   ${MULTILINE}    First line.
+   ...             Second line.
+   ...             Third line.
+   ...             separator=\n
+
+The `separator` option is new in Robot Framework 7.0, but also older versions
+support configuring the separator. With them the first value can contain a
+special `SEPARATOR` marker:
+
+.. sourcecode:: robotframework
+
+   *** Variables ***
    ${MULTILINE}    SEPARATOR=\n
    ...             First line.
    ...             Second line.
    ...             Third line.
 
+Both the `separator` option and the `SEPARATOR` marker are case-sensitive.
+Using the `separator` option is recommended, unless there is a need to
+support also older versions.
+
 __ `Dividing data to several rows`_
 
 Creating list variables

src/robot/parsing/lexer/statementlexers.py

@@ -182,6 +182,11 @@ class VariableLexer(TypeAndArguments):
     ctx: FileContext
     token_type = Token.VARIABLE
 
+    def lex(self):
+        super().lex()
+        if self.statement[0].value[:1] == '$':
+            self._lex_options('separator')
+
 
 class KeywordCallLexer(StatementLexer):
     ctx: 'TestCaseContext|KeywordContext'

src/robot/parsing/model/statements.py

@@ -639,15 +639,24 @@ def from_params(cls, value: str, separator: str = FOUR_SPACES,
 @Statement.register
 class Variable(Statement):
     type = Token.VARIABLE
+    options = {
+        'separator': None
+    }
 
     @classmethod
-    def from_params(cls, name: str, value: 'str|Sequence[str]',
-                    separator: str = FOUR_SPACES, eol: str = EOL) -> 'Variable':
+    def from_params(cls, name: str,
+                    value: 'str|Sequence[str]',
+                    value_separator: 'str|None' = None,
+                    separator: str = FOUR_SPACES,
+                    eol: str = EOL) -> 'Variable':
         values = [value] if isinstance(value, str) else value
         tokens = [Token(Token.VARIABLE, name)]
         for value in values:
             tokens.extend([Token(Token.SEPARATOR, separator),
                            Token(Token.ARGUMENT, value)])
+        if value_separator is not None:
+            tokens.extend([Token(Token.SEPARATOR, separator),
+                           Token(Token.OPTION, f'separator={value_separator}')])
         tokens.append(Token(Token.EOL, eol))
         return cls(tokens)
 
@@ -662,13 +671,18 @@ def name(self) -> str:
     def value(self) -> 'tuple[str, ...]':
         return self.get_values(Token.ARGUMENT)
 
+    @property
+    def separator(self) -> 'str|None':
+        return self.get_option('separator')
+
     def validate(self, ctx: 'ValidationContext'):
         name = self.get_value(Token.VARIABLE)
         match = search_variable(name, ignore_errors=True)
         if not match.is_assign(allow_assign_mark=True, allow_nested=True):
             self.errors += (f"Invalid variable name '{name}'.",)
         if match.is_dict_assign(allow_assign_mark=True):
             self._validate_dict_items()
+        self._validate_options()
 
     def _validate_dict_items(self):
         for item in self.get_values(Token.ARGUMENT):

src/robot/running/builder/transformers.py

@@ -109,6 +109,7 @@ def visit_SettingSection(self, node):
     def visit_Variable(self, node):
         self.suite.resource.variables.create(name=node.name,
                                              value=node.value,
+                                             separator=node.separator,
                                              lineno=node.lineno,
                                              error=format_error(node.errors))
 
@@ -154,6 +155,7 @@ def visit_VariablesImport(self, node):
     def visit_Variable(self, node):
         self.resource.variables.create(name=node.name,
                                        value=node.value,
+                                       separator=node.separator,
                                        lineno=node.lineno,
                                        error=format_error(node.errors))
 

src/robot/running/model.py

@@ -624,15 +624,17 @@ def to_dict(self) -> DataDict:
 
 
 class Variable(ModelObject):
-    repr_args = ('name', 'value')
+    repr_args = ('name', 'value', 'separator')
 
     def __init__(self, name: str = '',
                  value: Sequence[str] = (),
+                 separator: 'str|None' = None,
                  parent: 'ResourceFile|None' = None,
                  lineno: 'int|None' = None,
                  error: 'str|None' = None):
         self.name = name
         self.value = tuple(value)
+        self.separator = separator
         self.parent = parent
         self.lineno = lineno
         self.error = error

src/robot/variables/tablesetter.py

@@ -17,7 +17,7 @@
 from typing import Sequence, TYPE_CHECKING
 
 from robot.errors import DataError
-from robot.utils import DotDict, is_string, split_from_equals
+from robot.utils import DotDict, split_from_equals
 
 from .resolvable import Resolvable
 from .search import is_assign, is_list_variable, is_dict_variable
@@ -47,33 +47,35 @@ def _get_items(self, variables: 'Sequence[Variable]'):
 
 class VariableResolver(Resolvable):
 
-    def __init__(self, value: 'str|Sequence[str]', error_reporter=None):
-        self.value = self._format_value(value)
+    def __init__(self, value: Sequence[str], error_reporter=None):
+        self.value = value
         self.error_reporter = error_reporter
         self._resolving = False
 
-    def _format_value(self, value):
-        return value
-
     @classmethod
     def from_name_and_value(cls, name: str, value: 'str|Sequence[str]',
+                            separator: 'str|None' = None,
                             error_reporter=None) -> 'VariableResolver':
         if not is_assign(name):
             raise DataError(f"Invalid variable name '{name}'.")
-        klass = {'$': ScalarVariableResolver,
-                 '@': ListVariableResolver,
+        if name[0] == '$':
+            return ScalarVariableResolver(value, separator, error_reporter)
+        if separator is not None:
+            raise DataError('Only scalar variables support separators.')
+        klass = {'@': ListVariableResolver,
                  '&': DictVariableResolver}[name[0]]
         return klass(value, error_reporter)
 
     @classmethod
     def from_variable(cls, var: 'Variable') -> 'VariableResolver':
         if var.error:
             raise DataError(var.error)
-        return cls.from_name_and_value(var.name, var.value, var.report_error)
+        return cls.from_name_and_value(var.name, var.value, var.separator,
+                                       var.report_error)
 
     def resolve(self, variables):
         with self._avoid_recursion:
-            return self._replace_variables(self.value, variables)
+            return self._replace_variables(variables)
 
     @property
     @contextmanager
@@ -86,7 +88,7 @@ def _avoid_recursion(self):
         finally:
             self._resolving = False
 
-    def _replace_variables(self, value, variables):
+    def _replace_variables(self, variables):
         raise NotImplementedError
 
     def report_error(self, error):
@@ -98,41 +100,45 @@ def report_error(self, error):
 
 class ScalarVariableResolver(VariableResolver):
 
-    def _format_value(self, values):
-        separator = None
-        if is_string(values):
-            values = [values]
-        elif values and values[0].startswith('SEPARATOR='):
-            separator = values[0][10:]
-            values = values[1:]
-        return separator, values
-
-    def _replace_variables(self, values, variables):
-        separator, values = values
-        # Avoid converting single value to string.
-        if self._is_single_value(separator, values):
-            return variables.replace_scalar(values[0])
+    def __init__(self, value: 'str|Sequence[str]', separator: 'str|None' = None,
+                 error_reporter=None):
+        value, separator = self._get_value_and_separator(value, separator)
+        super().__init__(value, error_reporter)
+        self.separator = separator
+
+    def _get_value_and_separator(self, value, separator):
+        if isinstance(value, str):
+            value = [value]
+        elif separator is None and value and value[0].startswith('SEPARATOR='):
+            separator = value[0][10:]
+            value = value[1:]
+        return value, separator
+
+    def _replace_variables(self, variables):
+        value, separator = self.value, self.separator
+        if self._is_single_value(value, separator):
+            return variables.replace_scalar(value[0])
         if separator is None:
             separator = ' '
-        separator = variables.replace_string(separator)
-        values = variables.replace_list(values)
-        return separator.join(str(item) for item in values)
+        else:
+            separator = variables.replace_string(separator)
+        value = variables.replace_list(value)
+        return separator.join(str(item) for item in value)
 
-    def _is_single_value(self, separator, values):
-        return (separator is None and len(values) == 1 and
-                not is_list_variable(values[0]))
+    def _is_single_value(self, value, separator):
+        return separator is None and len(value) == 1 and not is_list_variable(value[0])
 
 
 class ListVariableResolver(VariableResolver):
 
-    def _replace_variables(self, values, variables):
-        return variables.replace_list(values)
+    def _replace_variables(self, variables):
+        return variables.replace_list(self.value)
 
 
 class DictVariableResolver(VariableResolver):
 
-    def _format_value(self, values):
-        return list(self._yield_formatted(values))
+    def __init__(self, value: Sequence[str], error_reporter=None):
+        super().__init__(tuple(self._yield_formatted(value)), error_reporter)
 
     def _yield_formatted(self, values):
         for item in values:
@@ -147,9 +153,9 @@ def _yield_formatted(self, values):
                     )
                 yield name, value
 
-    def _replace_variables(self, values, variables):
+    def _replace_variables(self, variables):
         try:
-            return DotDict(self._yield_replaced(values, variables.replace_scalar))
+            return DotDict(self._yield_replaced(self.value, variables.replace_scalar))
         except TypeError as err:
             raise DataError(f'Creating dictionary failed: {err}')
 

utest/parsing/test_model.py

@@ -908,6 +908,28 @@ def test_valid(self):
         )
         get_and_assert_model(data, expected, depth=0)
 
+    def test_separator(self):
+        data = '''
+*** Variables ***
+${x}      a    b    c    separator=-
+${y}      separator=
+'''
+        expected = VariableSection(
+            header=SectionHeader(
+                tokens=[Token(Token.VARIABLE_HEADER, '*** Variables ***', 1, 0)]
+            ),
+            body=[
+                Variable([Token(Token.VARIABLE, '${x}', 2, 0),
+                          Token(Token.ARGUMENT, 'a', 2, 10),
+                          Token(Token.ARGUMENT, 'b', 2, 15),
+                          Token(Token.ARGUMENT, 'c', 2, 20),
+                          Token(Token.OPTION, 'separator=-', 2, 25)]),
+                Variable([Token(Token.VARIABLE, '${y}', 3, 0),
+                          Token(Token.OPTION, 'separator=', 3, 10)]),
+            ]
+        )
+        get_and_assert_model(data, expected, depth=0)
+
     def test_invalid(self):
         data = '''
 *** Variables ***