JEP-11 Lexical Scoping

[innovate-invent]

Lexical Scoping

JEP	11
Author	James Sayerwinnie
Created	24-Feb-2015
SemVer	MINOR
[Discussion #24]	#24

Abstract

This JEP proposes a new function let() (originally proposed by Michael
Dowling) that allows for evaluating an expression with an explicitly defined
lexical scope. This will require some changes to the lookup semantics in
JMESPath to introduce scoping, but provides useful functionality such as being
able to refer to elements defined outside of the current scope used to evaluate
an expression.

Motivation

As a JMESPath expression is being evaluated, the current element, which can be
explicitly referred to via the @ token, changes as expressions are
evaluated. Given a simple sub expression such as foo.bar, first the
foo expression is evaluted with the starting input JSON document, and the
result of that expression is then used as the current element when the bar
element is evaluted. Conceptually we’re taking some object, and narrowing down
its current element as the expression is evaluted.

Once we’ve drilled down to a specific current element, there is no way, in the
context of the currently evaluated expression, to refer to any elements outside
of that element. One scenario where this is problematic is being able to refer
to a parent element.

For example, suppose we had this data:

{"first_choice": "WA",
 "states": [
   {"name": "WA", "cities": ["Seattle", "Bellevue", "Olympia"]},
   {"name": "CA", "cities": ["Los Angeles", "San Francisco"]},
   {"name": "NY", "cities": ["New York City", "Albany"]},
 ]
}

Let’s say we wanted to get the list of cities of the state corresponding to our
first_choice key. We’ll make the assumption that the state names are
unique in the states list. This is currently not possible with JMESPath.
In this example we can hard code the state WA:

states[?name==`WA`].cities[]

but it is not possible to base this on a value of first_choice, which
comes from the parent element. This JEP proposes a solution that makes
this possible in JMESPath.

Specification

There are two components to this JEP, a new function, let(), and a change
to the way that identifiers are resolved.

The let() Function

The let() function is heavily inspired from the let function commonly
seen in the Lisp family of languages:

• https://clojuredocs.org/clojure.core/let

• http://docs.racket-lang.org/guide/let.html

The let function is defined as follows:

any let(object scope, expression->any expr)

let is a function that takes two arguments. The first argument is a JSON
object. This hash defines the names and their corresponding values that will
be accessible to the expression specified in the second argument. The second
argument is an expression reference that will be evaluated.

Resolving Identifiers

Prior to this JEP, identifiers are resolved by consulting the current context
in which the expression is evaluted. For example, using the same
search function as defined in the JMESPath specification, the
evaluation of:

search(foo, {"foo": "a", "bar": "b"}) -> "a"

will result in the foo identifier being resolved in the context of
the input object {"foo": "a", "bar": "b"}. The context object defines
foo as a, which results in the identifier foo being resolved as
a.

In the case of a sub expression, where the current evaluation context
changes once the left hand side of the sub expression is evaluted:

search(a.b, {"a": {"b": "y"}) -> "y"

The identifier b is resolved with a current context of
{"b": "y"}, which results in a value of y.

This JEP adds an additional step to resolving identifiers. In addition
to the implicit evaluation context that changes based on the result
of continually evaluating expressions, the let() command allows
for additional contexts to be specified, which we refer to by the common
name scope. The steps for resolving an identifier are:

• Attempt to lookup the identifier in the current evaluation context.

• If this identifier is not resolved, look up the value in the current
  scope provided by the user.

• If the idenfitier is not resolved and there is a parent scope, attempt
  to resolve the identifier in the parent scope. Continue doing this until
  there is no parent scope, in which case, if the identifier has not been
  resolved, the identifier is resolved as null.

Parent scopes are created by nested let() calls.

Below are a few examples to make this more clear. First, let’s
examine the case where the identifier can be resolved from the
current evaluation context:

search(let({a: `x`}, &b), {"b": "y"}) -> "y"

In this scenario, we are evaluating the expression b, with the
context object of {"b": "y"}. Here b has a value of y,
so the result of this function is y.

Now let’s look at an example where an identifier is resolved from
a scope object provided via let():

search(let({a: `x`}, &a), {"b": "y"}) -> "x"

Here, we’re trying to resolve the a identifier. The current
evaluation context, {"b": "y"}, does not define a. Normally,
this would result in the identifier being resolved as null:

search(a, {"b": "y"}) -> null

However, we now fall back to looking in the provided scope object {"a": "x"}, which was provided as the first argument to let. Note here that
the value of a has a value of "x", so the identifier is resolved as
"x", and the return value of the let() function is "x".

Finally, let’s look at an example of parent scopes. Consider the
following expression:

search(let({a: `x`}, &let({b: `y`}, &{a: a, b: b, c: c})),
       {"c": "z"}) -> {"a": "x", "b": "y", "c": "z"}

Here we have nested let calls, and the expression we are trying to
evaluate is the multiselect hash {a: a, b: b, c: c}. The
c identifier comes from the evaluation context {"c": "z"}.
The b identifier comes from the scope object in the second let
call: {b: \y`}. And finally, here’s the lookup process for the a` identifier:

• Is a defined in the current evaluation context? No.

• Is a defined in the scope provided by the user? No.

• Is there a parent scope? Yes

• Does the parent scope, {a: \x`}, define a? Yes, ahas the value of"x", so ais resolved as the string"x"`.

Current Node Evaluation

While the JMESPath specification defines how the current node is determined,
it is worth explicitly calling out how this works with the let() function
and expression references. Consider the following expression:

a.let({x: `x`}, &b.let({y: `y`}, &c))

Given the input data:

{"a": {"b": {"c": "foo"}}}

When the expression c is evaluated, the current evaluation context is
{"c": "foo"}. This is because this expression isn’t evaluated until
the second let() call evaluates the expression, which does not
occur until the first let() function evaluates the expression.

Motivating Example

With these changes defined, the expression in the “Motivation” section can be
be written as:

let({first_choice: first_choice}, &states[?name==first_choice].cities[])

Which evalutes to ["Seattle", "Bellevue", "Olympia"].

Rationale

If we just consider the feature of being able to refer to a parent element,
this approach is not the only way to accomplish this. We could also allow
for explicit references using a specific token, say $.
The original example in the “Motivation” section would be:

states[?name==$.first_choice].cities[]

While this could work, this has a number of downsides, the biggest one being
that you’ll need to always keep track of the parent element. You don’t know
ahead of time if you’re going to need the parent element, so you’ll always need
to track this value. It also doesn’t handle nested lexical scopes. What if
you wanted to access a value in the grand parent element? Requiring an
explicit binding approach via let() handles both these cases, and doesn’t
require having to track parent elements. You only need to track additional
scope when let() is used.

Implementation Survey

C#

JMESPath.NET implements this proposal.

To this end, the project authors had to introduce a new abstraction to the AST object that implements function calls.

/// <summary>
/// The <see cref="IScopeParticipant" /> interface lets
/// implementations participate in a stack of contexts
/// to assist evaluating expressions.
///
/// This supports the <see cref="LetFunction" />.
/// </summary>
public interface IScopeParticipant
{
    void PushScope(JToken scope);
    void PopScope();
}

This abstraction is actually only used for the implementation of the let() function itself.

public class LetFunction : JmesPathFunction
{
    public LetFunction(IScopeParticipant scopes)
        : base("let", 2, scopes)
    {
    }

    public override void Validate(params JmesPathFunctionArgument[] args)
    {
        System.Diagnostics.Debug.Assert(base.Scopes != null);

        EnsureObject(args[0]);
        EnsureExpressionType(args[1]);
    }

    public override JToken Execute(params JmesPathFunctionArgument[] args)
    {
        scopes_?.PushScope(args[0].Token);

        try
        {
            var expression = args[1].Expression;
            var result = expression.Transform(Context);

            return result.AsJToken();
        }
        finally
        {
            scopes_?.PopScope();
        }
    }
}

This allows to register the function evaluation context to the corresponding lexical scope.

Additionally, the following abstraction was introduced:

```c#
public interface IContextEvaluator
{
    JToken Evaluate(string identifier);
}

The IContextEvaluator abstraction encapsulates context evaluation logic required to extract the proper value
from the stack of lexical scopes. The implementation follows the specification requirements:

public sealed class LexicalScopes : IScopeParticipant, IContextEvaluator
{
    private readonly Stack<JToken> scopes_
        = new Stack<JToken>()
        ;
    public JToken Evaluate(string identifier)
    {
        if (scopes_.Count == 0)
            return JTokens.Null;

        foreach (var scope in scopes_)
        {
            if (scope[identifier] != null)
                return scope[identifier];
        }

        return JTokens.Null;
    }

    public void PushScope(JToken token)
    {
        scopes_.Push(token);
    }
    public void PopScope()
    {
        scopes_.Pop();
    }
}

The lexical scope stack contains a series of JSON objects referred to by the JToken type in C#.
When evaluating a JMESPath expression, identifier expressions are evaluated. That’s where scope
evaluation must take place.

public class JmesPathIdentifier : JmesPathExpression
{
    private readonly string name_;
    internal IContextEvaluator evaluator_;

    public JmesPathIdentifier(string name)
    {
        name_ = name;
    }

    public string Name => name_;

    protected override JmesPathArgument Transform(JToken json)
    {
        var jsonObject = json as JObject;
        return jsonObject?[name_] ?? Evaluate(name_);
    }

    public override string ToString()
    {
        return $"JmesPathIdentifier: {name_}";
    }

    public JToken Evaluate(string identifier)
        => evaluator_?.Evaluate(identifier) ?? JTokens.Null;
}

When evaluating an identifier against the current JSON object, the implementation first uses the current context
which is specified as an argument of the corresponding expression. If the identifier does not refer to an existing
value, the identifier switches to using the IContextEvaluator abstraction referred to above to find the required
value out of the stack of lexical scopes.

No dependency were required to implement this JEP.

Other languages

Given that most object-oriented languages support the concept of abstractions via interfaces (or prototypes) and that
an expected implementation would map grammar constructs to some form of AST, it seems reasonable to believe that a
similar implementation as the one shown here for C# could be achieved with the following languages:

• C++
• Java
• JavaScript
• TypeScript
• Go
• Rust

Although I have no experience on other languages, there is no reason to believe it would be any different or even harder
than the simple implementation shown here.

[innovate-invent]

I have an implementation in python:
https://github.com/brinkmanlab/BioPython-Convert/blob/73473ea6a7cd2ac5006d9fbfb131b5d937bb400c/biopython_convert/JMESPathGen.py#L65-L73
https://github.com/brinkmanlab/BioPython-Convert/blob/73473ea6a7cd2ac5006d9fbfb131b5d937bb400c/biopython_convert/JMESPathGen.py#L133-L135

Also, I think I still have that script to convert test json files to yaml. Do you care to write up the rest of the yaml?

[springcomp]

Also, I think I still have that script to convert test json files to yaml. Do you care to write up the rest of the yaml?

Sure. functions/functions_let.json contains compliance tests.
But I will change that into the actual yaml specification for the function.

And I will add a reference to the pull request in the test repository.

[innovate-invent]

I converted the json to yaml, stubbing the let.yml file.
Let me know if you have any concerns with these changes.

I have also rebased the branch off of #69

[springcomp]

I converted the json to yaml, stubbing the let.yml file. Let me know if you have any concerns with these changes.
I have also rebased the branch off of #69

Thanks you for the legwork.
I have fixed some typos and rendering issues.

I did not find how to have at least the first example displayed, before the show all.. expansion.

I noticed the examples for function let are quite verbose and difficult to read.
Maybe not all of them need to be displayed on the site.

I also have pushed a pull request to use monospace font instead in an attempt at making it easier to read.

[innovate-invent]

I did not find how to have at least the first example displayed, before the show all.. expansion.

The examples will display before "show all" if the input data is shorter than 60 characters. We might need to add some trivial examples just for that purpose.