Consider depending on an improved JMSPath library

[springcomp]

Describe the feature

More than a feature request, this issue is my formal way to introduce JMESPath Community.

For many months we at JMESPath Community worked on brainstorming ideas for new features and curated a reasonably scoped set of new features as our first milestone.

The goal is to pick the JMESPath language were it left off and steward improvements to the specification at a quicker pace with a clear governance.

Use Case

Amongst the new features being proposed one can find:

• Arithmetic expressions : allowing simple numeric operations.
• String manipulation functions.
• Ability to refer to arbitrary lexical scopes.
• and more.

Proposed Solution

As we are wrapping up and nearing completion of this first batch of improvements, we are publishing release candidates of the Python implementation of the spec.

If those sound like useful features for AWS CLI, I would like to enquire about the required steps and process to have this library included as a dependency to this repository.

I would be happy to submit a pull request in due course obviously. Please let me know if that is something you would consider.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CLI version used

N/A

Environment details (OS name and version, etc.)

N/A

[tim-finnigan]

Hi @springcomp thanks for the feature request. I think it’s great that you and others are working on extending the functionality of JMESPath. Unfortunately the CLI team can’t consider updating the dependency to an unofficial forked version. I was trying to get more context on the decisions leading to the JMESPath Community and it looks like some of the main discussion on this can be found in jmespath/jmespath.site#94. Please let us know if there's anything you wanted to expand on related to that discussion.

The JMESPath Community is still very new and from that discussion it sounds like some of the proposed improvements could potentially be incorporated into the original JMESPath. If you wanted to share any examples on how the new features might be used in specific AWS CLI commands I think that would be interesting.

[springcomp]

Hi @tim-finnigan thank you for your feedback.

Well, as you've seen, the original feature set of JMESPath has been frozen more than six years ago and the project did not receive any significant updates since then.

I originally considered putting myself forward for taking over stewardship of the specification more than three years ago but eventually could not commit the amount of time needed to give such a project justice.

Then I reevaluated my commitment early this year in the discussion you referred to – three years after my initial proposal – having observed that the original project still had not received any updates in the meantime.

The conditions that lead me to start this effort are the numerous requests from the community made in one of the various JMESPath related repositories. To focus this effort, I think I can convince you that we followed all the right steps. We combed through all the JMESPath repositories and collected the feature requests. We categorized and structured those requests accordingly. From those, we curated a set of consensual improvements to the specification while keeping backwards compatibility to the original specification.

@jamesls and I had a short contact earlier this year and I offered to cancel this effort as James suggested he could start accepting new proposals. I contributed back all my work to the original repository - at that time - mainly typos, updated Python language dependencies, updated embedded JavaScript JMESPath engine and continuous integration GitHub actions.

Unfortunately the original project has not received any new progress since then, nearly 6 months ago.

I think JMESPath deserves more love and that an augmented feature set would benefit users. I actually am an Azure CLI user and hardly ever use AWS CLI. However, the benefits I see that could be had there with an improved JMESPath library would be exactly the same here.

The set of proposals we made stays true to the original specification.

For instance, the JEP-11 Lexical Scoping proposal was actually considered and even accepted by James but was eventually left to linger in pull requests limbo. This features allows a JMESPath expression to refer to any arbitrary lexical scope earlier in the expression evaluation.

Likewise, the JEP-13 Object Manipulation Functions was briefly discussed with James and seemed to have reached consensus before being ignored as well.

Some of our proposals are coming from frequently requested features, like our JEP-14 String Functions.

Some proposals were even alluded to early on in the design of the JMESPath specification, like our JEP-17 Arithmetic Expressions which would allow simple numeric operations like calculating the total cost of cloud resources out of a JSON report of billing usage, for instance.

All those proposals have been implemented and contributed to back in the original Python library. I would love nothing more than JMESPath Community to be made useless in favor of the original project.

I humbly submit that you please consider my request carefully before dismissing it, as I think the JMESPath Community improvements could benefit a wide range of users.

[springcomp]

FYI jmespath-community/1.1.0rc2 has been published.
This marks the final release candidate before publishing the final release version in a few weeks.

[tim-finnigan]

Hi @springcomp thank you for following up with that detailed response. I brought this issue up again for team discussion and the consensus was still that the CLI team is not going to consider changing the JMESpath dependency at this time.

JMESpath has been a CLI dependency for years and is maintained by one of the original CLI team members while JMESPath Community is still very new. As mentioned earlier, it would help if you could provide specific examples of which AWS CLI commands would benefit from the newer features you’re describing. In the future the CLI team may look into this feature request again but for now I’m going to close this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[springcomp]

@tim-finnigan Here is a quick example using aws list-secrets.

Let’s say this returns the following JSON:

{
  "SecretList": [
    {
      "Name": "sandbox_config",
      "Description": "sandbox config file",
      "SecretVersionsToStages": {
        "2895f1a4-bcae-46b2-9a45-4f06f490d8ed": [
          "AWSCURRENT"
        ],
        "336a5030-cd81-4626-a100-1aa68b70b5b1": [
          "AWSPREVIOUS"
        ]
      }
    }
  ]
}

And we wanted to retrieve the current version key.
The following expression illustrates the following improvements:

• let() function that allows to create arbitrary scopes for later use.
• from_items() and zip() functions for manipulating objects out of keys.

SecretList[].{Name:Name, VersionKey:let({k:keys(SecretVersionsToStages), v: values(SecretVersionsToStages)[*][0]}, &from_items(zip(v, k)).AWSCURRENT)}

This would return the following output JSON document:

[
  {
    "Name": "sandbox_config",
    "VersionKey": "2895f1a4-bcae-46b2-9a45-4f06f490d8ed"
  }
]

[springcomp]

@tim-finnigan Here is another example using aws get-user.

Let’s say this returns the following JSON:

{
    "User": {
        "UserId": "a1b2c3d4-5678-90ab-cdef-22222EXAMPLE",
        "AccountId": "a1b2c3d4-5678-90ab-cdef-11111EXAMPLE",
        "PrimaryEmail": "[email protected]",
        "DisplayName": "Martha Rivera",
        "LicenseType": "Pro",
        "UserRegistrationStatus": "Registered",
        "RegisteredOn": "2018-12-20T18:45:25.231Z",
        "InvitedOn": "2018-12-20T18:45:25.231Z",
        "AlexaForBusinessMetadata": {
            "IsAlexaForBusinessEnabled": false,
            "AlexaForBusinessRoomArn": "null"
        },
        "PersonalPIN": "XXXXXXXXXX"
    }
}

And we wanted to return the username part of the PrimaryEmail address.
The following expression illustrates the following improvements:

• split() function to split strings.

User.{UserId: UserId, Username: split(PrimaryEmail, '@')[0]}

This would return the following output JSON document:

{
  "UserId": "a1b2c3d4-5678-90ab-cdef-22222EXAMPLE",
  "Username": "marthar"
}