The future of JMESPath

[springcomp]

Dear all,

After many careful thoughts, I would like to re-open the discussion we had with regards to the status of this project.

I would like to present an unofficial fork of this project that I – together with my co-author @jdevillard – can commit to maintain for the foreseeable future provided this is received well amongst the community. I would love nothing more than close this fork and rely on this original project. However, the slow progress of this project makes it impossible to invest in JmesPath enhancements if there is no guarantee that this makes it to the various language libraries.

This is a preliminary clone of the original page that I will update depending upon community requirements:

• Align the existing JEP proposals and have the community vote on their status. Some could be including in the still unofficial but current specification. Some could be marked specifically as "extensions" and provisions could be made in the various libraries to clearly delineate standard vs extensions.
• Update links to the various official libraries with added references to the more up-to-date but still unofficial ports.

I would like to welcome members to the corresponding GitHub organization so that we could make sure to have a strong community effort.

To be honest, I have never maintained a community open-source project and have frankly no idea to do this. However, I feel there is a need and I would welcome any help in this regards.

Please, do not get me wrong - I would like this project to serve as an umbrella for up-to-date libraries, specifications and a committing to study the status of various pending proposals. My goal is not to shortcircuit the original maintainer. But I feel it is long overdue that some updates happened to the status of this projet.

Please, dear community, if this offends anyone, please feel free to let me know.

[springcomp]

@chris-armstrong I just realize that you started the same effort.
Although I have started to cherry-pick various fixes and enhancements to my fork,
should we use your fork instead ?

I'm mainly interested in the spec and proposals.
I see that you have proposals as well.

I was planning to split the spec from the website itself.
Once that's done, I can certainly close my fork.

I'm interested to hear what everyone thinks.

[innovate-invent]

@jamesls I think it is time for you to delegate your role. @springcomp and @jdevillard seem to be willing to commit resources.

@springcomp can you describe the organisation that is backing you? I believe you are working on this on behalf of an employer.
If you are committed to this project then it might be a good start to pull out the current jeps and proposals into a jmespath.jep repository that @jamesls described in #65

I believe the goal was to separate the website code base and issues from the core specification. This new repository would need to include all the necessary documentation for contributing to the JMESPath specification, including some guiding principles.

I am always free to offer input but I don't have the resources to regularly contribute. Thank you for your interest in this project.

[springcomp]

@springcomp can you describe the organisation that is backing you? I believe you are working on this on behalf of an employer

@innovate-invent I will be involved from a personal standpoint and not with any corporate backing. Although this would be on a best effort basis, and cannot make any guarantees, I believe I can certainly dedicate a small amount of time on a continuous basis. Like a few hours every month.

My goal is to foster a community effort and I'm more than willing to share the work. That's why I created an organization instead of hosting this on my private GitHub account.

[springcomp]

I believe the goal was to separate the website code base and issues from the core specification. This new repository would need to include all the necessary documentation for contributing to the JMESPath specification, including some guiding principles.

Of course that is the goal. Just wanted to bootstrap the effort with more concrete steps than previously when I participated in the discussion.

[innovate-invent]

Right, are you planning on forking https://github.com/jmespath/jmespath.jep into your organisation? Or did you have a different direction planned?

[springcomp]

Right, are you planning on forking https://github.com/jmespath/jmespath.jep into your organisation? Or did you have a different direction planned?

I think one repository for both the spec and its proposals would make sense. This would be separate from the website. That's the direction I'd like to take.

But I cannot alone decide if I like a particular implementation. So there would be at least a community consensus for each proposal. I'm currently prototyping the JEP-11 Lexical Scoping proposal, and that's where you find out about trade-offs.

[innovate-invent]

Would you want to keep the repo as a fork so that the issues remain with the original? I think we would have to break the fork relationship to enable issue moderation on the fork.

JEP-11 is probably the best first item to be integrated.

[springcomp]

Would you want to keep the repo as a fork so that the issues remain with the original? I think we would have to break the fork relationship to enable issue moderation on the fork.

@innovate-invent You are right. In the long run, either I will close the site or break the fork relationship.

JEP-11 is probably the best first item to be integrated.

I have instantiated the JEP repository with existing proposals.

[springcomp]

Dear community,

@jdevillard and myself have now setup the various forks required for ensuring the future of JMESPath.

A JMESPath Site fork has been setup.

It is clearly marked as being unofficial.
I have merged most of the pending pull requests that made sense to fix various typos.
If you feel you have submitted a pull request that has been left out, please contact us.

As the goal to make sure the JMESPath specification receives ongoing improvements
we decided to fix the initial version of the spec as of 2016-09-05.
More specifically, the version is  2016-09-05-9e8d0e3 in reference to the commit that last updated the spec.

The ABNF specification grammar has been split to make it easy to refer to directly.

The libaries page has also been updated to retrieve the compliance level
of the latest version of the spec. I expect libary implementors will want to
opt in to this declaration. The compliance level is retrieved at publication time
from the original GitHub repositories of each library implementation.

We expect to add various regression tests to the JMESPath compliance test suite.
The test suite has thus been unofficially forked for this purpose as well.

Early days

In the coming weeks, we will probably start adding a CONTRIBUTING GUIDE, various
documentations, and start accepting pull requests. In particular, library authors
may want to report compliance level from their GitHub releases. We will work out
a way to display multiple – past, current and preview – versions of the specification from the website.

Next steps

Specification

The very first step would probably to merge JEP-12 in the specification and, thus, create an updated version. My understanding is that 'Raw String Literals' have been accepted but are not officially part of the specification yet.

The JMESPath proposals site has also been forked unofficially for this purpose.
proposals have been converted to markdown for wider consumption and it will start accepting pull requests as well.

As we have never maintained a sizeable community effort, I’m not sure how to proceed.
But we will probably setup a voting system for accepting proposals. We would like the library authors to have a voice too.

Functions contributions

The proposals site will also be accepting functions suggestions.
As this probably does not require a large consensus, I expect many functions could
be added quickly. For this reason, we came up with the concept of function sets.

A function set is a versioned set of functions that library authors choose to implement. We expect to surface the compliance level of each libraries against any particular function set in the site. Maybe as a comparison matrix or in any other way that makes sense.

As an early step, we have split the functions specifications to their own files.
This is a rather clumsy step but I’m sure we can improve from there.

What Next

As I said, @jdevillard and I are both new to this. Please, help us out.

As a critical step to start accepting issues, we will need to break the fork relationship at one point. It is with great regret that I started this initiative.

So please, dear community, make your voice heard. And please, if you can read this @jamesls, this is not too late for me to fold and offer constructive help with my contributions to the original – official – effort you had started in the first place.

[innovate-invent]

Thank you for all of this work!

Would you be interested in listing off specific tasks that you have planned/need done? I and others would be more likely to pick away at them if we knew the best way to help.

[innovate-invent]

I just spotted some very recent activity by @jamesls : jmespath/jmespath.py#268

[springcomp]

Would you be interested in listing off specific tasks that you have planned/need done? I and others would be more likely to pick away at them if we knew the best way to help.

@innovate-invent Indeed, I planned to do this using issues, but forks cannot receive issues and I would rather wait for wider feedback before breaking the fork relationship. But maybe by this time, we would already had feedback if our initiative was that interesting…

As I alluded to in my contribution above, we need help on:

Setup up community guidelines

• CONTRIBUTING GUIDE.
• CODE OF CONDUCT.
• Reach out to library authors to gauge interest.
• Consider a voting and governance system for proposals acceptance
• … ?

Setup issue templates

• JMESPath enhancement proposals templates.
• New functions suggestion templates.

Website

• Display multiple versions of the specification + documentation

Actions

Decide on what to do next.
My personal feelings are:

• Add a lot of compliance tests that we used while implementing our JmesPath.Net library.
• Implement JEP-11
• Re-introduce JEP-12 in the spec
• Consider implementing the ES6 style syntax for multi-select-hash.
• Consider implementing the ternary operator. Or maybe something like an iff function.
• Decide on the status of items(), from_items() and zip() functions.
• …

[springcomp]

Dear all,

I have made what I think is good progress curating some content and backlog that could be candidate for the next release of a specification.

• I have initialized a milestone that we could build with the community. It currently contains the JEP-09 Lexical Scoping proposal only, that I assume will be accepted, but we need to decide what we want to include.
• There is a reasonably complete set of proposals from requests that I could identify in various projects. Please, do make your voice heard.
• I also added some issues to discuss before some of them could be turned into proposals. Please, let me know your feedback.

[jamesls]

Hey everyone, I wanted to chime in here with my thoughts.

First, thanks to @springcomp for the interest in the project, and in wanting to move the project forward with improvements to the spec as well as the main site. I think that's great.

To give some context to the current state of things, the JMESPath spec has been purposefully static for a while, with no new language features being added. This was originally intended to give the JMESPath library authors a chance to catch up and allow their libraries to stabilize. That way you could have consistent behavior/semantics regardless of how you were using JMESPath.

I've let too much time pass without enhancing the spec, and there seems to be several consistent themes of features that users are requesting. I would like to address that now and am open to adding new functionality to JMESPath.

I would like to have a process in place where we define exactly how changes are proposed and added to JMESPath. I'd like this process to also include community involvement, so that anyone that wants to participate in these dicussions has the opportunity to do so. This will likely evolve over time as the process is refined.

There's also the more pragmatic day to day concerns. While I try to check in on several of the JMESPath repos (e.g. jmespath.py), I've not had time to keep up with all of the various repos in the jmespath org, including this jmespath.site repo. I will spend time this week triaging and getting this repo into a better state, but long term I would love to hand over maintenance of this repo to interested members of the community.

I think the next steps would be:

1. Flesh out the entire process for proposing new changes to JMESPath.
2. Start the review process on existing proposals in the jmespath.jep repo.
3. Triage through the existing issues/prs on this repo.

Feel free to share any thoughts/concerns you have.

[springcomp]

@jamesls

Thank you for your feedback.

JMESPath.site Updates

There's also the more pragmatic day to day concerns. While I try to check in on several of the JMESPath repos (e.g. jmespath.py), I've not had time to keep up with all of the various repos in the jmespath org, including this jmespath.site repo. I will spend time this week triaging and getting this repo into a better state, but long term I would love to hand over maintenance of this repo to interested members of the community.

For the record, before we broke the fork relationship, I had already made a batch of changes by merging all currently pending pull requests concerning minor changes to the site like typos and references to implementations. I also updated the embedded version of JMESPath that powers the live samples and tutorials.

I also updated the generation to support up to date libraries, while keeping the Sphinx-based generation. This was also converted to GitHub actions workflow.

Finally, I split the grammar file and also split each functions documentation into their own files for better sharing.

You can find all this in the master branch of the jmespath-community/jmespath.site repository we initiated.

If you think this work is a reasonable start, I would be happy to contribute this back in this repository. I posit that you going through this tedious work again would be a waste of time.

JMESPath Enhancement Proposals

I would like to have a process in place where we define exactly how changes are proposed and added to JMESPath. I'd like this process to also include community involvement, so that anyone that wants to participate in these dicussions has the opportunity to do so. This will likely evolve over time as the process is refined.

The effort that we had started is completely aligned with those goals. You may have seen the discussions page were we made every effort to outline pre-proposals with the objective that sufficiently fleshed out and thought through discussions would become JEPs that could be voted on and accepted by the community.

Those discussions were arrived at by first combing through all the existing repositories – site, spec, and library implementations; all the pull requests, discussions and issues that were raised; as well as current competing forks. We are confident we made a great job categorizing those requests in a structured way.

Again we would be happy to contribute this back to you.

The workflow we envisioned is currently demonstrated with a pending pull request with the goal to accept the JEP-11 Lexical Scoping proposal (aka the let() function), with a preliminary version published for demonstration purposes.

JEP process and site generation

Please, let me stress that – since the master branch referred to above – we made extensive changes in the main branch so that we streamlined everything into only two repositories:

Specification
https://github.com/jmespath-community/jmespath.spec

• JEPs have been converted to Markdown for better consumption.
• Compliance tests are embedded in the documentation for functions and other grammar constructs.
• The specification is updated automatically from pull requests.

Website
https://github.com/jmespath-community/jmespath.site

• The website draws its content from the spec repository and generates the samples, specification rendering, etc.
• It requires no manual steps as everything is generated automatically.
• It includes examples, references to libraries, tutorials and examples from Wikis.
• We also pioneered Frequently Used Patterns to serve as a help for newcomers into alternatives to perceived missing features in the spec.

The website also displays every versions (previous, current and on-going preview version being worked on) simultaneously with a dropdown box. It is easy to refer to a previous version of the spec and see what changed.

Again, we would be happy to contribute those changes here.

Final words

As you may have seen, @jdevillard, @innovate-invent and I would be happy to steward the specification in a more sustainable way. It would be a honor to have your endorsement if you feel we have done a good job.

[jamesls]

If you think this work is a reasonable start, I would be happy to contribute this back in this repository. I posit that you going through this tedious work again would be a waste of time.

Yeah this would be great. I would probably break the contributions down into several categories of changes for jmespath.site, from least controversial to change that may need more discussion:

1. Typo fixes, rephrasing to make things more clear
2. Infrastructure related changes, javascript library updates, switching to github actions, sphinx updates, etc.
3. Proposed new content changes, new pages, etc.
4. Proposed UI/layout/theme changes.
5. Implementation/internal changes, making the site data driven from a separated spec repo, etc.

(1), (2) and (3) should be pretty quick, (4) and (5) may take a bit more time. I'd be happy to accept contributions, otherwise I plan on addressing (1) and (2) in the next day or two.

Those discussions were arrived at by first combing through all the existing repositories – site, spec, and library implementations; all the pull requests, discussions and issues that were raised; as well as current competing forks. We are confident we made a great job categorizing those requests in a jmespath-community/jmespath.spec#38.

Awesome, thanks for doing that. I'll spend some time over the next few days familiarizing myself with those proposals.

[jamesls]

I put together a draft of what a more formalized jep process could look like: jmespath/jmespath.jep#14 . It still needs some details worked out, but I'd like to get a discussion started around it.

[springcomp]

@jamesls I pushed some pull-requests that may help you:

Yeah this would be great. I would probably break the contributions down into several categories of changes for jmespath.site, from least controversial to change that may need more discussion:

1. Typo fixes, rephrasing to make things more clear
2. Infrastructure related changes, javascript library updates, switching to github actions, sphinx updates, etc.
3. Proposed new content changes, new pages, etc.
4. Proposed UI/layout/theme changes.
5. Implementation/internal changes, making the site data driven from a separated spec repo, etc.

(1), (2) and (3) should be pretty quick, (4) and (5) may take a bit more time. I'd be happy to accept contributions, otherwise I plan on addressing (1) and (2) in the next day or two.

#98 implements point (1) referred to above from previously existing pending pull requests in this repository.

#99 and #100 and #102 and #103 implement point (2) referred to above. #99 fixes the generation with modern Python libraries, while #100 implements a GitHub actions workflow to build the site. It currently targets GitHub pages so use this as a starting point for what you want to do. #102 updates the embedded JMESPath JavaScript implementation. Finally, #103 adds references to C++ and Elixir implementations of JMESPath.

#101 goes somewhat beyond but proved a nice improvement for us. It splits the grammar ABNF into its own file for better sharing and reference. It also splits every function documentation to their own include files for better concurrent work on the specification as we anticipated the need to add functions at a more frequent pace.

[jamesls]

Just a quick update, I've merged all of the linked pull requests except for #100 / #101 which will take some time to go through. I'd like to switch over to github actions, but keep the existing infrastructure in place for now. I also like the idea of splitting up the spec/grammar, just thinking through if it makes more sense in a repo that's separate from the front-end website.

Thanks for all your work on this!

[innovate-invent]

I would separate the website from the spec repo. That way you don't have commits for formatting and presentation being mixed in with specification changes.

[springcomp]

@jamesls how can we best help you moving forward. Please let us know.

[springcomp]

Dear all,

While brainstorming ideas, we are making good strides at drafting some proposals.

Please let us know your feedback and feel free to share your ideas as well.

[springcomp]

Dear all,

Our vision for what could constitute the next major milestone for JMESPath has been detailed.

It includes the following proposals:

• General Behaviour
  • Documenting provisions for third-party functions.
  • Enabling string slices.
• New Syntax
  • arithmetic-expression.
  • Compact syntax for multi-select-hash.
• New Functions
  • Lexical Scoping and the let() function.
  • Object Manipulation items(), from_items() and zip() functions.
  • String functions.
  • Grouping group_by() function.

All those features are demonstrated here

Please, feel free to head over the preview page and provide feedback.

[springcomp]

Dear all,

I'm proud to share the new version of the specification for what we call JMESPath Community.

You can even test drive all the changes!

Please, feel free to share your feedback.

[sebastien-rosset]

After reading this issue, I'm still confused about the direction. There seems to be two variants of the specs. Does that man library developers will have to decide which version of the spec they implement?

[springcomp]

After reading this issue, I'm still confused about the direction. There seems to be two variants of the specs. Does that mean library developers will have to decide which version of the spec they implement?

Yes, I’m afraid so 🤷‍♂️

JMESPath Community was started as a community effort to try and bring improvements at a much quicker pace than with this original repository. At the same time, JMESPath Community contributes to discussions and proposes pull requests to this repository and various upstream libraries.

JMESPath Community is designed to be compatible with this spec and will only bring improvements requested from the community but not necessarily approved here by James.

[jamesls]

Just want to clarify a few things here to prevent confusion.

There is one official version of the JMESPath specification, which is the
version maintained in this repo here. Other variants you see
references to are forks of the specification. While these forks were created
with good intentions, keep in mind that they are not officially maintained
by anyone in the JMESPath org.

If you are a library developer, implementing the official JMESPath
specification gives you the most compatibility with other tools that
use JMESPath, which generally implement the official JMESPath specification.

I realize that I have been fairly lax with people linking to unofficial
forks within the repos in this org. However, now that JMESPath feature
development has picked back up, I want to make sure we minimize confusion for
people visiting these repos.

In the interest of a clear message to new users that come here,
I'm going to ask that we start directing people towards
the official jmespath repos, including the official jmespath enhancement
proposal process. We encourage
and welcome contributions from everyone. We are slowly working through our
backlog of existing issues and proposals so it may take time, but we are
making progress.

Hope this helps clarify things, and thanks everyone for your understanding.
I look forward to everyone's contributions, and I'm excited to see JMESPath
evolve!

[springcomp]

@jamesls Thanks for clarifying and I personally would like to thank you for your work.

If at any time I caused confusion, I would like to apologize. My intentions were always to bring constructive input and I also always tried to contribute back to this organization’s various upstream repositories.

I’m looking forward to contributing again in this repository.