Adding PowerQuery language definition

[mattmasson]

Hello!

I'd like to add a definition for the Power Query formula language (also know as M).

Search for existing Power Query (.pq) files

One potential extension conflict is with Makefile.pq files, but this extension isn't registered in the languages.yaml file. I'm not sure if there a way to exclude these files explicitly, or if it won't be a problem.

Thanks!

[mattmasson]

Just wanted to bump this PR.

[Alhadis]

👋 Hey @mattmasson, sorry to keep you waiting!

The in-the-wild usage looks a little too light: of the 72 URLs collected from the current search results, I only counted 21 unique repositories, distributed between 16 GitHub users:

https://github.com/EXASOL/powerbi-exasol/
https://github.com/Hugoberry/PowerQueryConnectors/
https://github.com/Hugoberry/PowerQueryExtensions/
https://github.com/Kjonge/PowerBIStravaConnector/
https://github.com/Microsoft/DataConnectors/
https://github.com/boschand/PBiDataConnector/
https://github.com/cwebbbi/PowerBITextAnalytics/
https://github.com/fluf1024/PQGoogleSpreadsheet/
https://github.com/foxan/SnagR/
https://github.com/hohlick/PowerQueryModules/
https://github.com/jeroenterheerdt/PowerBI-CBS-OpenData/
https://github.com/mattmasson/PowerQuery/
https://github.com/migueesc123/CalendarCreator4PowerBI/
https://github.com/migueesc123/MeetupConnector/
https://github.com/migueesc123/PowerBIRESTAPI/
https://github.com/migueesc123/WooCommerceConnector/
https://github.com/migueesc123/YoutubeAnalytics/
https://github.com/mprozil/PBIGoogleFitDataConnector/
https://github.com/necroze/justaclone/
https://github.com/ruthpozuelo/GoogleAnalyticsConnector/
https://github.com/zhll2046/test/

Please make sure you've read our guidelines on what we require of a language or extension before it can be registered on GitHub. Note this part in particular:

In most cases we prefer that each new file extension be in use in hundreds of repositories before supporting them in Linguist.

Are there any other file extensions we could be checking, which have wider usage across GitHub?

[mattmasson]

There are a lot of code files using the .M extension (which get incorrectly classified as matlab/m/objective-c...). This was the best combination of common words included in Power Query code I could find, which adds another 73 code files.

Other searches could be M topics and PowerQuery topics.

[mattmasson]

To add some more data points...

The Power Query / M language is published on MSDN, used heavily in Microsoft Power BI, and is built into modern (2016+) versions of Excel. We have about 3 million people using it on a monthly basis. We've only recently published our SDK, so I expect we'll see a big uptake in the number of github projects in the next few months.

[cwebbbi]

I'd like to support Matt here - this language is about to explode in popularity over the next few months.

[Alhadis]

Adding preemptive support on the premise that there will be ample in-the-wild usage in the near future... hrrm, no, sorry. My recommendation is to wait for the language to grow in popularity and usage, then revisit this with a pull-request.

[spiritek]

I think this would be great, though the fact that the source for Power Query queries are predominately stored inside Excel or Power BI files with no easy way to export would explain the lack of usage as it is usually bundled within the file that uses it. Maybe if MS made an easy way to import/export the queries source control might get used more for Power Query/BI

[roryneary]

Power Query ships with every installation of excel 2016, it exists in Power BI desktop and it's even available as the extract transform and load language for the common data model in Microsoft Dynamics. It is very much 'here to stay' so anything that supports it is very much welcome.
I agree with @spiritek comments that because it sits inside other files it lends the appearance of it not being so widely used. A better search would be for .pbix as every one of these files will contain M code.

[Seddryck]

@Alhadis I don't know how you're counting the repositories with a .pq extension but I'd like to point that some are missing. You'll find here two examples (this list is not exhaustive, just a quick check).

• https://github.com/tycho01/pquery
• https://github.com/Seddryck/Power-Query-M-session

Due to lack of support for the power query language in github, many users are using the .txt extension such as https://github.com/nimblelearn/datapackage-m

[Alhadis]

Due to lack of support for the power query language in github, many users are using the .txt extension such as https://github.com/nimblelearn/datapackage-m

It doesn't make any sense to do that, though. I'd say the owner of the repository you're linking to is simply unaware of the correct extension for PowerQuery files.

I don't know how you're counting the repositories with a .pq extension but I'd like to point that some are missing.

Well, I used whatever happened to turn up in public search results. If those two were missing, it's possible they weren't indexed previously (or they were previously private repositories that've since been made public).

/cc @lildude Could you handle this? I'm unsure how to respond.

[lildude]

I have to agree with @Alhadis. As things currently stand, from what I've read here, and investigated myself, the PowerQuery formula language is not popular enough on GitHub to warrant merging of this PR just yet and we can't pre-emptively plan for future growth as this opens up a huge can of worms for every other aspiring language.

Now a few points and questions based on comments in this PR...

One potential extension conflict is with Makefile.pq files, but this extension isn't registered in the languages.yaml file.

This PR will result in them all being identified as PowerQuery so either Makefile.pq needs to be added to the Makefile definition (at a quick glance it looks like a Makefile, but I could be wrong) as a specific filename or the .pq extension needs to be added, but then a heuristic will also be needed to differentiate the two usages.

There are a lot of code files using the .M extension (which get incorrectly classified as matlab/m/objective-c...).

If this is the case, is there a reason you've not included the .M extension in this PR?

If the concern is conflict with the MatLab files, you can add a heuristic to differentiate these two.

The Power Query / M language is published on MSDN, used heavily in Microsoft Power BI, and is built into modern (2016+) versions of Excel. We have about 3 million people using it on a monthly basis.

... and ...

I think this would be great, though the fact that the source for Power Query queries are predominately stored inside Excel or Power BI files with no easy way to export would explain the lack of usage as it is usually bundled within the file that uses it.

... and ...

I agree with @spiritek comments that because it sits inside other files it lends the appearance of it not being so widely used. A better search would be for .pbix as every one of these files will contain M code.

And there's the rub. If this is all true, then this is currently the biggest limiting factor. The predominant usage and storage of the language, as this stands today, is in "zip" files so not readily readable, and thus not searchable anyway.

That said...

We've only recently published our SDK, so I expect we'll see a big uptake in the number of github projects in the next few months.

... and ...

I'd like to support Matt here - this language is about to explode in popularity over the next few months.

... would certainly help with increasing the popularity of this language, especially if you're encouraging the use of separate .pq or .m files.

If you're right, and I don't doubt it, it won't be long (a few months?) until we see the usage needed to merge this PR. We really appreciate great PRs like this and I hope you can understand the direction we're coming from.

[mattmasson]

Will submit an updated PR