[RfC] new openHAB marketplace structure proposal

[Hilbrand]

In had this posted before in #6673, but it felt a bit buried so I moved it to it's own issue. With the idea to update it based on feedback

Here is a proposal for the new market place extension.

The idea is a marketplace is a git repository. The marketplace extension will clone this repo and regulary update to get the latest updates of that repo. It should be possible to configure one or more git repositories to get add-ons from. The repository only contains meta data and will not host the add-ons.
The word 'add-on' is intended in the broadest way. Theoretically an add-on could also be a python rule. (Except for the fact that different types of add-on might need a different implementation and thus the definition of an add-on is practically terms limited by what is supported by the implementation).
In the repository there will be a repository.json file and a directory per add-on:

|-repository.json
|-Example-1
|       \__addon.json
|        |_logo.png // optional
|-Binding-2
|       \__addon.json
|        |_logo.png
...

Each directory contains a addon.json file and optional a logo.png file. we might want to dictate the logo size. The repository.json contains some generic information about the repository. For example:

{
  "name": "openHAB Marketplace Add-ons",
  "url": "https://openhab.community/addons",
  "maintainer": "John Do"
}

The addon.json file contains the meta information about the add-on. The idea is that it supports multiple versions.

{
  "name" : "Example",
  "description" : "This is an example binding.",
  "url" : "http://mydomain", 
  "license" : "EPL-2.0 ",
  "maintainer" : "My Name",
  "package_type" : "bundle",
  "package_format": "kar",
  "packages" : [
    { 
      "version" : "1.0",
      "maturity_level" : "production",
      "version_compatibility" : {
        "from" : "2.5.0",
        "to" : "2.5.*"
       },
      "package_url" : "https://mydomain/mydownloads/my.cool.binding-1.0.kar",
      "readme_url" : "https://mydomain/mycoolbinding/1.0/readme.html",
    },
    { 
      "version" : "2.0-SNAPSHOT",
      "maturity_level" : "alpha",
      "version_compatibility" : {
        "from" : "3.0",
       },
      "package_url" : "https://mydomain/mydownloads/my.cool.binding-2.0-SNAPSHOT.kar",
      "readme_url" : "https://mydomain/mycoolbinding/2.0/readme.html",
    }
  ]
}

Accepted maturily_level values: alpha, beta, production

My questions are:

1. What to think of this approach?
2. What about the content of the json files? Is something missing? Or should something be different? All is open for discussion. It's just a proposal.

I'm working on a poc for this approach so any feedback would be great.

P.s. the approach and structure is not completely original, but inspired by another add-on repository 😉

[Hilbrand]

In the other topic @mvalla posted feedback. I've inlined those comments here:

Nice proposal.
What I would miss from current Marketplace in order of importance is:

development_status (beta, production, etc)

I've added the maturity_level. This terminology is used in the Eclipse IoT marketplace

screenshot_url (multiple)

I tried to limit the amount of additional information and would assume if screenshots are relevant they should be in the readme. Having screenshots also would mean the user interface must display them somewhere, and I don't know where. Or are you thinking about screenshots like in Android marketplace?

source_url

Might be a good addition.

submitted_by (and id and not just a descriptive name)

Can you give an example? I'm not sure if I understand what your mean?

first and last being useful to filter/search on.

[mvalla]

Having screenshots also would mean the user interface must display them somewhere, and I don't know where. Or are you > thinking about screenshots like in Android marketplace?

Yes, an UI on the new OH 3 UI will show the marketplace bindings and perhaps a web UI will show them from a browser. Yes, screenshot like in Android and Eclipse marketplace. Not maybe urgent, but nice to have.

submitted_by (and id and not just a descriptive name)
Can you give an example? I'm not sure if I understand what your mean?

like using as value the GitHub username of the submitter, so that I can search/filter all bindings from the same developer. Again not urgent but nice to have.

[Hilbrand]

Screenshots might be interesting if the marketplace also would support ui widgets. So might be interesting. Possible just have the option to add additional image files with a specific naming convention. We'll see how the ui itself will develop on this part.

There is an entry maintainer that covers it I think? Maybe that name should be something else? One should be able to filter on that at least.

[bwosborne2]

First a disclaimer. I am not a Java programmer.

i like the ideas used in Home Assistant where they have a curated/moderated cataloged repo for trusted community addons as well as permitting personal repos of addons. That it much more open and community focused than requiring either inclusion in the distribution or manual installation.

[ghys]

I'd like to float the idea, of considering what's HABPanel is doing for its widget gallery and use the Discourse API on community.openhab.org instead of a repository to index the community add-ons. Basically what this does is transform the topics in a category with the widgetgallery tag i.e. this output:
https://community.openhab.org/tags/c/apps-services/habpanel/widgetgallery.json
into a more structured:

http://demo.openhab.org:8080/rest/habpanel/gallery/community/widgets

http://demo.openhab.org:8080/rest/habpanel/gallery/community/widgets/103475

There the code automatically seeks a *.widget.json attachment in the OP (or a GitHub link), there could be something similar here, i.e. mandating a marketplace tag and properly formatted addon.json attachment to consider the entry as valid. You can also define standard tags like alpha, beta for maturity levels and some for the type (widget, binding...).
Note that the GitHub repositories are also supported.
That's also what drives this page on the website: https://www.openhab.org/about/showcase.html

I know it's a little unconventional but it's had its little success and IMHO the following advantages for a community-driven gallery:

• Post-moderation and reactive community moderation vs. pre-moderation i.e. no waiting for a curator to approve entries or changes; there could be frustration if there's a bottleneck in the reviews and a few days of delay, not only for publishing new entries but also updating existing ones with new versions. Instead, the author gets an instant gratification of having its addon directly visible. Yes, it's putting a great deal of trust in the author and the community to self-moderate but OTOH bad or malicious entries can quickly be removed by any member above a certain level: you just remove the "marketplace" tag and it's gone. There are a lot of active members in that position who visit the forum everyday and would happily do it, so a bad entry would only stay for a couple hours. Bad actors who repeatedly break the rules can be banned. Having a review process for additions in the community gallery can be considered in the end as just an alternative distribution, only with the code review process skipped. I do realize it could backfire - while HABPanel widgets are relatively benign, addons can do harm... but would a central curating authority test & audit every addon anyways?
• there's no Git/GitHub involved, not everyone is comfortable with it, or wants to bother creating a repo, or has an account - if the addon is just one snippet of code the author wants to share, they could simply attach it to their post, this reduces the barrier to entry significantly;
• you get an automatic discussion thread for the entry where people can comment, including with the author (as the OP they are automatically subscribed to their own thread so they are likely to respond to inquiries quicker);
• you can easily attach images, screenshots without uploading them separately (Discourse gives a link to the first image in the index so you can easily display it in a gallery);
• you get a number of views, likes, replies to the entry so you can sort by popularity;
• it's managed by the openHAB Foundation instead of a third-party.

[bwosborne2]

Ideally there would be a way to include addons, through the REST API without tying to the Java programming language. That is the power of a REST API.

[kaikreuzer]

Thanks @Hilbrand for taking this challenge 👍 .

Just a few comments:

The word 'add-on' is intended in the broadest way.

I would not suggest doing so. So far, add-ons are fairly cleanly defined. They belong to one of the (few) pre-defined categories, are implemented as an OSGi bundle with a strict naming convention (org.openhab.<addontype>.<id>), installed as a Karaf feature, documented with a README, which is published at https://www.openhab.org/addons/.
I would keep the term "add-on" for exactly this.
A "marketplace" can be certainly much broader though, i.e. besides add-ons, other stuff can be offered there - like "HABPanel widgets", "rule templates", "Jython helper libraries", etc. I would not want to put constraints here, but I'd very much wish that we classify such things and for each document what it actually is, what is expected when adding one of these to the marketplace etc. Otherwise I think it can become messy quickly and easily.

use the Discourse API on community.openhab.org instead of a repository

Although I was always suggesting to go for e Git repo, I quite like @ghys' proposal and he's clearly got a point that this approach has already been proven to be a viable option as it has indeed gained quite some popularity among our community.
It also will be hard to challenge the long list of his arguments for it 😜.

[Hilbrand]

@ghys and @kaikreuzer I do like the idea of using discourse. It did trigger me on 2 things. First the idea behind the addons.json is that it contains all information relevant to the ui, but I didn't mention this here. To know what information is needed by the ui I think we might need to do 2 things. One is to write down what information is relevant for the ui and two what information needs to be in the addon.json that is not provided via the discourse api and thus might be needed in the addons My proposol is a first attempt to write this down. So any feedback on this is very appreciated.

On a related note. I did some experimenting some months ago with the rest api and found that the api for the marketplace is slightly different from the api for the bindings provided by openHAB. This makes it difficult to use the same functionallity in the ui as it needs slightly different implementations. If we could change this such that we could use the same api interface specification for provied addons as marketplace addons the ui would benefit from this too. (And while writing this I realize I miss an unique addons id in my addons.json,. Because one feature in my proposel for example is that I also wanted to address the lack of versions that is currently in the eclipse marketplace. I always found this missing. For example I would like the marketplace to offer an updated version of the binding availble for testing. With an addon specific UID the UI can show them as 1 entry with a selection box to allow the user to easily select a specific version of an addon).

This brings me to my second trigger. A more fundamental choice between a discourse based vs git based marketplace is that a discourse based is very bound to openHAB, or at least has a higher entry point if someone else want to provide such a service. While a git based has a lower entry for others to provide such a marketplace. This could be interesting for others to host their own. I don't necessarily favor this ability. It's just that it's the option to make this possbile.
But maybe more relevant is the difference between the current habpanel widgets and addons. For the widgets discourse is the only source to get the widgets. However for addons the primary source is what is provided with openHAB. If for bindings the marketplace would go through discource this would give the addons provided by openHAB a disadvantage because you can't see any statistics, or screenshots for those bindings. So it might be less attractive to submit the binding to be included because this might affect that information. And what if a binding is included with openHAB, the binding in discourse might look more attractive in the ui than the official binding.

So in summary I think there might be a use case for having 2 different type of markteplace infrastructures. And we might decide to use which infrastructure based on if we have a specific type of add-ons only available via a marketplace (like the habpanel widgets) or have a hybrid structure (like bindings and the Eclipse marketplace is).
But I think we can already specify what information the UI api would need to display and install those add-ons. So I would like to find out what information is still missing from the proposed addon.json.
Based on a complete insight we can even decide if some of that information doesn't need to be in the addon.json but is provided via other means (like the discourse api). This could be the starting point for an improved marketplace extension and we can even reuse the current habpanel api and the eclipse marketplace to implement that extension an provide 2 different ways to distribute addons or we can deside to implement only 1. in the end I have no specific attachment to either way. My primairy question is what information does an addon need to provide and I'm looking for input for that.

P.s. @kaikreuzer I meant by 'addon in the broadest way' not to redefine the word addon, but I don't now of any other word to cover mutliple different types, from bindings, to rule template, to widgets, etc. So i you know what might be a beter wordt that covers this I would use that.

[ghys]

the api for the marketplace is slightly different from the api for the bindings provided by openHAB. This makes it difficult to use the same functionallity in the ui

I don't think it necessary has to. In OH2 the 2 sources (distro & marketplace) were merged together, but I think it has several drawbacks. What I had in mind is indeed to manage the marketplace offerings with a completely separate API (/rest/marketplace) and UI. Maybe it would look like this in the menu (the marketplace menu appearing only when the /marketplace endpoint is referenced in /rest/:

which would take the user to a dedicated screen to discover marketplace add-ons.

As you mention the categories of the marketplace offerings might be completely different indeed than what the distribution currently offers, for instance:

• add-ons (i.e. an installable .jar bundle, like a binding or a voice API)
• rule templates
• UI widgets
• ...

We can make the UI emphasize the "official" nature of the add-ons which are included in the distro and passed through the review process, in some lists where a merge has to be done (like "select a binding" when adding a thing), we could add a visual cue:
(like the verified accounts on Twitter)
Not that the marketplace add-ons are necessarily untrustworthy but it might incentive enough for authors to strive to get this check mark as a proof of trust from the project.

A more fundamental choice between a discourse based vs git based marketplace is that a discourse based is very bound to openHAB, or at least has a higher entry point if someone else want to provide such a service. While a git based has a lower entry for others to provide such a marketplace. This could be interesting for others to host their own. I don't necessarily favor this ability. It's just that it's the option to make this possbile.

I think a sufficiently generic system of "marketplace providers" could solve this problem - the one maintained by the openHAB project would be based on Discourse but another vendor might provide a different marketplace provider as well.

[rkoshak]

As someone who really really wants a marketplace where he can deploy rule templates and rules helper libraries one really challenging feature I'd love to see that I've not seen mentioned is dependencies. For example, I've now a set of Python libraries that I'm maintaining and making available for those who want to use them. But they depend on the Helper Libraries and the Jython addon Scott maintains.

While it would be great to have the marketplace see these dependencies and make sure they are automatically installed a la apt and yum, at a minimum I would hope the dependencies to be documented in the addon.json and rendered in the UI. I expect this would be mostly a concern with stuff like HABPanel widgets and rule templates, but it feels like something that should be acknowledged if not addressed.

One other note, if we choose Discourse as the distribution mechanism we may need to change the setting that closes new theads after a month of no activity for those marketplace add-on posts. This is especially relevant if we want to have that post also serve as the help thread.

[kaikreuzer]

Not wanting to make things more complicated, but we so far have only talked about building a marketplace to browse stuff. I think we will hit the real problems when talking about installing entries - all our different categories (add-ons, rule templates, UI widgets, etc.) have very specific requirements/technologies on how they are installed. They might also have their own mechanisms for handling dependencies (like e.g. Karaf features do).

I am wondering, whether those different categories qualify for more or less independent marketplaces (while clearly the "openHAB Marketplace" would be a bracket over all of them). But the way they are presented, the fields that needs to be provided, the metadata that the installation mechanism requires, etc. might be very different for all of these.
I guess we should hence define the scope of our first step for the marketplace: Most discussion was around add-ons, so it might make sense to tackle them first (although I'd also love to see a solution for rule templates soon, but I fear that this will bring much additional complexity to the table...). Wdyt?

[bwosborne2]

Wouldn't UI Widgets be specific to the UI which would be independent of OH, just accessing OH through the REST API? It appears to me UI & UI widgets are VERY different from the other categories.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/continue-maintaining-my-binding-to-the-latest-branches/114010/16

[wborn]

The new Marketplace is open for business! 🙂