[marketplace] Future of Eclipse IoT Marketplace

[pfink]

What happends with the Eclipse IoT Marketplace in openHAB 3.x? Can we continue to use it or are we required to find a replacement solution. Any plans for this already?

Concrete question to the code of this repository: Should we remove the Eclipse IoT Marketplace Add-on from the master / 3.x branch?

[Hilbrand]

Good question. I'm not sure if it really is a requirement. However since it's outside our control we don't know what will happen with the market place and so we should have a replacement. I've seen @kaikreuzer mention this already. If I remember correctly the idea was to create a repository as a replacement. I think initially we could simply create a xml file using the same format as the current market place as an replacement. Where people can simply create pull requests for changes. (Although I would suggest some changes to the format). I guess we can always make it fancier. So if we would have a replacement we only need to adapt the current market place binding to the new situation. I don't know if someone has already plans, but if we would start with a simpel xml file based on the current format I can give it a start.

[kaikreuzer]

Concrete question to the code of this repository: Should we remove the Eclipse IoT Marketplace Add-on from the master / 3.x branch?

Yes, we have to remove it - as a non-Eclipse project, we must not rely on it any longer.

What @Hilbrand describes pretty much matches my ideas: Have a dedicated repo where updates can be done through PRs. XML might indeed not be the best format, maybe something like YAML could make sense. Best if we structure entries to be in dedicated files with maybe one index created automatically - this would reduce merge conflicts.
Afaik, nobody has started working on it, so if you want to have a go @Hilbrand, feel free! But maybe better wait until we have cleanly separated 2.5.1 from master...

[petero-dk]

When thinking about making an addon repository, would it not be a good idea to start on something someone has already built or at least look at other projects that solved the same issue?

I think I read @cdjackson suggesting that the UI for the users should be less confusing (having all addons in the same place) which also suggests that the underlying technology could be similar.

Here are some of the things that I though could be useful for the package management for OH:

• GitHub now has built in package repositories (https://github.blog/2019-05-10-introducing-github-package-registry/) perhaps using the maven style repository could work for this?
• Another way of using GitHub would be the releases, and having config files (yaml/xml) in the root of repositories describing the package(s). The naming convention of the releases could provide a way to managed updates just like today.

If Github was used as the package hosting it would be nice if the configuration of addon repositories acted like APT repositories for linux for users:

• there would be the openhab-addons
• there would be the openhab-addon-marketplace that points to other github repositories but are marked as approved
• users could add custom repositories (in the form user/repository) so there would not be the need to copy jars into the addon folder and autoupdates would also work for bindings in development. Additionally when the addon was accepted into the marketplace approved it would simple continue to "just-work".

This way it would be relativly simple to manage binding updates that are not bound to the version of the core. 2.5.x releases or packages would work on 2.5.x cores and autoupdate. 2.x core releases would mean breaking changes for addons. There could be a UI that shows updates and/or a openhab-cli command.

There are multiple projects that allow (albeit mostly developer focused) to download packages from github repositories. We should be able to steel some ideas from them, and other package management systems/addon systems.

Projects that I know if that uses GitHub as the package repository:

• The homebrew project for mac
• NPM for npm install visionmedia/express#branch

[benalexau]

A few quick comments:

• GitHub Packages as a Maven repo currently has many limitations.
• Hosted source repositories come and go. You probably remember when it was CSV on SourceForge, then Subversion on Google Code, currently Git on GitHub, and down the track probably SourceHut or GitLab or whatever.
• It's easier to encourage a third party addon ecosystem if users can easily release their work without approvals. Even simple approval steps like merging a PR still require lead time. Some alternatives may include a GitHub Action to auto-merge, keeping the URL list in a wiki, using GitHub Search to locate related projects, trawling the forum for JAR/KAR links etc.
• In the distant past I approached a similar use case using OSGi Bundle Repository (OBR) with a custom URL resolver (for a "httpgpg" scheme) so the OBR model didn't have to change. It just periodically batch downloaded user-provided GPG URLs and indexed them. It was very lightweight, decentralised and required zero ongoing effort. Plus having the bundles in one place facilitated download statistics for surfacing relative popularity metrics.

[kaikreuzer]

@petero-dk & @benalexau, many thanks for your input here!
I'd just like to clarify something about the idea: We are not talking about a package manager here and hosting the add-ons ourselves. The idea is to have a similar mechanism to the Eclipse Marketplace, which deals with the metadata management of the marketplace entries only - in the end, it is just a list of urls that point to anywhere on the web. For legal reasons, this should stay like this.

Hence the main question is, what metadata would we require (id, label, description, screenshots, version ranges, url,...) and what format shall we choose for storing that metadata, so that it is easy for people to maintain it through simple PRs.

As mentioned above, I could imagine that we simply say that every entry should be done as a separate YAML file in a folder that corresponds to the add-on category (binding, voice, persistence,...). At least this could be a simple starting point for this feature.

[bwosborne2]

Here are a few ideas used by Home assistant.

1. They have a method in the UI to add addon GitHub repos from individuals.
   2, There is also a community addon GitHub repo that is moderated by the HA developers and has standardized requirements.
2. They have an addon "store" addon that has an organized collection of community addons/

[Hilbrand]

Here is a proposal for the new market place extension.

The idea is a marketplace is a git repository.
The marketplace extension will clone this repo and regulary update to get the latest updates of that repo.
It should be possible to configure one or more git repositories to get add-ons from.
The repository only contains meta data and will not host the add-ons.
The word 'add-on' is intended in the broadest way.
Theoretically an add-on could also be a python rule.
(Except for the fact that different types of add-on might need a different implementation and thus the definition of an add-on is practically terms limited by what is supported by the implementation)
In the repository there is a repository.json file and a directory per add-on:

|-repository.json
|-Example-1
|       \__addon.json
|        |_logo.png // optional
|-Binding-2
|       \__addon.json
|        |_logo.png
...

Each directory contains a addon.json file and optional a logo.png file.
we might want to dictate the logo size.
The repository.json contains some generic information about the repository.
For example:

{
  "name": "openHAB Marketplace Add-ons",
  "url": "https://openhab.community/addons",
  "maintainer": "John Do"
}

The addon.json file contains the meta information about the add-on.
The idea is that it supports multiple versions.

{
  "name" : "Example",
  "description" : "This is an example binding.",
  "url" : "http://mydomain", 
  "license" : "EPL-2.0 ",
  "maintainer" : "My Name",
  "package_type" : "bundle",
  "package_format": "kar",
  "packages" : [
    { 
      "version" : "1.0",
      "version_compatibility" : {
        "from" : "2.5.0",
        "to" : "2.5.*"
       },
      "package_url" : "https://mydomain/mydownloads/my.cool.binding-1.0.kar",
      "readme_url" : "https://mydomain/mycoolbinding/1.0/readme.html",
    },
    { 
      "version" : "2.0",
      "version_compatibility" : {
        "from" : "3.0",
       },
      "package_url" : "https://mydomain/mydownloads/my.cool.binding-2.0.kar",
      "readme_url" : "https://mydomain/mycoolbinding/2.0/readme.html",
    }
  ]
}

My questions are:

1. What to think of this approach?
2. What about the content of the json files? Is something missing? Or should something be different? All is open for discussion. It's just a proposal.

I'm working on a poc for this approach so any feedback would be great.

P.s. the approach and structure is not completely original, but inspired by another add-on repository 😉

[mvalla]

Nice proposal.
What I would miss from current Marketplace in order of importance is:

• development_status (beta, production, etc)
• screenshot_url (multiple)
• source_url
• submitted_by (and id and not just a descriptive name)

first and last being useful to filter/search on.

[Hilbrand]

I've opened a new issue (#8448) as it feels it a bit buried here. @mvalla thanks for your feedback, I've commented on your feedback in the new issue.