Implement endpoints based on original package name

[AMDmi3]

There's demand to refer to Repology data only having original package name (and repository name likely). See #61 for instance. An URL schema should be invented to support this. We need this for at least project pages /project/<projname>/* (plus corresponding API endpoints) and badges:

/badge/vertical-allrepos/<projname>.svg
/badge/tiny-repos/<projname>.svg
/badge/version-for-repo/<repo>/<projname>.svg
/badge/version-only-for-repo/<repo>/<projname>.svg
/badge/latest-versions/<projname>.svg

• Should we use redirects (303? 307?) or just serve a target page with <link rel=canonical>
• Which package name should we use? Some repos provide multiple names (e.g. FreeBSD ports have path in ports tree devel/py-Jinja2 and (multiple) package names py27-Jinja2, py36-Jinja2, py37-Jinja2. May need to implement origin/uniquename field first (Track package lifetimes through keynames repology-updater#527).
• How to handle multiple choices? E.g. love package may refer to love, love05, love06 etc. projects (however these should go away as soon as we improve branches support (Introduce metapackage flavors/branches repology-updater#170).
• Should we redesign URL structure for consistency (this mostly affects badges)?

Project URL variants:

• /project-by-package/<repo>/<pkgname>/subpage (note the pkgname may contain slashes; also it's not apparent that repository name is used)
• /repository/<repo>/package/<package>/subpage
• /project?repo=<reponame>&package=<pkgname> (always redirect)
• universal variant /project/<pkgname|projname>/subpage which automatically resolves unambiguous cases, shows disambiguation page if there are multiple choices and may be refined to lower number of variants (to 1) /project/<pkgname>/subpage?repository=<reponame>

Badges variants:

  /badge/project/<projname>/vertical-allrepos.svg
+ /badge/project-by-package/<repo>/<pkgname>/vertical-allrepos.svg
  /badge/repo/<repo>/project/<projname>/version-for-repo.svg
+ /badge/repo/<repo>/package/<pkgname>/version-for-repo.svg

  /badge/vertical-allrepos.svg?project=<projname>
+ /badge/vertical-allrepos.svg?repo=<reponame>&package=<pkgname>
  /badge/version-for-repo.svg?repo=<reponame>&project=<projname>
+ /badge/version-for-repo.svg?repo=<reponame>&package=<pkgname>

[AMDmi3]

For the record, MacPorts guys want this feature as well. With that demand it should be implemented ASAP.

[AMDmi3]

Also I believe I've seen Repology package badges used globally on some Debian toolsite (somewhere around https://portfolio.debian.net?) - need to find that place, ensure that these badges link to repology projects by Debian package name (which would produce broken links in many cases), find Debian contact and make them switch to this feature as soon as it's implemented.

[AMDmi3]

See repology/repology-updater#944 for fresh ideas on this. In short, we may have multiple names (general, binary, source, each if appicable) to reference package/project from the outside, and these may be part of the URL.

[pabs3]

FTR, the Debian infrastructure currently only links to the Repology per-maintainer pages. For example my Debian portfolio and DDPO pages both link to my Repology maintainer page.

I want to add three Repology items to the Debian package tracker (for package maintainers) pages:

• the "action needed" panel should link to a page containing the problems detected by Repology for the package in the debian_unstable repo (Debian#898031)
• the "versions" panel should link to the Repology versions page for the right metapackage
• the "links" panel should link to the Repology packages page for the right metapackage

The best option for the second two items is for Repology to offer a redirect from the debian_unstable source package name to the Repology metapackage pages.

Later on we might also want to add similar links on the user-facing packages site.

[AMDmi3]

Let's put problems aside for now, as these need and will undergo a severe refactoring related to the way they are updated. Regarding the package specific urls there are four unresolved issues before we can start implementing it, that is why @pabs3 PR was lingering for months

1. Need to decide on an URL scheme, as it will be too painful to change later
2. Need to decide on the way of presenting per-package page (serve page with rel=canonical, 30x redirect/which one)
3. Need to make new name(srcname,binname) fields usage consistent across all repositories (New package name fields, take 2 repology-updater#944). Also decide on whether there should be fallbacks from one name fields to another.
4. Need to decide on how derived tables should be be updated Delta updates plan repology-updater#956 (as this would require pkgname + type(name,src,bin) -> projectname mapping table updated along with packages.

To not block this for much more time, I propose partial solutions for most of these issues, namely:

2. Just pick a random one without thinking over pros and cons, let it be 301. There's no problem in switching to another one later.
3. Limit feature with repositories for which this is resolved and/or which are interested in per-package links. Namely, Debian.
4. Just do it the easiest way - update once a while with rewriting whole table. With [3] it'd be even faster.

The only task left (and it does not really block the development of this feature) is thus to decide on url scheme. IMO we should build upon a consistent mapping between project-related urls and package-related urls, e.g. <prefix><project><suffix> → <prefix>-by-<type>-in/<repo>/<project><suffix>, which gives us project/<foo>/information → `project-by-{name,srcname,binname}-in/debian_stable//information. This more or less works for badges too.

[pabs3]

I suggest using 307 for redirects instead, since 301 means moved permanently and the mapping between a distro package and a Repology project could change over time. In any case, I definitely think we want redirects and not rel=canonical. https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection I had intended for this feature to not be restricted to Debian, it would definitely be useful for other distros to link to Repology. As long as the URL design accommodates that I'll be happy as the corresponding database changes can happen at a later stage. I had previously preferred /repository/<repo>/package/<package>/subpage but now I can see that your /project-by-*-in/<repo>/<package>/subpage proposal is a better name for this feature. For the badge URLs we can use /badge-by-*-in/ as a prefix. I'm not very good with databases, so I'm going to have to leave any database changes needed with you. I think one remaining question is if the current database would ever map a distro source/binary package name to multiple Repology project names and what to do when that occurs. I guess the code could just return a HTML table with links to each of the Repology projects.

…

-- bye, pabs https://wiki.debian.org/PaulWise

[pabs3]

Does Repology create corresponding metapackage/project names for all packages in Debian? If not then an additional API is needed so that the Debian tracker can show the Repology version link only for Debian source packages that are listed on Repology.

[AMDmi3]

Repology parses all Debian source packages and maps them into projects which names does not necessarily match the package names.

[pabs3]

It is completely untested but I've prepared a branch using the new URL scheme:

master...pabs3:package-urls

[AMDmi3]

This is now deployed in experimental mode. There are still inconsistencies in how different kinds of names are used so not all repositories are supported yet, for now it's only deb family. Feel free to submit urgent requests for additional repositories.

[pabs3]

Excellent, thanks for that! I noticed a couple of things: The binname support doesn't work for libc6 & debian_testing. The per-distro-per-distro-package problems pages are missing.

…

-- bye, pabs https://wiki.debian.org/PaulWise

[AMDmi3]

The binname support doesn't work for libc6 & debian_testing

As expected, we don't parse binary packages from debian.

The per-distro-per-distro-package problems pages are missing.

There are no per-package problem pages.

[pabs3]

There are no per-package problem pages.

Ah, but there were in my version of the changes. Should I re-implement the per-package problem pages feature?

…

-- bye, pabs https://wiki.debian.org/PaulWise

[ppmathis]

Based on the website linking to this issue, I will ask here despite it already being closed. Could you please consider adding the CentOS repositories to tools/project-by as well?

[AMDmi3]

Done

[xfournet]

Hi @AMDmi3 would be possible to add support for Scoop repository in tools/project-by ?

[AMDmi3]

@xfournet done. Assuming Scoop only contains binary packages.

[ryanprior]

@AMDmi3 can we add Guix support for this tool?

[AMDmi3]

@ryanprior can do, but I'm puzzled whether guix package names can be considered source package names, binary package names, both or neither.

[ryanprior]

I'll test it tonight!

[ryanprior]

@AMDmi3 I can't figure it out so far. Edit: figured it out, see below 😎

I wanted to try out a badge link for esbuild, so I navigated to:
https://repology.org/tools/project-by?repo=gnuguix&name_type=name&target_page=badge_latest_versions&name=esbuild

That doesn't work for me, it says "cannot find project."

esbuild's repology project versions page is here and shows a package in Guix, but the link is broken. It omits the package name and only includes the version.

Guix package: https://guix.gnu.org/packages/esbuild-0.7.16/
Link from project page: https://guix.gnu.org/packages/-0.7.16/

I tried a few other packages to make sure it's not just isolated to that one, and these show the same issue:

• https://repology.org/project/emacs/versions
• https://repology.org/project/visidata/versions
• https://repology.org/project/ripgrep/versions

Update

I did make some progress: by changing from name_type=name to name_type=binname it renders a badge. However, the badge shows the latest release version for esbuild, not the latest version in Guix. Part of why I want to use this tool is so that you can easily tell when browsing whether the Guix package is up-to-date or not. Is the badge supposed to show the latest package in Guix, or just the latest version in general?

This URL: https://repology.org/tools/project-by?repo=gnuguix&name_type=binname&target_page=badge_latest_versions&name=esbuild
Renders this image:
While the latest version in Guix (as reported correctly in repology) is 0.7.16 (until I update it! 😄)

Update 2

I keep reading and messing more and eventually found exactly what I want.
This URL: https://repology.org/badge/version-for-repo/gnuguix/esbuild.svg
Gives me the badge I was looking for:

So having found that I'm pleased! The issue with links to Guix packages mentioned above is still outstanding, but it seems unrelated to this tool so I opened #152 for that. The landing page for the tool could use some better documentation and explanation of how to use it - that's something I can take a stab at if you'd welcome a contribution.

Thank you!

[AMDmi3]

Gives me the badge I was looking for

That's likely not what you're looking for as this badge uses Repology project name which is not guaranteed to match package name in guix, that's what the project-by tool is intended to cope with.

That's how it's done:

It gives you the following link template:
https://repology.org/tools/project-by?repo=gnuguix&name_type=srcname&target_page=badge_version_for_repo&name=<NAME>
which, when you substitude guix package name, e.g.
https://repology.org/tools/project-by?repo=gnuguix&name_type=srcname&target_page=badge_version_for_repo&name=python-ilinkedlist
would redirect you to an actual badge URL, e.g.
https://repology.org/badge/version-for-repo/gnuguix/python:ilinkedlist.svg
Note that you cannot use badge with guix package name python-ilinkedlist directly.

could use some better documentation and explanation of how to use it - that's something I can take a stab at if you'd welcome a contribution.

This would be very welcome. The thing is that I, as developer, am unable to view it from the perspective of user without repology inner workings knowledge and thus to write a proper documentation. Besides, I'm not a native English speaker.

[maxcrees]

Do you mind adding support for Adélie to tools/project-by? (we only ship binary packages)

[AMDmi3]

@sroracle Done!

[nandahkrishna]

@AMDmi3 Would you be able to add support for Homebrew to the project-by tool?

[AMDmi3]

@nandahkrishna done

[vineethnara]

@AMDmi3 could you add support for Homebrew Casks too?

[AMDmi3]

@vineethnara done

[malmor]

Hey @AMDmi3, do you think support for Chocolatey could be added? ✌
I went over repology/repology-rs#81, where you mentioned that the repository requires source/binary name - but Chocolatey is still using "generic name" (repology/repology-rs#81 and https://repology.org/repositories/fields).

Does this mean that there needs to be something done first somewhere else?

Would be great to see Chocolatey added to project-by if possible 🙂

[AMDmi3]

Does this mean that there needs to be something done first somewhere else?

Well the reason for adding support by-request was to sort out package naming taxonomy. I'm still confused with it. Do you think that chocolatey packages classify as binary packages, we could switch chocolatey to binnames, otherwise we could go on with generic name.

[malmor]

For some context:
I'm using Renovate and want to update some Chocolatey dependencies (e.g. notepad++, lockhunter, git,). Renovate queries the Repology API (tools/project-by) searching for matching packages by binname or srcname (first binname, then srcname).

Do you think that chocolatey packages classify as binary packages

I'm not really sure what side-effects/consequences this might have. I guess for my use-case this would be great - but I can't really tell in general 🤷‍♂️

So if there are no cons I would love to see Chocolatey classified as binnames and white-listed for project-by. ✌

[malmor]

So what do you think - can we classify chocolatey packages as binary packages and enable them on the project-by endpoint? ✌

[AMDmi3]

I'm still thinking. It'd be the best to solve this ambiguity once and for all and enable the endpoint for all repositories.

[gzmarstone]

Not sure if this closed ticket is the right place for it, but I would like to see Amazon Linux 2 added to Project by package name.

Thanks!

[AMDmi3]

@gzmarstone done

[tedtramonte]

Big request for adding AUR to project-by!

[AMDmi3]

@tedtramonte done

[AMDmi3]

@malmor chocolatey finally done