Self-hosted composer registry on self-hosted GitLab gives wrong package url #10649

mrgesco · 2021-06-29T10:31:28Z

How are you running Renovate?

Self hosted

Please select which platform you are using:

GitLab self-hosted

Renovate version: 25.31.0

Describe the bug

I have created a Composer registry on a self-hosted GitLab instance and trying to use two composer repositories on a project.

When renovate parses the composer file and tries to check the version, it gives a 404 error while trying to check on packagist instead of the right repository url.

Relevant debug logs

Click me to see logs

DEBUG: Datasource 404 (repository=***REDACTED***)
       "datasource": "packagist",
       "lookupName": "***REDACTED***",
       "url": "https://packagist.org/p/***REDACTED***.json"
DEBUG: Failed to look up dependency ***REDACTED*** (repository=***REDACTED***, packageFile=composer.json, dependency=***REDACTED***)
....
DEBUG: packageFiles with updates (repository=***REDACTED***)
"config": {
    "composer": [
    {
        "packageFile": "composer.json",
        "deps": [
        ......
        {
            "depType": "require",
            "depName": "***REDACTED***",
            "currentValue": "^3.0",
            "datasource": "packagist",
            "lockedVersion": "3.0.0",
            "depIndex": 18,
            "updates": [],
            "warnings": [
            {
                "topic": "***REDACTED***",
                "message": "Failed to look up dependency ***REDACTED***"
            }
            ]
        }
        ],
        "lockFiles": ["composer.lock"],
        "registryUrls": [
           "https://wpackagist.org",
           "https://<GITLAB_URL>/api/v4/group/<GOURP_ID>/-/packages/composer",
           "https://packagist.org"
        ],
    }
...
DEBUG: exec completed (repository=***REDACTED***, branch=renovate/lock-file-maintenance)
       "cmd": "composer update --with-dependencies --ignore-platform-reqs --no-ansi --no-interaction --no-scripts --no-autoloader",
       "durationMs": 6604,
       "stdout": "",
       "stderr": "Loading composer repositories with package information\nUpdating dependencies\nLock file operations: 0 installs, 1 update, 0 removals\n- Installing ***REDACTED***/***REDACTED*** (3.0.0):

Have you created a minimal reproduction repository?

https://github.com/mrgesco/renovate-20210629

I have provided a minimal reproduction repository

The text was updated successfully, but these errors were encountered:

rarkins · 2021-06-29T10:51:16Z

Where's the repo?

mrgesco · 2021-06-29T11:33:03Z

Where's the repo?

Sorry... issue edited

github-actions · 2021-06-29T11:33:58Z

Thank you for providing a reproduction! 🎉 🚀

The Renovate team will take a look at the reproduction repository.

mrgesco · 2021-09-06T09:50:19Z

Any update on this issue?

HonkingGoose · 2021-09-06T10:16:40Z

I think we're still at the stage where we're figuring out what's going wrong, and if/how we want to fix it.

herndlm · 2021-12-07T08:24:38Z

Interesting, I seem to be having the same problem but not related to GitLab. My composer.json has 2 custom repos, basically looking like this

"repositories": [
        {
            "type": "composer",
            "url": "https://foo/bar/"
        },
        {
            "type": "composer",
            "url": "https://baz"
        }
]

And I see that renovate is able to lookup packages from baz but not the ones from foo/bar because it tries to look on packagist.

DEBUG: Datasource 404 (repository=xxx)
         "datasource": "packagist",
         "lookupName": "vendor/package",
         "url": "https://packagist.org/p/vendor/package.json
  DEBUG: Failed to look up dependency vendor/package (repository=xxx, packageFile=composer.json, dependency=vendor/package)

I can though access https://foo/bar/p/vendor/package.json just perfectly fine.

I'll try to debug / fix this later, but any hints would be appreciated. It's very strange that one repo works and the other doesn't, maybe this is a very simple thing that can be fixed easily

Somewhere else in the log the registries were dumped as in

"registryUrls": [
           "https://foo/bar/",
           "https://baz",
           "https://packagist.org
         ]

so it seems like the parsing was done correctly. @rarkins sorry to tag you directly but can you point me to the code that is triggering the lookup for a package with a specific registry URL? I wonder why it would trigger it incorrectly for package a but correctly for package b. And I hope that my issue is the same as here and I can fix it :)
UPDATE: I found the bug in my case and will prepare a PR. This could very well fix the GitLab issue as well, we'll see

herndlm · 2021-12-08T08:39:43Z

@mrgesco this might have been fixed with my changes that were released in 29.34.0. Let me know if it works now. If not fixed yet - I might be able to help out here. But I need at least a redacted output of your https://<GITLAB_URL>/api/v4/group/<GOURP_ID>/-/packages/composer/packages.json

LeoniePhiline · 2022-02-10T17:57:49Z

@herndlm

I have the same issue:

Context

renovate-bot running in self-hosted GitLab does not attempt fetching packages from self-hosted GitLab composer package registry.

Version

31.68.3

Docker image: renovate/renovate:31.68.3@sha256:d1c48ba1673039c42d59062ba917c4c7379aeba84d2d3807d4236fc4341f9995

Based on a copy of https://gitlab.cobytes.io/development/renovate-runner/-/blob/main/templates/renovate.gitlab-ci.yml

Configuration

`composer.json`

All redacted details are in <angle brackets>.

{
  "name": "<vendor>/<parent-package>",
  "description": "<Parent package>",
  "authors": [
    {
      "name": "LeoniePhiline",
      "email": "leonie@<vendor>.com"
    }
  ],
  "type": "typo3-cms-extension",
  "license": "GPL-3.0",
  "repositories": [
    {
      "type": "composer",
      "url": "https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer/packages.json"
    }
  ],
  "require": {
    "<vendor>/<dependency>": "^10.4",
    "typo3/cms-core": "^10.4",
    "typo3/cms-extbase": "^10.4"
  },
  "extra": {
    "typo3/cms": {
      "extension-key": "<parent_package>"
    }
  },
  "autoload": {
    "psr-4": {
      "<Vendor>\\<ParentPackage>\\": "Classes/"
    }
  }
}

Private repository

Output of https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer/packages.json:

{
  "packages": [],
  "metadata-url": "/api/v4/group/34/-/packages/composer/p2/%package%.json",
  "provider-includes": {
    "p/%hash%.json": {
      "sha256": "6cf2ecc55b6d6549f2a1d21a7c9abba59ebaf7502519863c4378d3804e57471d"
    }
  },
  "providers-url": "/api/v4/group/34/-/packages/composer/%package%$%hash%.json"
}

However, according to the renovate debug logs, https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer/packages.json is never even requested.

Renovate only attempts to get the <vendor>/<dependency> package from packagist. (HTTP 404 from https://packagist.org/p/<vendor>/<dependency>.json and no additional request.)

Correctness

<vendor>/<dependency> is found in https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer/p2/<vendor>/<dependency>.json (via https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer/packages.json as configured in composer.json "repositories").

Log

All redacted details are in <angle brackets>.

Somewhat redacted, matching the composer.json above.
Somewhat shortened.

DEBUG: Found composer package files (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Found 1 package file(s) (repository=clients/<customer>/extensions/<parent_package>)
 INFO: Dependency extraction complete (repository=clients/<customer>/extensions/<parent_package>)
       "baseBranch": "typo3-10",
       "stats": {
         "managers": {"composer": {"fileCount": 1, "depCount": 3}},
         "total": {"fileCount": 1, "depCount": 3}
       }
DEBUG: Datasource 404 (repository=clients/<customer>/extensions/<parent_package>)
       "datasource": "packagist",
       "lookupName": "<vendor>/<dependency>",
       "url": "https://packagist.org/p/<vendor>/<dependency>.json"
DEBUG: Failed to look up dependency <vendor>/<dependency> (repository=clients/<customer>/extensions/<parent_package>, packageFile=composer.json, dependency=<vendor>/<dependency>)
DEBUG: Package releases lookups complete (repository=clients/<customer>/extensions/<parent_package>)
       "baseBranch": "typo3-10"
DEBUG: branchifyUpgrades (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Using group branchName template (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Dependency typo3/cms-core is part of group TYPO3 CMS (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Using group branchName template (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Dependency typo3/cms-extbase is part of group TYPO3 CMS (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: 2 flattened updates found: typo3/cms-core, typo3/cms-extbase (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Returning 1 branch(es) (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Fetching changelog: https://github.com/TYPO3-CMS/extbase (10.4.24 -> 11.5.6) (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: Fetching changelog: https://github.com/TYPO3-CMS/core (10.4.24 -> 11.5.6) (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: config.repoIsOnboarded=false (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: packageFiles with updates (repository=clients/<customer>/extensions/<parent_package>)
       "config": {
         "composer": [
           {
             "packageFile": "composer.json",
             "deps": [
               {
                 "depType": "require",
                 "depName": "<vendor>/<dependency>",
                 "currentValue": "^10.4",
                 "datasource": "packagist",
                 "depIndex": 0,
                 "updates": [],
                 "warnings": [
                   {
                     "topic": "<vendor>/<dependency>",
                     "message": "Failed to look up dependency <vendor>/<dependency>"
                   }
                 ],
                 "versioning": "composer"
               },
               {
                 "depType": "require",
                 "depName": "typo3/cms-core",
                 "currentValue": "^10.4",
                 "datasource": "packagist",
                 "depIndex": 1,
                 "updates": [
                   {
                     "bucket": "major",
                     "newVersion": "11.5.6",
                     "newValue": "^11.0",
                     "releaseTimestamp": "2022-02-08T08:20:56.000Z",
                     "newMajor": 11,
                     "newMinor": 5,
                     "updateType": "major",
                     "isRange": true,
                     "branchName": "renovate/major-typo3-cms"
                   }
                 ],
                 "warnings": [],
                 "versioning": "composer",
                 "sourceUrl": "https://github.com/TYPO3-CMS/core",
                 "homepage": "https://typo3.org",
                 "currentVersion": "10.4.24",
                 "isSingleVersion": false
               },
               {
                 "depType": "require",
                 "depName": "typo3/cms-extbase",
                 "currentValue": "^10.4",
                 "datasource": "packagist",
                 "depIndex": 2,
                 "updates": [
                   {
                     "bucket": "major",
                     "newVersion": "11.5.6",
                     "newValue": "^11.0",
                     "releaseTimestamp": "2022-02-08T08:20:56.000Z",
                     "newMajor": 11,
                     "newMinor": 5,
                     "updateType": "major",
                     "isRange": true,
                     "branchName": "renovate/major-typo3-cms"
                   }
                 ],
                 "warnings": [],
                 "versioning": "composer",
                 "sourceUrl": "https://github.com/TYPO3-CMS/extbase",
                 "homepage": "https://typo3.org",
                 "currentVersion": "10.4.24",
                 "isSingleVersion": false
               }
             ],
             "registryUrls": [
               "https://gitlab.<vendor>.io/api/v4/group/34/-/packages/composer",
               "https://packagist.org"
             ],
             "managerData": {"composerJsonType": "typo3-cms-extension"}
           }
         ]
       }
DEBUG: ensureOnboardingPr() (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: getBranchPr(renovate/configure) (repository=clients/<customer>/extensions/<parent_package>)
DEBUG: findPr(renovate/configure, undefined, open) (repository=clients/<customer>/extensions/<parent_package>)

How can I help?

What can I do to help?

viceice · 2022-02-10T18:45:35Z

create a public reproduction on gitlab.com

viceice · 2022-02-10T18:46:19Z

or better create a public composer package on gitlab.com and a public consuming reproduction repo on github.com

LeoniePhiline · 2022-02-10T21:46:22Z

@viceice PR #14160 fixes this.

Problem was that platform token was not used for 'packagist' context requests.

viceice · 2022-02-11T07:18:48Z

It's currently intended to not use the platform token for packagist. You should be able to simply add an additional host rule for packagist.

LeoniePhiline · 2022-02-11T14:02:26Z

Part of the issue is that when authentication to one of the composer manager's repositoryUrls fails, the user is not informed; not even in debug mode. This code path misses all diagnostics output.

This means that there are two separate issues:

If a platform provides an "all in one" feature set, such as gitlab (repo, ci/cd, package registries for multiple languages/ecosystems, docker dependency proxy etc.) then users would (today) expect renovate to authenticate with the token they were instructed to configure as environment variable. (Paraphrasing: "You only need to configure RENOVATE_TOKEN and GITHUB_COM_TOKEN.")

The documentation could reflect (easily understandable and obvious to new users) that the same credentials must be configured multiple times because renovate lacks the logic to know how to handle the "all in one" platform. It's just a matter of perspective in the current situation: Users experience their platform as "one" while renovate handles each of the platform's services as if they were entirely unrelated to each other. Next to that, Renovate's output should make it obvious that authentication against the platform's package registry fails as long as no duplicate configuration was provided, and therefore the package registry URL is skipped.
The other part is refactoring renovate into a more modern idea of what a platform is today, given that these have changed and gotten far more feature rich over the years. This includes that in the generic and specific platform initialization code, each platform should know what it provides and configure renovate to correctly consume all of its services. (Currently the generic code adds configuration after the specific platform init code has returned. This architecture brings mess with it when platforms need to be configured for the specific services the platform itself know about. It might be helpful to move more knowledge into the platform-specific code.)

mrgesco · 2022-03-16T16:20:57Z

@herndlm After a while I finally had time to recheck this bug.
With the update, the fix for me was to add a hostRule with the token to access my private registry

module.exports = {
    hostRules: [
      {
        matchHost: 'https://<GITLAB_URL>/api/v4/group/<group-id>/',
        username: '___token___',
        password: process.env.MY_SUPER_SECRET_TOKEN,
      }
    ],
    autodiscover: false,
    dryRun: false,
};

mrgesco added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Jun 29, 2021

This comment has been minimized.

Sign in to view

HonkingGoose added the reproduction:provided label Jun 29, 2021

HonkingGoose added datasource:packagist manager:composer Composer (PHP) package manager platform:gitlab GitLab Platform labels Jun 29, 2021

herndlm mentioned this issue Dec 7, 2021

feat(composer): support providers without a hash #13000

Merged

6 tasks

This was referenced Feb 10, 2022

fix(gitlab): Use gitlab platform token to authenticate against gitlab composer package registry LeoniePhiline/renovate#1

Closed

fix(gitlab): Use gitlab platform token to authenticate against gitlab composer package registry #14160

Closed

mrgesco closed this as completed Mar 16, 2022

github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Self-hosted composer registry on self-hosted GitLab gives wrong package url #10649

Self-hosted composer registry on self-hosted GitLab gives wrong package url #10649

mrgesco commented Jun 29, 2021 •

edited

Loading

rarkins commented Jun 29, 2021

This comment has been minimized.

mrgesco commented Jun 29, 2021 •

edited

Loading

github-actions bot commented Jun 29, 2021

mrgesco commented Sep 6, 2021

HonkingGoose commented Sep 6, 2021

herndlm commented Dec 7, 2021 •

edited

Loading

herndlm commented Dec 8, 2021 •

edited

Loading

LeoniePhiline commented Feb 10, 2022 •

edited

Loading

viceice commented Feb 10, 2022

viceice commented Feb 10, 2022

LeoniePhiline commented Feb 10, 2022

viceice commented Feb 11, 2022

LeoniePhiline commented Feb 11, 2022 •

edited

Loading

mrgesco commented Mar 16, 2022 •

edited

Loading

Self-hosted composer registry on self-hosted GitLab gives wrong package url #10649

Self-hosted composer registry on self-hosted GitLab gives wrong package url #10649

Comments

mrgesco commented Jun 29, 2021 • edited Loading

rarkins commented Jun 29, 2021

This comment has been minimized.

mrgesco commented Jun 29, 2021 • edited Loading

github-actions bot commented Jun 29, 2021

mrgesco commented Sep 6, 2021

HonkingGoose commented Sep 6, 2021

herndlm commented Dec 7, 2021 • edited Loading

herndlm commented Dec 8, 2021 • edited Loading

LeoniePhiline commented Feb 10, 2022 • edited Loading

Context

Version

Configuration

composer.json

Private repository

Correctness

Log

How can I help?

viceice commented Feb 10, 2022

viceice commented Feb 10, 2022

LeoniePhiline commented Feb 10, 2022

viceice commented Feb 11, 2022

LeoniePhiline commented Feb 11, 2022 • edited Loading

mrgesco commented Mar 16, 2022 • edited Loading

mrgesco commented Jun 29, 2021 •

edited

Loading

mrgesco commented Jun 29, 2021 •

edited

Loading

herndlm commented Dec 7, 2021 •

edited

Loading

herndlm commented Dec 8, 2021 •

edited

Loading

LeoniePhiline commented Feb 10, 2022 •

edited

Loading

`composer.json`

LeoniePhiline commented Feb 11, 2022 •

edited

Loading

mrgesco commented Mar 16, 2022 •

edited

Loading