Allow autodiscover filtering for repo topic

[FalconerTC]

What would you like Renovate to be able to do?

This is just an idea I'd like to pose, for self-hosted configs. There are already a couple topics open for enhancing autodiscover filtering abilities, including allowing filtering by team. I think it would be useful to be able to filter by repo topic, defining something like

"autodiscoverTopics": ['managed-by-renovate'].

This would work similar to repositories but allowing repos more direct control. It also enables renovate to be toggled easily without removing configs. /user/repos already returns repo topics so it would be easy to filter after fetching the repo list.

I'm considering this in trying to figure out how to allow repositories to opt-in to renovate without config changes each time, and without having to check every repo for a config on each run (the org has 500+ repos)

[rarkins]

I'm not sure how many of the supported platforms have the concept of topics, but this sounds fine to me as a filtering criteria for autodiscover. Keep the name generic but be sure to make sure the documentation specifies the limits of which platforms are supported/unsupported.

[pmorch]

Since the merge of #13075 (in releases after 30.4.1), it is possible to do that sort of custom autodiscover-filtering yourself by using JavaScript to gather the list of repositories and then provide that list of repositories in the exported configuration like this:

const got = require('got')

const endpoint = 'https://gitlab.example.com/api/v4/'
const token = 'my-gitlab-token'

// Get repositories matching this topic:
const topic = "helm"

module.exports = async function () {
    // Need to handle paging when more than 100 repos...
    const projectsResult = await got(endpoint + '/projects', {
        searchParams: {
            'topic': topic,
            'per_page': 100,
        },
        headers: {
            'PRIVATE-TOKEN': token
        },
        responseType: 'json',
    });

    const repositories = projectsResult.body.map( r => r.path_with_namespace );
    // console.log("repositories", repositories);
    return {
        token,
        endpoint,
        platform: 'gitlab',
        onboarding: false,
        dryRun: true,
        repositories
    }
}

[dhduvall]

This is my take on @pmorch's suggestion above, grabbing renovate's own Gitlab functions to do the work. I'm sure that isn't even remotely supported, but for me it's worth the risk of the functions changing out from under me.

const token = process.env.RENOVATE_TOKEN; // CI_JOB_TOKEN doesn't get enough access
const topic = "redfish,bluefish";

async function getRepos() {
    let { logger } = await import("/usr/src/app/node_modules/renovate/dist/logger/index.js");
    let { gitlabApi } = await import("/usr/src/app/node_modules/renovate/dist/modules/platform/gitlab/http.js");

    logger.debug("Autodiscovering GitLab repositories");
    try {
        const url = "projects"
            + "?membership=true"
            + "&per_page=100"
            + "&with_merge_requests_enabled=true"
            + "&min_access_level=30"
            + "&archived=false"
            + "&simple=true"
            + `&topic=${topic}`;
        const res = await gitlabApi.getJson(url, { token, paginate: true });

        repositories = res.body
            .filter((repo) => !repo.mirror)
            .map((repo) => repo.path_with_namespace);

        logger.debug({repositories: repositories}, `Discovered ${res.body.length} project(s)`);

        return repositories;
    } catch (err) {
        logger.error({ err }, "GitLab getRepos error");
        throw err;
    }
}

module.exports = async function () {
    repositories = await getRepos();

    return {
        token,
        repositories,
        // ...
    }
}

[viceice]

your imports will soon fail on our official docker images

[rarkins]

PRs welcome for autodiscoverTopics

[dhduvall]

your imports will soon fail on our official docker images

Ooh, good to know, thanks. Is there anything you can tell me about that? It wasn't readily apparent to me what change would cause that to happen. Is there anything I can do to make it work again?

PRs welcome for autodiscoverTopics

I'll give it a shot. I'm not sure what to do about this error, though—I was getting it prior to making any changes, too:

❯ yarn build
yarn run v1.22.19
$ run-s clean generate:* compile:* create-json-schema
$ rimraf dist tmp
$ node tools/generate-imports.mjs
generating imports
> data/azure-pipelines-tasks.json
> data/debian-distro-info.json
> data/kubernetes-api.json5
> data/node-js-schedule.json
> data/ubuntu-distro-info.json
> node_modules/emojibase-data/en/shortcodes/github.json
generating hashes
$ tsc -p tsconfig.app.json
$ ts-node tools/generate-schema.ts && prettier --write --cache "renovate-schema.json"
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/duvall/src/renovate/tools/generate-schema.ts
    at new NodeError (node:internal/errors:405:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:99:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:142:36)
    at defaultLoad (node:internal/modules/esm/load:86:20)
    at nextLoad (node:internal/modules/esm/hooks:726:28)
    at load (/Users/duvall/src/renovate/node_modules/ts-node/dist/child/child-loader.js:19:122)
    at nextLoad (node:internal/modules/esm/hooks:726:28)
    at Hooks.load (node:internal/modules/esm/hooks:370:26)
    at MessagePort.handleMessage (node:internal/modules/esm/worker:168:24)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:762:20) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

[viceice]

https://github.com/renovatebot/docker-renovate/blob/0c6d504b754be81e6fa9697f53cd8a19ade0ff55/Dockerfile#L37

that line will be removed. it was never official supported to import renovate as a library. so you need to add your own local modules if the node buildins are not enough.

I've never seen that error before with ts-node. which node version are you using? there was another conttibuter who had same issue. 🤔

[dhduvall]

https://github.com/renovatebot/docker-renovate/blob/0c6d504b754be81e6fa9697f53cd8a19ade0ff55/Dockerfile#L37

that line will be removed. it was never official supported to import renovate as a library. so you need to add your own local modules if the node buildins are not enough.

Thanks.

I've never seen that error before with ts-node. which node version are you using? there was another conttibuter who had same issue. 🤔

Yeah, I saw that and commented there, too, but that was just yesterday.

❯ node --version
v20.3.0

This is on MacOS with node (and yarn) installed from homebrew. But I got the same thing on Linux, where both were installed from packages as instructed by your local dev docs (slightly modified—I followed the instructions the node install spat out to install yarn). Same versions of both.

[viceice]

so maybe some node v20 incompatibility? please try node v18 lts to verify

[dhduvall]

❯ node --version
v18.16.0

❯ yarn build
yarn run v1.22.19
$ run-s clean generate:* compile:* create-json-schema
$ rimraf dist tmp
$ node tools/generate-imports.mjs
generating imports
> data/azure-pipelines-tasks.json
> data/debian-distro-info.json
> data/kubernetes-api.json5
> data/node-js-schedule.json
> data/ubuntu-distro-info.json
> node_modules/emojibase-data/en/shortcodes/github.json
generating hashes
$ tsc -p tsconfig.app.json
$ ts-node tools/generate-schema.ts && prettier --write --cache "renovate-schema.json"
 INFO: Generating json-schema
renovate-schema.json 90ms
✨  Done in 15.72s.

Thank you!