Support for Gerrit

[derkoe]

What would you like Renovate to be able to do?
It would be great to have support for Gerrit based Git repositories including the Gerrit change workflow.

Describe the solution you'd like
Create a Gerrit change for each dependency update and also update this changes on every run. Hashtags may be used to label changes (like GitHub labels).

Describe alternatives you've considered
Gerrit provides a REST API which could be used to query/modify changes.

Additional context
Gerrit is still used in a lot of projects especially by Google (https://gerrit.googlesource.com/, https://fuchsia.googlesource.com/).

[rarkins]

Renovate uses branches extensively. Will this map to Gerrit's concepts?

[derkoe]

Yes - i'll map to a "change" in Gerrit. The only difference is that Gerrit only allows one commit in such a "change" so all changes after the first commit have to be amended and then pushed.

Maybe I'll find some time in the next weeks to provide a proof of concept implementation

[rarkins]

Renovate only every makes one commit off master and force pushes over the top if it needs to make a change. So far sounds compatible?

[derkoe]

Yes! Started with the implementation in my fork: https://github.com/derkoe/renovate/tree/feat/4109-gerrit

[ambles]

@derkoe how's your implementation so far? I wonder how does Google uses both Gerrit and Renovate together.

[derkoe]

Not very far (╯︵╰) - we are mostly using GitLab now, so I'm not sure if I'll ever finish that.

Does Google use Renovate with Gerrit? Where did you find that.

[rarkins]

I think Google's use of Renovate is limited to its open source repositories, where they use GitHub Issues

[tprochazka]

To create Gerrit PR it is only necessary to do simple push command
git.exe push origin refs/heads/master:refs/for/master
just instead of master there must by special reference refs/for/master

And you basically don't need to amend the last commit if you want just update the previous PR, you need just use the same Change-Id: I8deb93e6ffba6005234b0d46d5ae9f285f9775a8 inside of the commit message. So if you create a completely new commit with the same Change-id it will update the previous, of course only if it is still open.

[avm99963]

I myself use Gerrit and would like to join in the efforts to make renovate available for gerrit, so I'll fork @derkoe's branch and try to figure out how to continue implementing it (I'm new to renovate!).

If someone else is also interested in continuing @derkoe's work, please say so and we can coordinate!

[NiasSt90]

hi,
I've started a prototype implementation. The changes can be found here
it's still a proof-of-concept, therefore no tests and documentation...

the current state works (inclusive automerge/submit) with some drawbacks (see below).
You need some special configuration entries (explained and shown at the bottom).

Warning: I'm new to renovate and still study the documentation and code to understand all of the details. Perhaps i'm misunderstood some details/features. Then correct me plz.

Renovate-with-Gerrit Workflow

Here are my (current) translation of the renovate-branch/pull-request-model to the gerrit-change-model:

1. no branches, we map the branchname to a gerrit-topic or hashtag with:
   Config.js: branchPrefix: "master%topic=" to use of topics or branchPrefix: "master%t=" for hashtags ,
   that's the most important part, it allows to identify existing gerrit-changes based on the branchname generated from renovate like master%topic=testing-library-react-13.x.
   This way we don't need branches on gerrit-side but still has a unique name to find an existing change based only on the branchname.
2. install the gerrit-commit hook in initRepo
   to create a change-id if not already exists in commit-message.
3. allow to set a remotePushPrefix, called from (gerrit)initPlatform with 'refs/for/'
4. Config.js platformCommit: true to allow re-use of an existing change-id
   Renovate commitAndPush at first and then try to create a Pull-Request. For Gerrit the "Pull-Request" is a Change and was created automatically after pushing to special ref refs/for/$branch.
   Therefore the gerrit commitFiles function will allways try to search for an existing Change with the same branch AND topic/hashtag. If found it re-use the Change-Id in the commit message. This allows updating existing Gerrit-Changes.
5. createPr() always search for an existing Gerrit-Change with Branch-and-Topic/Hashtag
   Important: because of first-commitAndPush and later call to createPr Gerrit has already created a Change for this commit. The Renovate-Branchname looks like master%topic=react-dom-18.x and could be split into branch/topic.
6. for each pull-request/change we add the gerrit Code-Review label with +2 (if available) on our own to allow "automerge".
7. getBranchStatus(branchNameWithTopic) will check if all the changes found with branchname and topic (should only be one) are submittable, this includes validation of Verified labels from CI or similar gerrit-specific submit rules,
8. changed some refspecs in some git-commands to avoid syntax errors because of the generated branchname with %topic... inside.
   i.e. diff origin/$branchname changed to $branchname@{u}, i hope this has no effect on other platforms.

(current) Drawbacks

1. where to place the pull-request body and title?
   currently i'm add the body as a message and check all previous messages for "same-content". Delete a message is not an option (admin-priviledge). The pull-request title is currently unused.
2. no option to "ignore" a PullRequest (Gerrit-Change).
   For all non-automerge changes we need an option to allow "abandon" a PR. But a user/dev can't abandon a change from other users. Perhaps the user can down-votes (-2?) and later renovate calls "abandon"?

Unfinished stuff

• gerrit (markdown?) syntax for pull-request body?
• Platform interface functions with (for me) unknown semantics like:
  getBranchStatusCheck/setBranchStatus, getRawFile, getJsonFile, deleteLabel, ensureCommentRemoval, getIssueList

Needed Renovate Config.js

I'm use the Gerrit-HTTP REST API with http-basic-auth (Prefix /a/). Therefore you need to generate a custom password for the gerrit-enovate-account.

These are the important part:

module.exports = {
  platform: 'gerrit',
  endpoint: '$GERRIT_HTTP_URL',
  username: "$USER",
  password: "$HTTP-PASSWORD",
  
  platformCommit: true, //needed to allow commit with an existing change-id
  branchPrefix: "master%topic=", //Important: don't use branches, we use Gerrit-Topics or HashTags  with %t=
  onboardingBranch: "master%topic=renovate-config", //Important: don't use branches, we use Gerrit-Topics
  gitNoVerify: ["push"], //Important: allow commit-hook

};

[NiasSt90]

New revision (amend-commit) uploaded.

The (major) drawback,
that for each renovate call a new and empty Gerrit-PatchSet was created is gone.

The (fake) branch-handling was improved, for each open gerrit-change a local branch was created (from initRepo) and registered with the commitSha.

All open gerrit-changes are automatically "updated" if origin/$basebranch was updated.
This was done thru a special platform.commitFiles function.

[NiasSt90]

New revision (amend-commit) uploaded.

• assignees and reviewers Support added
• added Gerrit Platform documentation

In my opinion the proof of concept is done.
After my vacation i would start to write the tests.

Any comments?

[rarkins]

Hi @NiasSt90, thanks for keeping us updated with your progress. I have never used Gerrit so have to admit some of it is a little 🤯 for me. Should we try to address the drawbacks first, e.g. things which might make this unworkable? Can you describe them in a bit more detail (not just what you tried, but what the high level requirements are) so that we can try to work out a solution? Also feel free to email me for an invite into our contributor-only Slack group

[Shegox]

@NiasSt90 I'm actually having some capacity as well and were wondering if you like support with implementing Gerrit support? Anything we can try/test/implement to help you?

[mf75]

I don't know, whether nias (a friend of mine) will be able to answer himself, but let me just inform you, that he finished his vacation with some accident, he still struggles with.
He will be all right and back on the keys soon.
Stay tuned and wish him all the best.

[leoluk]

My company might be willing to fund this feature. Reach out to me if interested.

[PhilipAbed]

info about: getBranchStatusCheck/setBranchStatus, getRawFile, getJsonFile, deleteLabel, ensureCommentRemoval, getIssueList

branch status checks: are the status of the build+test+checks that you see in github, on every pull request in renovate,
every platform has some sort of checks made before approving the pull request, use the platform API to get those.
basic use is auto-merge ( some other side functionality), but the normal flows usually work without needing this.

getRawFile/getJsonFile: are a way to get the files from the repository, usually GetJsonFile files calls GetRawFile data, and converts it to json.

deleteLabel: is used for rebaseLabel , so we can rebase PRs based on a specific label being set.

ensureCommentRemoval: will have to call the repository's pull request/comment and make sure it is removed ( if necessary, these flows require specific comments to be made beforehand, like auto-merge comment)

getIssueList: is a way to get all issues from a specific repo, check the code see what it does exactly, could be open issues only in order to find the Dependency Dashboard, or Open a new Dependency Dashboard, read about it.

for any help needed, please ask.

Hope nias gets better.

[sselberg]

First and foremost, you should advertise this effort in the Gerrit community, I'm sure you will get a lot of valid input from there.
Some nits from just reading through this thread.

• I would advice against using topics, I think hashtags are more useful it you want to add metadata to the change you are creating to be able to query for it in Gerrit.
• branchPrefix: "master%topic=" this indicates to me that you limit renovate to only work against the master branch (if this results in git push origin HEAD:refs/for/master%topic=$ID
• Using Create Change and the Change Edit endpoints in the REST API you don't need a local git copy to update dependencies.

[magnusbaeck]

One problem with the use of topics is that a change can have just a single topic, and users may want to use topics for other purposes (e.g. atomic cross-repo commits with the Submit Whole Topic feature).

[NiasSt90]

hello,
i'm back at work (for quite some time) and now i've restarted to work at this task.
I've updated the above comment and branch with some new/changed features (hashtags).

First and foremost, you should advertise this effort in the Gerrit community, I'm sure you will get a lot of valid input from there. Some nits from just reading through this thread.

okay, i would do this soon.

• I would advice against using topics, I think hashtags are more useful it you want to add metadata to the change you are creating to be able to query for it in Gerrit.

okay, i've modified my prototype to allow both, topics or hashtags (with %topic= or %t=).
It works with hashtags too, but there was a small difference/problem. There could be more than one hashtag for each change. Currently i'm use the "first" hashtag to identify an existing renovate pull-request.

• branchPrefix: "master%topic=" this indicates to me that you limit renovate to only work against the master branch (if this results in git push origin HEAD:refs/for/master%topic=$ID

yes, only a single branch is currently possible. I'm still searching for an idea how to change this.

Renovate use a local git repo for all platforms in the same way (hard coded git-commands), a specific platform can't break out of this workflow (currently).
It creates branches and commits locally and push them afterwards.
The pull-request will be created after the created branches were pushed.
But for Gerrit the creation of a pull-request (change) is done automatically by the push refs/for/$branch...

• Using Create Change and the Change Edit endpoints in the REST API you don't need a local git copy to update dependencies.

that is not an option (or useful).
Renovate needs a local copy of the repo to analyze it for possible dependency upgrades (that's the main work of renovate).
The platform specific implementation part (see interface Platform in platform/types.ts) is rather small and there is much work with hard coded git-commands equal to all platforms.

[NiasSt90]

New revision (amend-commit) uploaded.

It's now possible to ignore pull-requests by vote them with -2 (reject).
If a change was rejected by a user then renovate will "abandon" this change on the next run.

[anicke]

Nice, I have tested the WIP and created a bunch of Gerrit reviews with updates and it works like a charm! I had to do some smaller things like wrap the project query parameter in a "encodeURIComponent" to get it working here (my project contains "/" chars). My fork can be found here. I also added some tests that you can copy.

[NiasSt90]

IIUC the model of:

1. push to ref
2. REST call
   And the creation of the extra "refs/renovate/..." ref, Is only necessary because there's a need to mimic the GitHub (et.al.) integration if you don't want to refactor Renovate.

If you can make renovate push to any ref, why you from just pushing to refs/for/refs/heads/$target and skip the REST call? That should be all you need to use Gerrit as it was intended and the end-result should be the same (bar the unnecessary "refs/renovate/..." refs).

sadly no, at the time of the commitAndPush the target-branch (i.e. baseBranch) is not known
(or better: not passed down to the platform.commitAndPush).

I will do some more investigations into this, perhaps there is a (clean) way to provide these information to the platform.commitAndPush.
Perhap i'm simply add the "baseBranch" to CommitFilesConfig.

[NiasSt90]

New Prototyp V3

You can find these Gerrit-V3 Branch here.

It's based on V1 and don't need special permission (except to allow +2 Code-Review label) for some refs.
It works with multiple baseBranches and don't need a special branchPrefix.
To get this to work i've added the baseBranch as parameter to commitAndPush(). I think that should be no problem, the information is always there (for each invocation). I've modified the utils/git-push method a little bit to allow to push custom-src/dst-refs.

This way it's possible to build a gerrit-push url like:

refs/for/${config.baseBranch}%t=sourceBranch-${config.branchName}

The original branchName was stored into an Gerrit hashtag (with prefix) to allow find existing PRs.

As in V1 from initRepo() all existing (renovate-) PRs were checked out as local branches (additional with origin/ prefix).

This way most util/git/* commands are work as expected or in detail:

• git.isBranchModified()
  We check if the last uploader was different to the renovate user and store this into "branchIsModified[branchname]".
  This is currently the only way to allow a user to change a renovate-PR.
  Otherwise, renovate would silently overwrite the user-changes.

• git.isBranchConflicted()
  This on tries to merge "origin/${branchName}" into baseBranch and check for conflicts.
  From initRepo() we fetched all open gerrit-PRs to the local branch-name "origin/${branchName}" to let this work as expected.

• git.isBranchBehindBase()
  Always returns true, because it currently use the "--remotes" option.
  No workaround for Gerrit currently possible. But should be no big problem or?

• git.deleteBranch()
  Fails always because the branches are not exists in real. This should be no real problem.

[sselberg]

Awesome!

[sselberg]

Will config.branchName always contain "renovate"?
It might be nice to explicitly see who created the HashTag if it doesn't. i,e "%t=Renovate-sourceBranch-$(config.branchName)

p.s. IIRC there are extension-points for hashtag creation validation so that once a set hash-tag pattern is established it wouldn't be to difficult to write a plugin that validates hashtag creation so that only the renovate-user (and f.i. admins) have the right to create hashtags with a certain pattern (f.i. "sourceBranch-renovate/.*").

[NiasSt90]

Will config.branchName always contain "renovate"? It might be nice to explicitly see who created the HashTag if it doesn't. i,e "%t=Renovate-sourceBranch-$(config.branchName)

That depends on the branchName configuration.
The default starts with "renovate/" ("branchPrefix").
But every user can change this like he wants. But in my opinion the defaults are good enough for Gerrit users.

p.s. IIRC there are extension-points for hashtag creation validation so that once a set hash-tag pattern is established it wouldn't be to difficult to write a plugin that validates hashtag creation so that only the renovate-user (and f.i. admins) have the right to create hashtags with a certain pattern (f.i. "sourceBranch-renovate/.*").

okay, to avoid/prevent potential abuse?

I hope the maximum length for a Gerrit hashtag is not that short, i couldn't found this in the documentation so far.

[NiasSt90]

V3 Patchset Update

whats new/changed:

• isBranchBehindBase changed to work with Gerrits "local" fake-branches (origin/$branchname)
  this reduce the number of update requests and shouldReuseExistingBranch works correct now.
  Further improvements possible with a working canSkipBranchUpdateCheck(), i'm currently unsure why this not works (always false cause of missing fingerprint property).
• prTitle (changes) will be stored to the commit-msg (Gerrit change.subject property)
• gerritAutoApprove config introduced (defaults to true) to enable/disable +2 Code-Review label
• pr.bodyStruct.hash now filled with correct hash from current prBody (further reduce noise/update calls)

[sselberg]

Will config.branchName always contain "renovate"? It might be nice to explicitly see who created the HashTag if it doesn't. i,e "%t=Renovate-sourceBranch-$(config.branchName)

That depends on the branchName configuration. The default starts with "renovate/" ("branchPrefix"). But every user can change this like he wants. But in my opinion the defaults are good enough for Gerrit users.

p.s. IIRC there are extension-points for hashtag creation validation so that once a set hash-tag pattern is established it wouldn't be to difficult to write a plugin that validates hashtag creation so that only the renovate-user (and f.i. admins) have the right to create hashtags with a certain pattern (f.i. "sourceBranch-renovate/.*").

okay, to avoid/prevent potential abuse?

Yes, not related to this effort in any other way than that it shouldn't be difficult to write such a Gerrit plugin, for any Renovate users that doesn't want to consider the complexity of anyone removing or adding "Renovate-hashtags" and I'd be happy to do it.

I hope the maximum length for a Gerrit hashtag is not that short, i couldn't found this in the documentation so far.

There should be a technical limit, but in any case it's above 400 chars (empirically tested) so I think it will suffice.

[NiasSt90]

V3 Patchset Updated

the branch/diff can be found here

• rebased on current main
• some small fixes (i.e. a gerrit-url containing a pathname should work now)
• some refactorings and more unit tests

I think we're on the home straight.

I've updated the lib/modules/platform/gerrit/index.md with some details/TODOs.

[NiasSt90]

V3 Patchset updated again and now it has reached the test-coverage requirements.

[NiasSt90]

Gerrit 3.7 brings better (but not complete) markdown support.
It still lacks of image and <details> syntax.

[NiasSt90]

V3 Patchset updated again.

I've changed the "preset" loading feature a little bit.
Now you can use:

extends: [ "local>All-Projects:renovate#HEAD", ]

to extend the configuration with the file content renovate.json from the All-Projects HEAD branch.

Important: grant read access for the renovate account to the project/refspec (above "refs/meta/config").

I'm still unsure how to "fix" or implement the lib/workers/repository/onboarding/branch/config.ts#getOnboardingConfig behavior.
Splitting the config.repository with "/" looks like a bug to me.
What if the repo-name contains a slash?

The translation of the different file lookups to a extends: [orgPreset] config entry is still unclear to me.
I don't know how to get a useful behavior from these implementation with a Gerrit server.

[NiasSt90]

V3 Patchset rebased and updated again

gerritAutoApprove (i.e. add Code-Review=+2 label to each created change) is now an Opt-In feature.
You have to enable it in your config.

I've now done some first tests (two cloned but real repos) in our company infrastructure with success.
I think i will create the pull-request soon.

[lucdiniz]

V3 Patchset rebased and updated again

gerritAutoApprove (i.e. add Code-Review=+2 label to each created change) is now an Opt-In feature. You have to enable it in your config.

I've now done some first tests (two cloned but real repos) in our company infrastructure with success. I think i will create the pull-request soon.

Hello @NiasSt90,

First of all thanks for creating this integration with Gerrit.

I have a question, all my topic branch names are created like this "sourceBranch-renovate/all-minor-patch", there is any config to have a this String "sourceBranch", configured by repo?

I ask this because in our gerrit behaviour all reviews opened with same topic name will be submited together even in different repos hosted in same Gerrit instance.

Thanks

[NiasSt90]

V3 Patchset rebased and updated again

gerritAutoApprove (i.e. add Code-Review=+2 label to each created change) is now an Opt-In feature. You have to enable it in your config.
I've now done some first tests (two cloned but real repos) in our company infrastructure with success. I think i will create the pull-request soon.

Hello @NiasSt90,

First of all thanks for creating this integration with Gerrit.

I have a question, all my topic branch names are created like this "sourceBranch-renovate/all-minor-patch", there is any config to have a this String "sourceBranch", configured by repo?

I ask this because in our gerrit behaviour all reviews opened with same topic name will be submited together even in different repos hosted in same Gerrit instance.

Looks like you are trying to use an really old prototype.

Exactly for this reason I no longer use the Gerrit "topic" property to store the (virtual) branch name of the pull request.
For this I now use the "hashtags" property (array) with the prefix "sourceBranch-".

The branch with the latest prototype can be found here.
Warning: i'm force pushing to this branch!

[josesa-xx]

The problem we see in our gerrit service is that hashtag functionality is disabled.

I get greeted from gerrit push for review with this lovelly message:
! [remote rejected] HEAD -> refs/for/master%t=tagstest (cannot add hashtags; noteDb is disabled)

Explanation from service owners is that they are currently using "reviewdb" that doesn't support hashtags.

Still using topic with static name for all reviews is not good, as all reviews with same topic name are grouped and can only be submitted together :(

Is there any way to disable usage of topic name altogether ?
or use a topic name derived from repo name?

[magnusbaeck]

Explanation from service owners is that they are currently using "reviewdb" that doesn't support hashtags.

Support for reviewdb was dropped in Gerrit 3.0, released in May 2019 and long EOL.

[sselberg]

The problem we see in our gerrit service is that hashtag functionality is disabled.

I get greeted from gerrit push for review with this lovelly message: ! [remote rejected] HEAD -> refs/for/master%t=tagstest (cannot add hashtags; noteDb is disabled)

Explanation from service owners is that they are currently using "reviewdb" that doesn't support hashtags.

Ouch, this means that you are running a Gerrit version that is 4 1/2 years old and was EOL 2 years ago (2.16).

All your Gerrit admin needs to do is to upgrade to the latest v2.16 version and migrate to NoteDB:
https://www.gerritcodereview.com/2.16.html#migration-to-notedb
https://gerrit-documentation.storage.googleapis.com/Documentation/2.16/note-db.html#migration
This is very well tested and shouldn't present you with any problems, if you encounter any issues reach out to the Gerrit community and we will help out:
https://groups.google.com/g/repo-discuss
https://discord.com/channels/775374026587373568/1019941974842806302

What they really should do is upgrade to an actively supported version such as 3.7, but that's a separate issue.

[josesa-xx]

Explanation from service owners is that they are currently using "reviewdb" that doesn't support hashtags.

Support for reviewdb was dropped in Gerrit 3.0, released in May 2019 and long EOL.

Yes unfortunately we are stuck until IT service upgrades service to 3.x (which they aren't updating claiming performance impacts)

[viceice]

Explanation from service owners is that they are currently using "reviewdb" that doesn't support hashtags.

Support for reviewdb was dropped in Gerrit 3.0, released in May 2019 and long EOL.

Yes unfortunately we are stuck until IT service upgrades service to 3.x (which they aren't updating claiming performance impacts)

you should change your it service 😏

[felipecrs]

We could discuss this all day long... It's also my case with Gerrit 2.14. And there's nothing I can do to change it. :)

It's actually comforting to hear that I'm not the only one facing the same situation.

Nevertheless, I don't think the implementors of Gerrit in Renovate should lift a finger to support EOL versions of Gerrit either.

[sselberg]

Yes unfortunately we are stuck until IT service upgrades service to 3.x (which they aren't updating claiming performance impacts)

They don't have to upgrade to 3.x to get hashtags support, notedb is supported in 2.16 (see my post above).

If anything you get better performance with NoteDB, especially once they upgrade to a newer Gerrit version.

[josesa-xx]

Yes unfortunately we are stuck until IT service upgrades service to 3.x (which they aren't updating claiming performance impacts)

They don't have to upgrade to 3.x to get hashtags support, notedb is supported in 2.16 (see my post above).

If anything you get better performance with NoteDB, especially once they upgrade to a newer Gerrit version.

Thanks for the info, I'll try to reason with them and get NoteDB added, as we currently are using 2.16 based version.

[renovate-release]

🎉 This issue has been resolved in version 37.112.0 🎉

The release is available on:

• GitHub release
• 37.112.0

Your semantic-release bot 📦🚀