Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge and resolve conflicts #1

79 changes: 70 additions & 9 deletions docs/usage/private-modules.md
Original file line number Diff line number Diff line change
@@ -38,7 +38,7 @@ Although npm presets were the first type supported, they are now deprecated and
However if you do still use them, private modules should work if you configure the `npmrc` file including token credentials in your bot admin config.
Credentials stored on disk (e.g. in `~/.npmrc`) are no longer supported.

The recommended way of using local presets is to configure then using "local" presets, e.g. like `"extends": ["local>myorg/renovate-config"]`, and ensure that the platform token has access to that repo.
The recommended way of using local presets is to configure then using "local" presets, e.g. `"extends": ["local>myorg/renovate-config"]`, and ensure that the platform token has access to that repo.

It's not recommended that you use a private repository to host your config while then extending it from a public repository.
If your preset doesn't contain secrets then you should make it public, while if it does contain secrets then it's better to split your preset between a public one which all repos extend, and a private one with secrets which only other private repos extend.
@@ -52,7 +52,7 @@ In summary, the recommended approach to private presets is:
## Dependency Version Lookups

Whenever Renovate detects that a project uses a particular dependency, it attempts to look up that dependency to see if any new versions exist.
If such a package is private, then Renovate will need to be configured with the relevant credentials.
If such a package is private, then Renovate must be configured with the relevant credentials.
Renovate does not use any package managers for this step and performs all HTTP(S) lookups itself, including insertion of authentication headers.

Configuring Renovate with credentials requires `hostRules`.
@@ -64,7 +64,7 @@ It can be either a "platform" name (e.g. `github`, `azure`, etc) or a "datasourc
`baseUrl` can be used if you want to only apply the credentials for a nested path within the host, e.g. `https://registry.company.com/nested/path/`.
If the same credentials apply to all paths on a host, then use `hostName` instead, e.g. `registry.company.com`.
Finally, to apply credentials to all hosts within the domain, use `domainName`, e.g. `company.com`.
You need to pick only one of these and not try to use multiple at the same time, or it will be a config error.
You need to pick _only one_ of these and not configure more than one of these fields within the same host rule, otherwise it will error.

In addition to the above options to match against a host, you need to add the credentials.
Typically they are either `token`, or `username` + `password`.
@@ -91,7 +91,7 @@ Here is an example of some host rules:
Renovate applies theses `hostRules` to every HTTP(s) request which is sent, so they are largely independent of any platform or datasource logic.
With `hostRules` in place, private package lookups should all work.

## Release Notes
## Looking up Release Notes

When Renovate creates Pull Requests, its default behavior is to locate and embed release notes/changelogs of packages.
These release notes are fetched from the source repository of packages and not from the registries themselves, so if they are private then they will require different credentials.
@@ -102,7 +102,7 @@ It can be confusing for people who host their own source code privately to be as

Currently the preferred way to configure `github.com` credentials for self-hosted Renovate is:

- Create a read-only Personal Access Token (PAT) for a `github.com` account. It can be any account, and may even be best to be an empty account created just for this purpose.
- Create a read-only Personal Access Token (PAT) for a `github.com` account. This can be any GitHub account, it might be better to create a "empty" account just for this purpose.
- Add the PAT to Renovate using the environment variable `GITHUB_COM_TOKEN`

## Package Manager Credentials for Artifact Updating
@@ -147,7 +147,68 @@ For those found, a command similar to the following is run: `dotnet nuget add so

For every poetry source, a `hostRules` search is done and then any found credentials are added to env like `POETRY_HTTP_BASIC_X_USERNAME` and `POETRY_HTTP_BASIC_X_PASSWORD`.

<!-- TODO:
* Describe admin vs repo config of hostRules
* App details: no public->private presets lookup, encrypted
-->
## WhiteSource Renovate Hosted App Encryption

The popular [Renovate App on GitHub](https://github.com/apps/renovate) is hosted by WhiteSource.
If you are a user of this app, and have private modules, then the following is applicable.

### Private presets with public repositories

If you have a preset in a private repo but reference ("extend") it from a public repository then it won't work.
This is because public repositories are provided with a token scoped to only that particular repository, and not for all repositories within the organization.
This is a security measure so that if a the token is accidentally leaked publicly, the damage is limited to the public repository it leaked to and not to every repository within the organization.

The solution to this is that you should break your presets into public and private ones, and reference only the public ones from public repositories.

### Encrypting secrets

It is strongly recommended that you don't commit secrets to repositories, including private ones, and this includes secrets needed by Renovate to access private modules.
Therefore the preferred approach to secrets is that the bot administrator configures them as `hostRules` which are then applied to all repositories which the bot accesses.

If you need to provide credentials to the hosted Renovate App, please do this:

- Encrypt each secret string using <https://app.renovatebot.com/encrypt>. Note: this encrypts using the app's public key fully in the browser and does not send the original secret to any server. You can download this file and perform the encryption fully offline if you like.
- Wrap each secret field in an [encrypted](https://docs.renovatebot.com/configuration-options/#encrypted) object and paste in the encrypted secret value instead. An example is shown below:

```json
{
"hostRules": [
{
"hostName": "registry.npmjs.org",
"encrypted": {
"token": "3f832f2983yf89hsd98ahadsjfasdfjaslf............"
}
},
{
"baseUrl": "https://custom.registry.company.com/pypi/",
"username": "bot1",
"encrypted": {
"password": "p278djfdsi9832jnfdshufwji2r389fdskj........."
}
}
]
}
```

### Access to GitHub Actions Secrets

The WhiteSource Renovate App does not run using GitHub Actions, but such secrets would be a bad fit for the app anyway for the following reasons:

- The app would be granted access to _all_ the repository/org secrets, not just the ones you want
- If Renovate wants access to such secrets, it would need to ask for them from every user, not just the ones who want to use this approach (GitHub does not support the concept of optional permissions for Apps, so people do not have the option to decline)

## Admin/Bot config vs User/Repository config for Self-hosted users

"AdminBot config" refers to the config which the Renovate Bot administrator provides at bot startup, e.g. using environment variables, CLI parameters, or the `config.js` configuration file.
User/Repository config refers to the in-repository config file which defaults to `renovate.json` but has a large number of alternative filenames supported.

If there is a need to supply custom rules for certain repository, it can still be done using the `config.js` file and the `repositories` array.

If per-repository config must be done within the repository, it is still recommended against committing secrets directly (including e.g. `.npmrc` files with tokens) and instead encrypting them with a custom public key first.
For instructions on this, see the above section on encrypting secrets for the WhiteSource Renovate App but instead:

- Save a copy of the <https://app.renovatebot.com/encrypt> HTML file locally, or host it locally
- Generate a public/private key pair for the app using the instructions in [privateKey](https://docs.renovatebot.com/self-hosted-configuration/#privatekey)
- Replace the existing public key in the HTML with the public key you generated in the step prior
- Use the resulting HTML encrypt page to encrypt secrets for your app before adding them to user/repository config
- Configure the app to run with `privateKey` set to the private key you generated above
1 change: 1 addition & 0 deletions lib/config/presets/internal/monorepo.ts
Original file line number Diff line number Diff line change
@@ -119,6 +119,7 @@ const repoGroups = {
nuxtjs: 'https://github.com/nuxt/nuxt.js',
openfeign: 'https://github.com/OpenFeign/feign',
opentelemetry: 'https://github.com/open-telemetry/opentelemetry-js',
OpenTelemetryDotnet: 'https://github.com/open-telemetry/opentelemetry-dotnet',
picasso: 'https://github.com/qlik-oss/picasso.js',
pnpjs: 'https://github.com/pnp/pnpjs',
playwright: 'https://github.com/Microsoft/playwright',
2 changes: 1 addition & 1 deletion lib/datasource/maven/index.ts
Original file line number Diff line number Diff line change
@@ -8,8 +8,8 @@ import * as mavenVersioning from '../../versioning/maven';
import { compare } from '../../versioning/maven/compare';
import type { GetReleasesConfig, Release, ReleaseResult } from '../types';
import { MAVEN_REPO } from './common';
import type { MavenDependency } from './types';
import {
MavenDependency,
downloadMavenXml,
getDependencyInfo,
getDependencyParts,
13 changes: 13 additions & 0 deletions lib/datasource/maven/types.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import type { XmlDocument } from 'xmldoc';

export interface MavenDependency {
display: string;
group?: string;
name?: string;
dependencyUrl: string;
}

export interface MavenXml {
authorization?: boolean;
xml?: XmlDocument;
}
27 changes: 8 additions & 19 deletions lib/datasource/maven/util.ts
Original file line number Diff line number Diff line change
@@ -5,9 +5,10 @@ import { HOST_DISABLED } from '../../constants/error-messages';
import { logger } from '../../logger';
import { ExternalHostError } from '../../types/errors/external-host-error';
import { Http, HttpResponse } from '../../util/http';
import type { ReleaseResult } from '../types';

import type { ReleaseResult } from '../types';
import { MAVEN_REPO, id } from './common';
import type { MavenDependency, MavenXml } from './types';

const http: Record<string, Http> = {};

@@ -135,16 +136,12 @@ function containsPlaceholder(str: string): boolean {
return /\${.*?}/g.test(str);
}

export interface MavenDependency {
display: string;
group?: string;
name?: string;
dependencyUrl: string;
}

interface MavenXml {
authorization?: boolean;
xml?: XmlDocument;
export function getMavenUrl(
dependency: MavenDependency,
repoUrl: string,
path: string
): url.URL | null {
return new url.URL(`${dependency.dependencyUrl}/${path}`, repoUrl);
}

export async function downloadMavenXml(
@@ -182,14 +179,6 @@ export async function downloadMavenXml(
return { authorization, xml: new XmlDocument(rawContent) };
}

export function getMavenUrl(
dependency: MavenDependency,
repoUrl: string,
path: string
): url.URL | null {
return new url.URL(`${dependency.dependencyUrl}/${path}`, repoUrl);
}

export function getDependencyParts(lookupName: string): MavenDependency {
const [group, name] = lookupName.split(':');
const dependencyUrl = `${group.replace(/\./g, '/')}/${name}`;
7 changes: 2 additions & 5 deletions lib/datasource/sbt-package/index.ts
Original file line number Diff line number Diff line change
@@ -2,11 +2,8 @@ import { logger } from '../../logger';
import * as ivyVersioning from '../../versioning/ivy';
import { compare } from '../../versioning/maven/compare';
import { MAVEN_REPO } from '../maven/common';
import {
MavenDependency,
downloadHttpProtocol,
getDependencyInfo,
} from '../maven/util';
import type { MavenDependency } from '../maven/types';
import { downloadHttpProtocol, getDependencyInfo } from '../maven/util';
import { parseIndexDir } from '../sbt-plugin/util';
import type { GetReleasesConfig, ReleaseResult } from '../types';

8 changes: 7 additions & 1 deletion lib/util/git/index.ts
Original file line number Diff line number Diff line change
@@ -450,13 +450,19 @@ export function getBranchList(): string[] {
export async function isBranchStale(branchName: string): Promise<boolean> {
await syncBranch(branchName);
try {
const { currentBranchSha, currentBranch } = config;
const branches = await git.branch([
'--remotes',
'--verbose',
'--contains',
config.currentBranchSha,
]);
return !branches.all.map(localName).includes(branchName);
const isStale = !branches.all.map(localName).includes(branchName);
logger.debug(
{ isStale, branches, currentBranch, currentBranchSha },
`IsBranchStale=${isStale}`
);
return isStale;
} catch (err) /* istanbul ignore next */ {
checkForPlatformFailure(err);
throw err;
3 changes: 2 additions & 1 deletion lib/workers/branch/automerge.ts
Original file line number Diff line number Diff line change
@@ -11,6 +11,7 @@ export type AutomergeResult =
| 'branch status error'
| 'failed'
| 'no automerge'
| 'stale'
| 'not ready';

export async function tryBranchAutomerge(
@@ -48,7 +49,7 @@ export async function tryBranchAutomerge(
err.message.includes('Not possible to fast-forward')
) {
logger.warn({ err }, 'Branch is not up to date - cannot automerge');
return 'not ready';
return 'stale';
}
if (err.message.includes('Protected branch')) {
if (err.message.includes('status check')) {
25 changes: 25 additions & 0 deletions lib/workers/branch/index.spec.ts
Original file line number Diff line number Diff line change
@@ -448,6 +448,31 @@ describe(getName(__filename), () => {
expect(platform.ensureCommentRemoval).toHaveBeenCalledTimes(0);
expect(prAutomerge.checkAutoMerge).toHaveBeenCalledTimes(1);
});
it('ensures PR when impossible to automerge', async () => {
getUpdated.getUpdatedPackageFiles.mockResolvedValueOnce({
updatedPackageFiles: [{}],
} as PackageFilesResult);
npmPostExtract.getAdditionalFiles.mockResolvedValueOnce({
artifactErrors: [],
updatedArtifacts: [{}],
} as WriteExistingFilesResult);
git.branchExists.mockReturnValue(true);
automerge.tryBranchAutomerge.mockResolvedValueOnce('stale');
prWorker.ensurePr.mockResolvedValueOnce({
prResult: PrResult.Created,
pr: {},
} as EnsurePrResult);
prAutomerge.checkAutoMerge.mockResolvedValueOnce({ automerged: false });
commit.commitFilesToBranch.mockResolvedValueOnce(null);
await branchWorker.processBranch({
...config,
automerge: true,
rebaseWhen: 'conflicted',
});
expect(prWorker.ensurePr).toHaveBeenCalledTimes(1);
expect(platform.ensureCommentRemoval).toHaveBeenCalledTimes(0);
expect(prAutomerge.checkAutoMerge).toHaveBeenCalledTimes(1);
});
it('ensures PR and adds lock file error comment if no releaseTimestamp', async () => {
getUpdated.getUpdatedPackageFiles.mockResolvedValueOnce({
updatedPackageFiles: [{}],
10 changes: 10 additions & 0 deletions lib/workers/branch/index.ts
Original file line number Diff line number Diff line change
@@ -380,6 +380,16 @@ export async function processBranch(
logger.debug('Branch is automerged - returning');
return { branchExists: false, result: BranchResult.Automerged };
}
if (
mergeStatus === 'stale' &&
['conflicted', 'never'].includes(config.rebaseWhen)
) {
logger.warn(
'Branch cannot automerge because it is stale and rebaseWhen setting disallows rebasing - raising a PR instead'
);
config.forcePr = true;
config.branchAutomergeFailureMessage = mergeStatus;
}
if (
mergeStatus === 'automerge aborted - PR exists' ||
mergeStatus === 'branch status error' ||
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -212,7 +212,7 @@
"@types/js-yaml": "3.12.6",
"@types/json-dup-key-validator": "1.0.0",
"@types/linkify-markdown": "1.0.0",
"@types/luxon": "1.26.3",
"@types/luxon": "1.26.4",
"@types/markdown-it": "12.0.1",
"@types/markdown-table": "2.0.0",
"@types/moo": "0.5.4",
8 changes: 4 additions & 4 deletions yarn.lock
Original file line number Diff line number Diff line change
@@ -1525,10 +1525,10 @@
resolved "https://registry.yarnpkg.com/@types/linkify-markdown/-/linkify-markdown-1.0.0.tgz#50c357a0a60bb220209f19310a79b939e1b8e2a3"
integrity sha512-p3rl2HtugwjkH8qjEjUi5JByoZkVc2lz6E7D96X5/FqNyj2/jlgIrMlyrbck0U+HyjhOIy0XyTwES3RRz+1koQ==

"@types/[email protected].3":
version "1.26.3"
resolved "https://registry.yarnpkg.com/@types/luxon/-/luxon-1.26.3.tgz#3d548f5f6f59723c292a8545da97631a85738b95"
integrity sha512-2TELN+Pd3Ocde87sKJMSQ9Wdj0zc/okHK3/+fOQHr3CaWv4jtVtcMzmt1Foww1+5YvPd9B5vL3XR6u5KF0daEA==
"@types/[email protected].4":
version "1.26.4"
resolved "https://registry.yarnpkg.com/@types/luxon/-/luxon-1.26.4.tgz#77bbd5a1aa74ca31edff2aee4f8313ea0d93ca5c"
integrity sha512-OIvbVLZQUjyZofqSFpre2VsgvKy0V0JQdRgN0k3H1DTGRdxHiaQjT16+H2gyuhAS9r8B2PQEwrSiqP6/Zka3pQ==

"@types/[email protected]":
version "12.0.1"