Support Kotlin when updating Gradle files

[JLLeitschuh]

Trying to add a project like this one: https://github.com/JLLeitschuh/ktlint-gradle

Gives the following error:

Repo must contain a build.gradle.

The bot should support build.gradle.kts files, or even better, supporting *.gradle.kts files and *.gradle files.

---

More in depth information about this issue can be found here:

• Fully Featured Gradle Support with the Gradle Tooling API #1164
• https://github.com/dependabot/feedback/issues/172#issuecomment-637591057

[greysteil]

I'm am totally, 100% keen on this.

Dependabot's Gradle support is still very alpha, because parsing the build files is tricky. (Well, that's disputable, but my Java is terrible, so our parser isn't very good yet - the code is here and if you fancy helping out on it I'd love the help.) I'd love to add Kotlin support as soon as Dependabot is handling standard build.gradle files properly.

[JLLeitschuh]

You might want to consider executing users's builds to see what dependencies actually get loaded instead. You could parse the output of running the dependencies command.
Additionally, you may want to consider encouraging users to use the dependency locking feature which might provide something that is easier to parse & updated in an automated PR.

https://docs.gradle.org/current/userguide/dependency_locking.html

[rahulsom]

@greysteil The code you linked has moved away. It's been 3 months :). Is that still open?

[greysteil]

Definitely still open, and I will get to it, but I'm fighting on lots of different fronts at the moment!

Sorry about the lack of progress on this one! I completely rewrote the Gradle parser (it's a lot more robust now), and adding Kotlin support should be possible in future. Progress has been a little slow as I've had two folks join the team (🎉) and needed to do some meaty refactors to get the codebase into a place everyone was happy with, but we're hoping we can be firing on all cylinders in the new year.

All Dependabot's Gradle-related code can now be found here.

[dmsergeevN26]

Hello! I was wondering if there's any work being done to add support for Kt? Thank you!

[greysteil]

Not right now, but we'd like to add it once we have capacity. (We're currently all hands on deck dealing with user issues post GitHub-acquisition!)

[JLLeitschuh]

Related #1164

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[rahulsom]

Poke

[feelepxyz]

@rahulsom just pinned it 👌

[MorrisonCole]

FYI: in the meantime, https://github.com/ben-manes/gradle-versions-plugin is a nice alternative! (But I too wish that Dependabot supported Kotlin 😉 )

[kganger]

Really would like to see dependency support for kotlin build files!! IMHO .kts is the new standard for gradle

[JLLeitschuh]

The current state is best documented here. It also explains why this problem isn't as easy as it is for NPM and why the current implementation for Groovy isn't completely accurate.

#1164

[fabian-braun]

Would it be possible to implement support for .kts files in "beta"-form leveraging the simple parser? It sounds like the fully featured support through the Gradle Tooling API is still far in the future. Would you welcome contributions? I suppose the starting point would be here?

[JLLeitschuh]

This is a project that is really important to me and also important for me that we get it right.
I'd really like to avoid a half-baked solution on this one. As a member of the Gradle team, I've had a variety of different conversations both with the GitHub team and my own team about how to do this right. Unfortunately, due to the nature of how Gradle works, we have yet to come up with any sort of 'perfect' solution. Many of them are flawed in one way or another.

The dependabot problem is actually two problems:

1. Figuring out what dependencies your project has
2. Updating your project dependencies

We've had a variety of different thoughts on this, one of those would be to break apart the two problems so they are distinctly separate from each other.

Figuring out what dependencies your project has

As far as I'm aware, dependabot figures out that your project has a vulnerable dependency via the dependency scanning logic that GitHub uses to generate the dependency graph. I'm not certain if dependabot does this parsing or not.

We were thinking that this data could most likely be provided by Gradle to GitHub using a GitHub action that GitHub automatically applies when it detects a Gradle project. Once the Gradle build is complete, it would send the dependency data to a GitHub API automatically to update the project's dependency graph.

Updating your project dependencies

This is a much harder problem. Due to the nature of Gradle using fully fledged programing languages (Groovy, Java, and Kotlin) for it's configuration, means that tracking down the location where a dependency version is declared becomes nearly impossible in any non-trivial build logic.

One solution for this is having either having some sort of widespread use of dependency lock files. Newer versions of Gradle already support locking, but it is not enabled by default. Older versions could support it through an official Gradle plugin.

There are a variety of problems with this solution, namely:

• Lock files are not in widespread use currently
  • This could be remedied with a bulk PR to all 2 million GitHub repos using Gradle to add them. (I've done this personally on a smaller scale to fix a security vuln in 1,596 projects using maven POM files for the GitHub Security Lab Bug Bounty Program).
• People have negative initial experiences with lock files. Modifying versions in a build.gradle[.kts] file no longer causes the version to change as expected.
  • The NPM ecosystem was able to accept package-lock.json files. However, NPM support far more rich version selectors than Gradle (eg. >, >=, ~, ect...).
  • Gradle could support more rich version descriptors. However, this is not a backwards compatible solution.
• Once lock files reach any appreciable size they become human unreadable.

Summary

Its unfortunate that it's been almost 2 years since my initial post on this topic and we still don't have a good solution here.

I continue to be open to ideas and discussions here about this problem. If anyone is similarly very passionate about this problem I'd like to work with others on this problem.
As always, I'm always reachable in the Gradle Community Slack channel.

There is more relevant information about this issue here: #1164

[chali]

I would lean towards using dependency locks.

tracking down the location where a dependency version is declared becomes nearly impossible in any non-trivial build logic.

Agreed. Our experience from linting gradle build files (just groovy) you will encounter a lot of creative ways how to declare dependencies. Leading to missed dependencies or false positives and undermining trust in your tool.

We are using the predecessor of Gradle locking at this moment with the end goal to move to Gradle dependency locks. Internal CI tooling updates locked dependencies by delegating to Gradle update dependencies. The results are good so far. We would love to see a similar feature in open source world too.

Lock files are not in widespread use currently

This is a kind of chicken or egg problem. To use locks you need tooling to easily update them to build the tooling you need people using locks.

Modifying versions in a build.gradle[.kts] file no longer causes the version to change as expected.

I believe that Gradle fails when there is an inconsistent state between a lock and a build file.

Once lock files reach any appreciable size they become human unreadable.

I would argue that lock files are intended to be machine-readable. I would say human interface is dependencyInsight

[JLLeitschuh]

I believe that Gradle fails when there is an inconsistent state between a lock and a build file.

This problematic for dependabot then. Unless we support richer version specifications for gradle, if you have your versions locked to a specific version in the build.gradle.

What we don't want is for dependabot to fail updating dependencies 99% of the time because of how tightly locked down most users make their dependencies. You really don't want dependabot creating PRs that will always fail to build.

[chali]

I would suggest calling ./gradlew dependencies --write-locks from dependabot. That will give users the flexibility to pin some dependencies and update only those that are defined with a dynamic rich version.

The limitation is that dependabot has to be able to invoke grade. Unless there is a wrapper that could be challenging. However, I see any motivation to add a wrapper to your repository as a plus :-)

An example of such an approach in our build file The dependencies we manage have a dynamic latest.release but jackson is fixed to a specific version

I'm not saying that version declaration couldn't be improved but it is rich enough.

[Jerbell]

Until Kotlin DSL is supported I have a bit of a hack:
Add this to your parent build.gradle.kts (subprojects section for multi-module)

    tasks.register("createDependabotFile") {
        doLast {
            mkdir("$projectDir/dependabot")
            val file = File("$projectDir/dependabot/build.gradle")
            file.writeText( "// Do not edit manually! This file was created by the 'createDependabotFile' task defined in the root build.gradle.kts file.\n")
            file.appendText("dependencies {\n")
            project.configurations.getByName("runtimeClasspath").allDependencies
                    .filter { it.group != rootProject.name && it.version != null }
                    .forEach { file.appendText("    compile '${it.group}:${it.name}:${it.version}'\n") }
            project.configurations.getByName("testRuntimeClasspath").allDependencies
                    .filter { it.group != rootProject.name && it.version != null }
                    .forEach { file.appendText("    testCompile '${it.group}:${it.name}:${it.version}'\n") }
            file.appendText("}\n")
        }
    }

It will create a dummy build.gradle file in a dependabot directory. Have dependabot analyse that build.gradle & make suggestions. When you update your relevant kts file, then pull requests get closed automatically.
You might want to switch runtimeClasspath & testRuntimeClasspath to be runtimeClasspath & testRuntimeClasspath respectively.
I have this task run after Gradle sync in IntelliJ.
Feel free to improve!

[Hakky54]

Thank you @Jerbell and @teolemon for sharing your hacks, I used it with a multi module project and it worked. Just sharing here the github action for others who are seeking for a temp solution till the actual feature is available.

dependabot.yml

version: 2
updates:
  - package-ecosystem: "gradle"
    directory: "module-a/dependabot"
    schedule:
      interval: "daily"
  - package-ecosystem: "gradle"
    directory: "module-b/dependabot"
    schedule:
      interval: "daily"
  - package-ecosystem: "gradle"
    directory: "module-c/dependabot"
    schedule:
      interval: "daily"

[huehnerlady]

Any update on this issue? Gradle supports kotlin for quite a while now and this issue is also open for over 2 years now. Is there any progress on this?

[JLLeitschuh]

I, at Gradle, have been engaging in conversations with the GitHub team about this issue. But there is no timeline nor any commitment here yet. The component we are most likely to fix first is the 'Figuring out what dependencies your project has' as I detailed in my earlier post.
#2238 (comment)

This will most likely be driven by a GitHub action since determining your resolved dependencies requires that Gradle's internal dependency resolver logic actually be executed to get accurate results.

At this time though, I can't commit to any sort of timeline on anything here. But this is something that I (personally) am passionate about solving in the near future.

[MysterAitch]

In the meanwhile, renovate painlessly supports kotlin dsl and does grouped updates.

[JLLeitschuh]

In the meanwhile, renovate painlessly supports kotlin dsl and does grouped updates.

I don't know what renovate is doing explicitly, but I'm guessing they are solving the problem "good enough" instead of "correctly". IE. There's probably a non-zero chance that a change that renovate makes doesn't actually upgrade the resolved dependency, only the declare dependency, thus leaving you still vulnerable. This is because Gradle dependencies declared in Gradle build files are not WYSIWYG. Plugins, other dependencies, and forced configuration rules can all change what dependency gets resolved.

If "good enough" instead of "correct" is fine for your organization or use case, don't let me stop you. My goal is go work with the dependabot team to get us as close to "correct" as possible.

[rarkins]

Renovate determines Gradle dependencies by dynamically creating a task and then calling the repository's gradle wrapper to execute it: https://github.com/renovatebot/renovate/blob/master/lib/manager/gradle/gradle-updates-report.ts#L41-L62

[MysterAitch]

If "good enough" instead of "correct" is fine for your organization or use case, don't let me stop you. My goal is go work with the dependabot team to get us as close to "correct" as possible.

An admirable goal and certainly one to aim for.

Unfortunately, I'm not sure I understand what alternative you are advocating for.

As far as I can tell (sorry if I missed something) -- until the "correct" solution you mention is available -- the choice for kotlin dsl support is between "good enough" (renovate, or any other of the available tools) and "not available" (dependabot).

[JLLeitschuh]

Renovate determines Gradle dependencies by dynamically creating a task and then calling the repository's gradle wrapper to execute it: https://github.com/renovatebot/renovate/blob/master/lib/manager/gradle/gradle-updates-report.ts#L41-L62

Looking at that, that looks at declared dependencies, not the dependencies that Gradle resolves (IE. after dependency resolution happens, and all of the other version constraints are considered, what version is actually used by your build).

[vitusortner]

Support for .gradle.kts files has been added recently https://github.blog/changelog/2020-12-18-dependabot-gradle-kotlin-and-composer-v2-support/. Can this issue be closed?

[Hakky54]

New year present came pretty quick!

[feelepxyz]

This is supported, we just forgot to close this! Closing 🎉

[savvisingh]

Thanks for adding this support, But sadly dependabot does not resolve dependencies if the versions numbers are defined in a separate file :(, Is there any workaround for this issue

[huehnerlady]

even with adding it as a variable in the same gradle file we now get 2 PRs for the same dependency 🤔

[feelepxyz]

@savvisingh @huehnerlady could you open new issues for the issues you've run into? Definitely shouldn't be creating two PRs to update the same dependency.

We've have plans to start cloning the full repository when updating dependencies which will make it easier to resolve any file references across the repo.