Dependabot takes hours for a common dependency upgrade in a mono repo.

[AlekhyaYalla]

Issue:
One of the issues we observed in dependency updates, is Dependabot taking a very huge amount of time in a mono repo for updating a dependency which is populated in more no of packages. In our case, we have a mono repo with yarn as a package manager and ~600 packages in it. React is one of the commonly added dependency in ~350 packages in this repo (both dependency & devDependency). While we try to update react using dependabot in this repo, it is taking nearly ~4hrs to finish the update task. The usual time by dependabot for doing an update of an irregular dependency in this repo is ~15-20min. The below are the observations from the investigation we did for common dependency.

Observations:

• While generating the updated lock file content in this method, it is passing the dependencies to-be-updated along with the requirements (These are the package.jsons that has the dependency in it, either devDependency or dependency).
• This method calls the respective package manager function (in our case, it is yarn) to update the dependencies. This method runs a loop to run the commands add & install in each and every package.json to autogenerate the lock file content.
• With our investigations, it is evident, that the time taken to update a dependency in the mono repo is directly proportional to the number of package.jsons that dependency presents in.

Wanted to know:

• In the case of mono repo, all these packages.jsons would be updating to the same version and having a common lock file in the repo level, is it okay to run a single command in lock file level/repo level and autogenerate the content of it?
• Are there anything else contributing to the longer runs in this area (apart from investigations)?
• If this is a known issue, are there any plans/planned to accommodate this?
• Is it possible to optimize these commands in Dependabot-core using “yarn workspace” or any mono repo tool, that would help in running these commands efficiently at the repo level, instead of running them in each and every package level?

@jurre @hmarr @feelepxyz Please share your thoughts on this

[jurre]

Hey @AlekhyaYalla, thanks for looking into this.

Yes, monorepo support in Dependabot is currently not great, and I believe you are right that we will perform a separate update for every package in the repo.

In the case of mono repo, all these packages.jsons would be updating to the same version and having a common lock file in the repo level, is it okay to run a single command in lock file level/repo level and autogenerate the content of it?

If there is a single lockfile, and there can be no conflicting (transitive) dependencies, we may be able to do this yeah. There might be some edgecases here that I am missing though.

Are there anything else contributing to the longer runs in this area (apart from investigations)?

There likely are, but nothing we have identified/tracked currently.

If this is a known issue, are there any plans/planned to accommodate this?

Nothing tracked currently, but we have been aware of it and would welcome fixes around this!

Is it possible to optimize these commands in Dependabot-core using “yarn workspace” or any mono repo tool, that would help in running these commands efficiently at the repo level, instead of running them in each and every package level?

Maybe? To be honest it's been some time since I've looked into this, and I don't have a good sense of where the bottlenecks and opportunities for improvements are, but relying on workspaces, which I believe is natively a part of yarn, sounds like a very promising direction.

I wish I could provide more guidance/insights into this, but the team has been heads down on other projects recently.

PS: Most of the methods you tried linking to link to the file instead of a specific line, so I wasn't able to figure out which methods you were looking at.

[AlekhyaYalla]

Hey @jurre Thanks a lot for replying impromptly. My bad, I missed the links to the methods, Please find them below
Update lock file content
Function for update
Loop with commands
(The links are in the same order as the original post)

Thanks for your insights. I'll spend some time on this and will be updating the same thread if there is any workaround for this in a monorepo.

[AlekhyaYalla]

Below are the conclusions. Please comment/correct me if any of these have different evaluations or If I missed any edge cases.

• Currently, if there are multiple package.json(s), yarn add & install running that many times. (Say 200 package.jsons, these commands run for 200 times in each directory)
• As this step is for generating the lock file content, as long as the directory contains the lock file its valid to run the add & install.
• I believe a mono repo is something where multiple packages are packed in the same repository, sharing the common dependencies & the lock file. So, in the case of a mono repo, we do have only one lock file at the root level, which is not neccessary.
• Running yarn add & install in each and every package will keep on rewriting the same lock file for the same version every time which is time-consuming and unnecessary.
• If we could run the add & install commands in one of the package.json for a version, the root lock file content will be generated. Hence, we can skip the rest of the package.json(s).
• This is when we have a single version to be installed. If there is more than one version, we do follow the same exercise for each and every version, making sure each version has run yarn add & install and generates the lock file content.
• If there are multiple dependencies to be installed, the above process will be followed for each dependency separately.

Here is the PR with proposed changes.
@jurre Please let me know if I'm missing any of the cases or being inclined. Would love to get feedback from you on this :)

[deivid-rodriguez]

I did not read this in detail, but since the linked PR was closed I guess this was fixed? Please reopen if not!