Multiple Package Manager Plugins in 1 repo

[hipstersmoothie]

Is your feature request related to a problem? Please describe.

Right now you can only use 1 package manager plugin per project. This means you can't use the chrome-web-store plugin the npm plugin in one repo.

Describe the solution you'd like

This limitation is mainly because currently auto works based on git project and has no concept of a package.

In the npm plugin I have demonstrated how you can do changelog management and separate releases for sub-packages. This is all based around around lerna and can't really be moved to core.

But since each package manager plugin relies on some extra file for the package manager (all except git-tag) we could pretty easily do something like:

Example: npm plugin

1. When loaded it tries to find a package.json (single or lerna)
2. auto considers anything in that folder to be a part of the package
3. auto manages a unique changelog for each package

This could potentially be a poor experience. Instead we could just add an extra config option to all package manager plugins for a folder. This could also support the git-tg plugin (and would be needed to make it work at all).

Potential Issues

• Each plugin currently commits, tags, and pushes. this would create a lot of noise. would probably have to move those git actions to core so it only happens once? although you might want separate commits for each package.
• Root changelog - probably wouldn't make sense in this type of project. each package manager plugin would also generate a changelog for each managed package

Describe alternatives you've considered

Nothing really.

[hipstersmoothie]

Thinking about it a little more I would probably make more sense to introduce a new top level config option.

{
  // Still have some global config at root
  "name": "Andrew Lisowski",
  "email": "[email protected]",
  // Plugins used for every package
  "plugins": ["conventional-commits"],
  "packages": [
    // Target specific directories and apply a set of plugins to them
    {
      "target": "www",
      "plugins": ["git-tag"]
    },
    {
      "target": "api",
      "plugins": ["npm"]
    },
    // Specify a pattern or even and array of patterns or directories
    {
      "target": ["packages/**",  "utils"],
      "plugins": ["npm"]
    },
    {
      "target": "web-store",
      "plugins": [
        "chrome-web-store",
        // Whole lifecycle is run for each package so you can have package plugins
        ["upload-assets", { "assets": ["./path/to/file"] }]
      ]
    }
  ]
}

To accomplish this I think we could use iterator functions to run shipit on multiple things and generate the same amount of commits

ex for latest:

1. run all at same time
2. yield before committing changelog
3. commit changelog all at once
4. yield until version bumped
5. commit version if necessary
6. run until end

[hipstersmoothie]

With this setup you could really skip out on lerna all together

{
  "name": "Andrew Lisowski",
  "email": "[email protected]",
  "packages": [
    {
      "target": "packages/**",
      "plugins": ["npm"]
    }
  ]
}

If no commits are matched to a package then no release would be made. This accomplishes independent monorepo management in a much simpler way and with no lerna. If you want a fixed version monorepo going the classic route would be the way.

Another cool feature of this is that you could have two leran projects in your repo with different versioning schemes (fixed and independent)

{
  "name": "Andrew Lisowski",
  "email": "[email protected]",
  "packages": [
    {
      "target": "fixed-monorepo",
      "plugins": ["npm"]
    },
    {
      "target": "independent-monorepo",
      "plugins": ["npm"]
    }
  ]
}

[adierkens]

This means you can use the chrome-web-store plugin the npm plugin in one repo.

I'm assuming you mean you can't use both in the same project?

Each plugin currently commits, tags, and pushes. this would create a lot of noise. would probably have to move those git actions to core so it only happens once? although you might want separate commits for each package.

It may make sense to do a hybrid approach. Maybe each plugin commits, but we only push once?

I'm not really sure how tags would work in the world where each folder is it's own packaged release. Do you create a tag for each version you make? Do you make one at the end?

Thinking about it a little more I would probably make more sense to introduce a new top level config option.

I like the idea of being able to customize the release experience per package. Can definitely see some benefits of being able to manage a s3/gh-pages deploy for a docs package vs. your library packages.

One thing to handle is the inclusion of a package in more than 1 release pipeline. What happens in the case of:

{
  "name": "Andrew Lisowski",
  "email": "[email protected]",
  "packages": [
    {
      "target": "packages/**",
      "plugins": ["npm"]
    },
    {
      "target": "packages/chrome-ext",
      "plugins": ["chrome-web-store"]
    },
  ]
}

Does the chrome-ext package get published to both npm and the webstore? Or does the npm release win because it's declared first?

[hipstersmoothie]

I'm not really sure how tags would work in the world where each folder is it's own packaged release. Do you create a tag for each version you make? Do you make one at the end?

I think we can match the lerna behavior. just a lot of tags on one commit. each tag then becomes it's own release

[hipstersmoothie]

Does the chrome-ext package get published to both npm and the webstore? Or does the npm release win because it's declared first?

In that case yes it would try to do both. But that's just bad configuration. You could use that as a feature and say publish a package to two different registries (ex: npm and github package registry)

{
  "name": "Andrew Lisowski",
  "email": "[email protected]",
  "packages": [
    {
      "target": "packages/**",
      "plugins": [["npm", { registry: "https://npm" }]]
    },
    {
      "target": "packages/**",
      "plugins": [["npm", { registry: "https://github-package-registry" }]]
    },
  ]
}

[zephraph]

This sounds interesting... and complicated 😅

I'm generally 👍 with having configuration over trying to be overly smart with detection (because that can break down in unexpected ways and be hard to test).

I guess my first question is what the default state of this looks like. If you just have an npm plugin, do you need to add configuration before it'll work? Same question with the others.

As for the git interaction, it seems to me like git interactions should generally just be a part of the plugin pipeline. If a plugin needs to do a commit, it should just be able to tap into a commit-able hook. Pushing should likely only be handled in core though as it has implications on things like how CI runs.

[hipstersmoothie]

If you just have an npm plugin, do you need to add configuration before it'll work?

Configuration that works today should work in packages world. So no changes required. Most of the breaking changes would be node API side and not visible to normal users.

If a plugin needs to do a commit, it should just be able to tap into a commit-able hook.

I like this idea. Formalizing this git interaction within auto is probably useful. And if they don't use this hook it just means a little more noise (ex: an extra commit and an extra push)

[barreeyentos]

I also was thinking about how to use Auto with a monorepo and came up with a slightly different approach that may work for your need.

Basically if you have one monorepo you can have multiple sub-directories that each have their own .autorc file that can setup whatever plugins each sub project would need.

You can then choose to have a different prefix for each project so that each project can be released at different times if need be. For example a release of sub-project1 could be tagged with sub-project/v1.4.5 and another project would get the tag sub-project2/v9.9.9

Auto can then use something like git describe --tags --matches "sub-project1/*" to get and update tags for each project accordingly.

Just a thought.

[hipstersmoothie]

That’s actually pretty close to the approach that I’ve been taking. I’ve been working on this this week. I’ve had to add that flag to some of the commands so far (—matches) and it’s been going pretty smoothly. I’ve gone with the “packages” field approach and it’s basically just and array of “autorc”s To get the prefix I’ve added a hook for plugins to provide a name. This will also help signal to auto if the plugin is multi package compatible

…

On Wed, Jan 29, 2020 at 9:11 PM Alejandro Barrientos < ***@***.***> wrote: I also was thinking about how to use Auto with a monorepo and came up with a slightly different approach that may work for your need. Basically if you have one monorepo you can have multiple sub-directories that each have their own .autorc file that can setup whatever plugins each sub project would need. You can then choose to have a different prefix for each project so that each project can be released at different times if need be. For example a release of sub-project1 could be tagged with sub-project/v1.4.5 and another project would get the tag sub-project2/v9.9.9 Auto can then use something like git describe --tags --matches "sub-project1/*" to get and update tags for each project accordingly. Just a thought. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#917?email_source=notifications&email_token=AAJDEBGUZR5HF6P3OKRILTDRAJOOXA5CNFSM4KMLWYF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJWTQI#issuecomment-580086209>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJDEBDX72NB5Z7ZJPTDHVLRAJOOXANCNFSM4KMLWYFQ> .

[zephraph]

Going back and reading through this again, I'm actually pretty excited! The multiple tags prefixed w/ the package name sound really good. Also, I really like the packages field.

[vincentbriglia]

@hipstersmoothie is there an easy way today to publish to 2 npm registries with shipit, that you've seen? or some feature toggles in auto that I don't have a good overview on right now.

[hipstersmoothie]

@vincentbriglia would a plugin that publishes to the GitHub Package work? it could pretty much piggyback off of the npm plugin. It would be easy for next and latest release, canaries don't make as much sense. Thoughts?

[vincentbriglia]

@hipstersmoothie we have a case where we publish the same package to the NPM registry and the GHPR registry. Reason here is that the GHPR, even though advertised as such, doesn't proxy packages correctly. The underlying problem being that we have the same scope on npm and github.

We are mainly developing behind closed doors, we use the canary builds to have conversations/approvals with our design team from feature branches > next so canary builds are still useful for us in this context. The npm plugin currently provides for this functionality so it would be a shame to lose it.

I suppose the context with using GHPR is generally that it's the primary location to publish to in the private "organization" context and if a component is to be made public, npmjs.org is secondary. (I haven't seen anyone install public components from github). In the open source context, it's generally npmjs, no GHPR or another private package registry.

on a side note, since you mention a specific GHPR plugin: we've set some time aside for next week to create an auto plugin that would remove package versions based on some rules (orgs are limited to 50Gb of packages, and that includes docker images)

• remove last canary in range
• remove nth last next in range
• ...

I would also happily wait with you creating a specific ghpr publisher until work starts on v10, the ideas presented here seem very exciting and are perhaps more "future proof".

we could live without publishing public and private at the same time for the time being, but at least so you know this is the usecase from me.

[hipstersmoothie]

I was more thinking about a "secondary package registry" plugin. So the npm plugin would work as it does, publishing to whatever registry is configured. Then this new plugin would publish to a second registry (whether that's npm or ghpr doesn't matter) releasing whatever versions are on the HEAD commit.

on a side note, since you mention a specific GHPR plugin: we've set some time aside for next week to create an auto plugin that would remove package versions based on some rules (orgs are limited to 50Gb of packages, and that includes docker images)

This seems like a good feature. I didn't know those limits existed!

[vincentbriglia]

this new plugin would publish to a second registry (whether that's npm or ghpr doesn't matter) releasing whatever versions are on the HEAD commit

well that would definitely work! edit: also for our scenario(s)

[laughedelic]

Hi! What is the current state of Auto with respect to this thread? Is it possible to publish more than one thing from the same release?

since each package manager plugin relies on some extra file for the package manager (all except git-tag)

I think this is a wrong assumption. Does docker plugin rely on any file? It might be unnecessary for it, so maybe it's a bad example. Here is another one then: I'm interested in using Auto with sbt (the most common build tool for Scala) and it doesn't have any machine-readable JSON/XML config. An sbt build is configured with Scala code and to extract any information about the build you would need to communicate with sbt in some way.

This accomplishes independent monorepo management in a much simpler way

It also seems from this thread that the goal is to accommodate monorepo projects with multiple subprojects that have different publishing needs. What about a single project that produces different kinds of artifacts? For example a Docker image and a configuration archive (to be uploaded somewhere). Or a GitHub action which can be published as a library to NPM and as an action on GitHub Releases.

Each plugin currently commits, tags, and pushes. this would create a lot of noise. would probably have to move those git actions to core so it only happens once? although you might want separate commits for each package.

I think this is the main issue with the current implementation of plugins. This goes back to my confusion about the claim that each command "only does one thing really well". In my opinion the publish hook should really only publish for the given package manager, not create commits or push git tags. If each publishing plugin can focus only on implementation of its package manager specifics, common parts of the process can be reused and/or shared. Is it still possible to achieve this with Auto or is it too biased to the NPM-like projects?

[hazim1093]

Having the option to add prefixes for each project would be very useful for non-npm use-cases as well. e.g. In a non-npm case that I have at the moment, I am ok with keeping a .autorc file in each project and handle the rest during CI, as long as the tags created can be different for each project.

Since there is already an option to disable the version prefix v, would it be simple to allow changing the version prefix to any other string than v ? e.g. version prefix for project1 in a repo could be project1-v

You can then choose to have a different prefix for each project so that each project can be released at different times if need be. For example a release of sub-project1 could be tagged with sub-project/v1.4.5 and another project would get the tag sub-project2/v9.9.9

[verena-ifx]

Having the option to add prefixes for each project would be very useful for non-npm use-cases as well. e.g. In a non-npm case that I have at the moment, I am ok with keeping a .autorc file in each project and handle the rest during CI, as long as the tags created can be different for each project.

Since there is already an option to disable the version prefix v, would it be simple to allow changing the version prefix to any other string than v ? e.g. version prefix for project1 in a repo could be project1-v

You can then choose to have a different prefix for each project so that each project can be released at different times if need be. For example a release of sub-project1 could be tagged with sub-project/v1.4.5 and another project would get the tag sub-project2/v9.9.9

I am facing this issue as well right now and agree, that this would be a idea for a future release. Is there any news on this topic? For now, I just use a workaround by only calling auto shipit on my first package and then manually setting the canary version in the following packages and then using https://github.com/JS-DevTools/npm-publish to publish to npm.