Js component expansion

[garettarrowood]

Thank you for all the reviews and merges over the last couple days @shageman!

@benlangfeld and I have been discussing a PR to expand cobratest's component detection to include node js components as well. This would be a larger addition and we wanted to get your opinion on the possibility before we got started.

The first round implementation would only detect js components that had been altered themselves, and then run their tests. A second round would work on retrieving the js component dependencies. This may be done by intelligently parsing their package.json files. It may mean adding some dependency configuration directly to the js component (whether it be in the package.json itself or somewhere else). I am sure @benlangfeld may have some details to add as well.

What are your thoughts on such an expansion?

[benlangfeld]

For a little more context, we have a collection of Ruby components (simple gems or Rails engines), and use the Rails asset pipeline throughout the majority of our application. We also use react_on_rails, including Webpack, to build a bundle of Javascript code that our umbrella application depends on, as a parallel to the asset pipeline.

Further, this JS bundle is made up of several npm modules which form a parallel component graph to that which can be inspected via rubygems/Bundler. These Javascript components obey other rules of Rails components, including implementing a test.sh script.

Our goal is to merge the Ruby and Javascript component graphs into a single representation for the purposes of both cobratest (when executing our test suite in CI) and cobradeps for visualising the dependency graph of our components.

While there is little public documentation of this multi-language approach to Rails Components (something we also intend to resolve by way of blog posts etc), our immediate goal is conditional testing of the complete graph and I believe it's appropriate for cobratest to, at least optionally, support traversing an npm module graph (starting from the umbrella npm module) and merging it with the rubygems graph.

We propose to calculate the JS component graph based on the output from npm ls --json. For the time being, until the link: dependency specifier is available and can be relied on to establish which modules are components, we propose matching npm modules with component paths based on providing a prefix to filter by (for us @powerhome/) and matching the suffix against the component's directory name.

For example, the following components:

components/
|___A/
|   |___ package.json (defining `@myorg/A` package, dependency on `@myorg/B`)
|
|___B/
    |___ package.json (defining `@myorg/B` package)

Where a change is detected in components/B/*, components/A/test.sh and components/B/test.sh would be considered "to-test" by cobratest.

The output from npm ls --json in this case looks like this:

{
  "name": "@myorg/umbrella",
  "version": "0.0.1",
  "problems": [],
  "dependencies": {
    "@myorg/A": {
      "version": "0.0.1",
      "problems": [],
      "dependencies": {
        "@myorg/B": {
          "version": "0.0.1",
          "problems": []
        }
      }
    }
  }
}

[shageman]

This is an interesting one - with lots of implications.

The heuristic you propose to detect local components makes sense to me. That said, I have no insight into whether this is a common of organizing js packages.

If this functionality is only parallel to ruby component detection, we should ask why these two things should live together. One answer would be that both dependency detection mechanisms share some (more abstract) portion of logic that. And they certainly do: directory->component files assumption, transitively affected components.

This leads me to think that cobratest should evolve to be made up of three components itself: js dependencies, ruby dependencies, and abstract dependency calculation.

An open question then is: How to denote which js components depend on which ruby components? Also, what does it mean for a js component to depend on a ruby component? Or is it only ever going to be these js components depend on this ruby app?

[benlangfeld]

The heuristic you propose to detect local components makes sense to me. That said, I have no insight into whether this is a common of organizing js packages.

It certainly is fairly common for JS packages to be broken up in this way. See Babel and, more generally Lerna, for reference.

If this functionality is only parallel to ruby component detection, we should ask why these two things should live together. One answer would be that both dependency detection mechanisms share some (more abstract) portion of logic that. And they certainly do: directory->component files assumption, transitively affected components.

Precisely, these concerns are shared. Should we implement a parallel tool to cobratest to deal with the JS components (which we would presumably write in JS in order to better fit into a JS-only world without imposing Ruby), it would be almost entirely a clone of cobratest' logic.

This leads me to think that cobratest should evolve to be made up of three components itself: js dependencies, ruby dependencies, and abstract dependency calculation.

Sure. How far do you want to go in isolating these concerns? For such a small tool, I believe a full on plugin architecture would be excessive, and that a simple parallel to GemfileScraper would be sufficient.

An open question then is: How to denote which js components depend on which ruby components? Also, what does it mean for a js component to depend on a ruby component? Or is it only ever going to be these js components depend on this ruby app?

A Ruby component which depends on a JS component would have to itself be a npm module. A Ruby component which wishes to be available as a dependency of JS components likewise. In fact we have several components where this is the case; they have a .gemspec and a package.json, and two sets of dependencies. Some of these components even have both Ruby (Rails engine) and JavaScript (React) implementations within the same directory, and are so grouped to avoid having a pair of very similarly named components whose separation of concerns is not clear and which rarely change independently.

Of course this necessitates de-duplicating the calculated set of components, if a single component may appear in both Rubygems and npm dependency trees, but that is simple.

[shageman]

Alright, I am convinced that we should try this. I am excited to see where this goes!

…

On April 8, 2017 at 12:20 PM Ben Langfeld ***@***.***> wrote: > > > The heuristic you propose to detect local components makes sense to me. That said, I have no insight into whether this is a common of organizing js packages. > > > It certainly is fairly common for JS packages to be broken up in this way. See Babel and, more generally Lerna, for reference. > > > If this functionality is only parallel to ruby component detection, we should ask why these two things should live together. One answer would be that both dependency detection mechanisms share some (more abstract) portion of logic that. And they certainly do: directory->component files assumption, transitively affected components. > > > Precisely, these concerns are shared. Should we implement a parallel tool to cobratest to deal with the JS components (which we would presumably write in JS in order to better fit into a JS-only world without imposing Ruby), it would be almost entirely a clone of cobratest' logic. > > > This leads me to think that cobratest should evolve to be made up of three components itself: js dependencies, ruby dependencies, and abstract dependency calculation. > > > Sure. How far do you want to go in isolating these concerns? For such a small tool, I believe a full on plugin architecture would be excessive, and that a simple parallel to GemfileScraper would be sufficient. > > > An open question then is: How to denote which js components depend on which ruby components? Also, what does it mean for a js component to depend on a ruby component? Or is it only ever going to be these js components depend on this ruby app? > > > A Ruby component which depends on a JS component would have to itself be a npm module. In fact we have several components where this is the case; they have a .gemspec and a package.json, and two sets of dependencies. Some of these components even have both Ruby (Rails engine) and JavaScript (React) implementations within the same directory, and are so grouped to avoid having a pair of very similarly named components whose separation of concerns is not clear and which rarely change independently. Of course this necessitates de-duplicating the calculated set of components, if a single component may appear in both Rubygems and npm dependency trees, but that is simple. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub #5 (comment) , or mute the thread https://github.com/notifications/unsubscribe-auth/AALZzZPbqqjBWmLKEK4h5wgHKVNifkxGks5rt19jgaJpZM4M2xkU .

[benlangfeld]

Thanks for your reply @shageman . We no longer use this library; in the meantime, @garettarrowood and I built https://github.com/powerhome/cobra_commander which includes this suggested functionality and several other things that we stumbled upon requirements for in our large CBRA.