Use Nodejs for Bower and NPM

[francoispluchino]

Problem context

I know that you like this plugin to manage the Assets in a PHP project, because it only uses PHP, but there are still constraints (composer/composer#3082, #70, #75, slow RemoteFileSystem) that can not be resolved easily and quickly (or not). Of course, there are still technical solutions to overcome the problems, but it still remains the coutournements.

• For the PR #3082, we can move the require assets in the "extra" section, but in this case we lose all the benefit of using native Composer (all commands, dependencies resolution, install/update PHP packages even if there is a version mismatch of asset)
• For the camelcase problem for Bower, we can also move the require assets in the "extra" section, but in this case, we need to create our own version of the Installer and Locker. We lose more native compatibility
• For the problem of multiple dependencies of NPM, there is no solution, unfortunately
• For performance problems, there too, we can improve the RemoteFileSystem using CURL, if available, but again, the work is not easy for use the maximum native code that use the RemoteFileSystem of Composer, and will not solve the import of several version of the Solver to update.

In view of recent months concerning the 2 major problems (use the plugin in project mode, and the camelcase for Bower packages), you have noticed that it cannot be settled in directly in Composer (and there is a reluctance to manage assets using the Solver of Composer with or without a plugin).

Proposal

In this case, you also understand that the viability of this plugin is compromised, if we maintain the current implementation. I do not hide my disappointment of having to remove much of my work, qui au final, which ultimately will become useless, and return to my first prototype (using Nodejs, NPM, Bower and GIT).

I had already raised this issue in issue #60 there for several months now, but today, I think it is better to use nodejs.

The main idea of this plugin is, and always will ease to manage the assets for a PHP project. It is clear that having to use another language to do this, is not more accessible for projects wanting only the download of assets.

There are other plugin directly using nodejs, but the latters requires the mandatory preinstallation of Nodejs, NPM and/or Bower. Furthermore, for some of them, we can not use the native features of Bower or NPM, when they are required.

The idea (that I had already emitted in the issue #60) is to add the ability to automate the entire process of managing the dependencies of assets that are used with nodejs, whether for NPM or Bower.

Concerning nodejs, the plugin use the installed binary if it exists, or it download and copy the binaries in the vendors directory of the project:

• the nodejs binary depending on the platform (linux/mac/windows in 32/64 bit)
• the NPM module
• the Bower module

Of course, the installation is do only if necessary, and it is possible to add a constraint version of Nodejs engine, Bower and NPM in the extra section of the root Composer package, for validate the already installed nodejs, Bower and NPM, or select the last compatible version for the download binaries.

I have conducted tests on the use of binary downloaded on different platforms, and they working properly. But as indicated @cebe, this is not 100% sure, because of the Dynamic Link Library. But for this case, the plugin will always test (in the initialization), the proper functioning of nodejs, NPM and Bower, but also the installed version. The plugin will must also test the presence of GIT on the machine.

Using the plugin

To activate an asset dependency, this will be done in the "extra" section of composer.json of your package:

composer.json:

{
    "require": {
        "fxp/composer-asset-plugin": "~2.0"
    },
    "extra": {
        "fxp-bower-require": true
    }
}

and to declare an asset dependency:

bower.json:

{
    "dependencies": {
        "angular": "~1.4"
    }
}

The same work is done to NPM, but with extra.fxp-npm-require and the package.json file.

Regarding the root Composer package, if it directly requires assets, the procedure will be the same.

In this manner, we can use all the features of Composer, but also of NPM and Bower, while automating the entire process. Given that we can trigger the shell commands via Composer, we can perform all the tasks normally performed with Nodejs (grunt, glup, etc...), and therefore to execute the entire install/update of project with only one command (for a simple project, but also a complex project).

Example for Grunt with composer.json:

{
    "scripts": {
        "post-install-cmd": [
            "grunt"
        ],
        "post-update-cmd": [
            "grunt"
        ]
    }
}

Conclusion

I think this is the best compromise we can do, while retaining the main purpose of this plugin: facilitate the management of assets in a PHP project, while this new version requires imperatively GIT on the machine (for Bower).

Of course, the compatibility will be broken, and so I will deploy the stable version 1.0.0, create the branch 1.0, and begin the development of this new version in the master branch with 2.0-dev alias.

I will continue to do bug fixes for the 1.0 branch, but I will add no more new feature or improvements. On the other hand, I will continue to merge all PR for this branch, if any.

[moufmouf]

Hi François,

Believe me, I completely understand your disappointment. I started using composer-asset-plugin and had already included it in the Mouf framework (automatic detection of included Bower JS and CSS file in your project based on your library).

Anyway, I followed closely the issue with the Composer solver in project mode, so I understand you are stuck with the current implementation.

Still, I'm staying really keen to see any solution that can integrate Bower with Composer, so I'm really looking forward to this 2.0 version, even if it is not a "pure PHP" solution.

I was wondering what was the state of advance of the 2.0. Is it usable yet (I believe it is not yet ready according to my first tests). Any plan for a release date? Any way we can help?

[francoispluchino]

@moufmouf Currently, I have not started the development of the 2.0 branch (apart from some doc changes, the 2.0 version is the 1.0.0 version).

Alas, I cannot give you a release date for the 2.0 version, because I am very busy now, and so I do not have planned the work.

Thank you for your proposal, and any help is welcome. There are several tasks to do, including one that allows to verify the presence or absence of Nodejs, NPM and Bower, and if appropriate, download the binary of Nodejs depending the platform.

Another point, I found a display bug (Windows) with the special characters used by Nodejs, when the process is executed by Composer, so if you have a solution I'm interested.

[nsams]

@moufmouf as alternative you might want to have a look at this plugin that uses native npm+bower: https://github.com/koala-framework/composer-extra-assets

We are using it in production for some time now and are quite happy with it.

[moufmouf]

@nsams Thanks for pointing to your package! It looks your plugin does indeed a part of what @francoispluchino is trying to achieve. But it requires NodeJS to be installed first.
I've started working on a composer plugin that installs NodeJS & NPM locally if they are not available globally, just like @francoispluchino exposed above.
Using this plugin with your composer-extra-assets package, we might have a working solution, even if NodeJS and NPM are not available locally :)

I'll both keep you posted when I have a working version of that plugin.

[nsams]

@moufmouf feel free to create pull requests (we can include the local nodejs installation in composer-extra-assets)

[moufmouf]

And here we go with a first test release:
https://packagist.org/packages/mouf/nodejs-installer

It is working on Linux and Windows. I haven't tested MacOS yet.

@nsams I did not made a pull-request. Instead, I put the NodeJS installer in a separate package because some people might want to use Node in their PHP project for other things than asset management (for instance for websockets).

Do not hesitate to give me some feedback!

[nsams]

@moufmouf looking very promising! And yes - you are right - absolutely makes sense as separate package. Once the package is stable I'll look into integrating it to composer-extra-assets.

[francoispluchino]

@moufmouf @nsams If you wish to contribute to the 2.0 version of this plugin, I thank you in advance, but if this is not the case, please, do not use this issue for your own repositories.

I personally think it would be more interesting to work together, instead of work on separate projects for the same features.

Regarding the approach of @nsams, I find it unfortunate to use nodejs and not having the ability to use all the features of NPM and Bower, because all dependencies are stored in the composer.json file.

It would have been better to use files bower.json and/or package.json, especially for the root project (see this issue). With this approach, the user can use simultaneously or independently the tools of dependencies management Composer, NPM and Bower, because the plugin only adds/merge the dependencies assets in files bower.json and/or package.json of the root project.

If this is not the case, I think it is better not to use Nodejs, and keep using the Solver of Composer, but dedicated to the management of assets (see the Solution 1.2 of the issue #60).

[nsams]

Sorry for hijacking this thread!

I think using composer.json for bower dependencies makes sense as they have to be merged. And merging multiple bower.json files (execept the dependencies) isn't straight forward. Additionally it's not possible to create a bower.json in the root package containing all merged dependencies - where you can operate on using bower commands (We use it for bower link)

For package.json (which doesn't get merged) you are right - that would be possible. But I decided not to do that, simply to have it the same way as for bower dependencies. Both ways could be supported though.

Additional argument: if you clone a repository containing composer.json and packages.json (or bower.json) it isn't clear weather you should be using composer npm or bower to install dependencies.

What features of npm/bower do you want to use which their native format?

Another reason I like about composer-extra-assets: KISS. Not much complexity and code in the plugin.

One big minus: bower doesn't support .lock files (yet)

[moufmouf]

@francoispluchino Sorry, I really did not mean to highjack this thread. I asked what needed to be done to get 2.0 moving forward, you told me:

There are several tasks to do, including one that allows to verify the presence or absence of Nodejs, NPM and Bower, and if appropriate, download the binary of Nodejs depending the platform.

So I started working on a "Nodejs installer". Ok, I put it in a separate package but this is because I feel this could be used for other purposes than asset management, and therefore, that it deserves to be in its own package. But feel free to grab the code and integrate / modify / do whatever you want with it. I really did it to help.

I personally think it would be more interesting to work together, instead of work on separate projects for the same features.

👍 for that :)

Regarding whether to use bower.json and package.json rather than composer.json extra section is a good idea (because it maps closely to the JS tools) or a bad idea (because we can only merge dependencies and not much else)... I must say I simply don't know. I have almost no experience with Bower so I'd rather keep my mouth shut that give my opinion.

Here is an idea anyway with my limited understanding of the problem:

The plugin tackles one problem: "being able to have a PHP package that has JS dependencies". The goal is not to have a package that is both a PHP package and a Bower/NPM package.

Hence, would it make sense to support only a subsection of bower.json / package.json files (the subsection that can be easily merged), and trigger an error if more is provided?

For instance, maybe it does not make sense to have a "name" property in bower.json since this cannot be merged and since the PHP package we are working on is not supposed to be published on Bower?

Does it sound ok, or is it just stupid?

[schmunk42]

Related (performance): #116

[Swader]

What is the status of this? Will the next version require NodeJS? The reason I'm asking is because Node and NPM are absolute 100% horror on VMs, especially VMs hosted on Windows, and thus this plugin would become unusable.

The PHP-only nature of these solutions is appealing because PHP has no such problems and works flawlessly. Originally, I had used BowerPHP and then considered moving to this plugin, but if the next version will depend on NodeJS then it's unusable for me as someone who wants to provide cross-platform solutions for everyone.

Read about my previous approach here.

So if someone could please clarify what this all means and where the project is heading (it's been a while since this issue was updated), I'd appreciate it very much.

[francoispluchino]

For now, this is only an idea.

After lengthy discussion with the lead developers of Composer, the problem of the performance and the optimization of the Solver SAT will not be solved (but also, the install of this plugin into global mode). That's why I started to turn me to this solution ...

Currently, the plugin works pretty well - although a little slow - and for now, we remain in pure PHP.

[Swader]

After lengthy discussion with the lead developers of Composer

Yep, I read that PR, took a while and went in all kinds of directions. Shame things got so complicated.

Understood, thanks.

[kirkmadera]

The fxp/composer-asset-plugin works well for us. We get around the issue of the plugin needing to be installed first by building it into a generalized build script that runs composer install twice if the vendor directory is empty.

It's much preferred to install fxp/composer-asset-plugin within the project rather than globally. We kept running into Composer + fxp/composer-asset-plugin compatibility issues with the global installation because the global installation tends to get out of date in different environments.

I also agree with keeping the Composer solver in charge rather than Node.js. We use the plugin to include asset dependencies inside of other php dependencies that we manage. The ability to resolve dependency version constraints recursively is a must for us.

[hiqsol]

I like the idea of this plugin and since we use Yii framework we've depended heavily on it.
But I really suffered from the problems it creates: it slows down composer and complicates
integration with Travis and Scrutinizer. That's why I spend a lot of time trying to create other
solution. I've experimented a lot with different ideas but unfortunately just got convinced that
there is no way to reproduce this plugin functionality with another better plugin :)

But I've found entirely different way. We've created something like a server-side version of this plugin:

https://asset-packagist.org/

It provides installing of Bower/NPM packages with Composer only (resolving recursive dependencies too).

No Node.js and no plugins are required.

It works fine for our projects and substitutes this plugin while having none of mentioned problems.
Source code is available on github. This plugin is used inside.

[francoispluchino]

With which way do you fix the quota of Github API (5000 request/hr), as well as the private repositories? A first reviewed of your code, this question are not treated.

Instead, use your server as cache to accelerate the retrieve of versions for each package is interesting. Rightly, there is the PR #215 allowing to create a cache server compatible with the plugin.

[schmunk42]

@francoispluchino I'd have a more general question, since I think it's related, I post it here...

Why do you use the API for getting versions of a package. Isn't bower just cloning the repo and then it retrieves the the versions from git tags locally? I think that's the main reason why native bower is faster than this plugin.

I mentioned that in another issue ... when you've a repo like angular and angular/route - the plugin needs to make 30-40 API requests per package, just to retrieve all 4000+ versions.

Would it be an option to get this data from local clones?

[francoispluchino]

@schmunk42 By default the plugin use the same behavior that Composer: use the api, but you can use native GIT with no-api repo config (see GitHubDriver). Use GIT, it's problematic, because you must have the GIT installed on the computer/server, and you cannot access to private repositories (simply).

[hiqsol]

@francoispluchino

With which way do you fix the quota of Github API (5000 request/hr)

We update packages just periodically, so no problems with quota.
When composer makes requests to asset-packagist.org it receives already rendered static JSON files.

as well as the private repositories?

No, for the moment we can't help with private Bower/NPM packages.

[schmunk42]

@francoispluchino I played around with the no-api setting you linked above.

Results are amazing, IMHO.

All tests ran with GitHub OAuth token.

Example composer.json Yii 2.0 App with a few additional packages...

composer update

filled cache

via API

Reading /work/project/vendor/composer/installed.json
Reading ./composer.lock
Writing lock file
Generating autoload files

real    0m56.330s
user    0m2.910s
sys     0m1.280s

via git repo no-api

Reading /work/project/vendor/composer/installed.json
Reading ./composer.lock
Generating autoload files

real    0m24.622s
user    0m2.370s
sys     0m1.350s

⚡ x2.3 faster

---

composer update (with bower-asset/angular)

filled cache

via API

Generating autoload files

real    1m34.981s
user    0m3.260s
sys     0m1.020s

via git repo no-api

Reading ./composer.lock
Generating autoload files

real    0m30.440s
user    0m2.810s
sys     0m1.070s

⚡ x3.1 faster

---

composer install

cleared cache

via API

[95.3MB/460.40s] Writing lock file
[95.3MB/460.40s] Generating autoload files
[95.6MB/461.73s] Memory usage: 95.58MB (peak: 180.12MB), time: 461.73s

via git repo no-api

[93.9MB/133.91s] Reading ./composer.lock
[94.2MB/133.92s] Generating autoload files
[94.5MB/135.17s] Memory usage: 94.47MB (peak: 164.72MB), time: 135.17s

⚡ x3.5 times faster

[francoispluchino]

@schmunk42 It is quite possible to set up a mechanism to activate no-api option automatically if GIT is available, and do a fallback for private repositories that are not authenticated via SSH.

[schmunk42]

So my feature request would be to get an option which can be used to enable/disable no-api for bower and npm packages. Ideally with a wildcard or pattern.

There's a tradeoff between both approaches: git repos can get pretty large, i.e. bootstrap has ~80 MB as a git repo, but only 4,1 MB as zip. But scanning tags via the API also takes a while, even if you've scanned it before, i.e. angular has more than 4000 versions to fetch initially and you usually need to get the details for a lot of them, this gets really worse, if you've a lot of such packages. Instead, running git tags on a local repo is blazingly fast.

If you've filled your .composer-repo caches once you can also use them for other projects.

CC: @cebe @samdark @motin @Quexer69 @tonydspaniard

[francoispluchino]

The dist/source (git clone) is independent. With --prefer-dist, you download the ZIP even if you used the GIT driver (no-api).

[francoispluchino]

@schmunk42 With git-ls-remote, you don't have to download/clone the full repository.

[schmunk42]

I looked through the code of your plugin and composer, but it looks to me like only a no-api setting can force setting up the GitHubDriver for a repo. Currently this would need to be done per repo, right?

What would you suggest to enable this in a more global way, I imagine something like settting extra.asset-no-api to true or to an array with patterns...

[francoispluchino]

@schmunk42 Your proposal of a default value and an another array for each package is interesting.. Regarding names to use, I do not know yet.

Regarding performance, we can dramatically improve performance by improving:

• the download the versions using git-ls-remote
• using a shallow clone to download the history truncated to the first level (--depth=1) for all branches (ex. for Angular 1.x, the clone size is 8MiB versus 575MiB, and the resolution of deltas is much faster).

[schmunk42]

First step could be to have an addition setting for this if

like (dirty code)

if (
  (isset($this->repoConfig['no-api']) && $this->repoConfig['no-api']) || 
  (isset($this->pluginConfig['no-api']) && $this->pluginConfig['no-api'])
  ) {

[francoispluchino]

@schmunk42 Thanks to use the issue #226 for this. :-)

[francoispluchino]

You can read this comment to understand the reopening of this issue.

[francoispluchino]

Given that there is no a backward compatibility, and that it is impossible to have a version of the plugin installed globally, and another version installed in the project - because Composer will install the plugin in the project, but will only use the plugin installed globally - the plugin becomes a new project.

However, for the more reckless among you, you can start playing with Foxy (the new name of the plugin), and make returns and/or contribute. Do not forget, this is an alpha version, so a little leniency please :-).