Support individual packaging with multi-compile

[HyperBrain]

What did you implement:

Closes #181
Closes #120
Closes #173
Closes #117
Closes #116
Relates to #60

How did you implement it:

If package.individually: true is set in the service, the individual entry points (functions) are now compiled by using WebPack's multi-compile feature. That means, that each function is optimized separately as well as the externals for every function. The node_modules for each function are then packaged individually, so that every resulting function zip contains only the needed modules and the individually optimized function handler.

Additionally the plugin does not change serverless.config.servicePath anymore, so that other plugins and the framework itself behave as expected and work during any lifecycle event and Serverless does not interfere with the packaging of the optimized node modules anymore.

Further changes

• Added ESLint as preparation for 3.0.0, so that the implementation is conformant.
• Added unit tests for the new functionality (as well as for old ones where tests where missing).
• Added verbose output with --verbose

How can we verify it:

Use a simple service that defines multiple functions, where each function uses a different set of dependencies. Use a webpack config that uses lib.entries for entry and set the package wide individual packaging in the service.

Now run serverless package and check the generated function zip files in the .serverless folder.

I checked it with one of our production projects, including tests with serverless invoke local. Found no issues. The only thing I have to verify is, that deployments without individual packaging still work.

Todos:

• Write tests
• Write documentation
• Fix linting errors (not yet implemented)
• Make sure code coverage hasn't dropped (not yet implemented)
• Provide verification config / commands / resources
• [ ] Enable "Allow edits from maintainers" for this PR
• Update the messages below

Is this ready for review?: YES
Is it a breaking change?: NO

[johnf]

should be

return BbPromise.resolve().return(stats);

or the next module errors out with null for stats.

[HyperBrain]

Good catch. Will change it.

[johnf]

@HyperBrain What should be different if this works? More files inside the zip?

[HyperBrain]

No. It will package only the node modules that are used by the function that is being packaged. Previously if you enabled individual packaging, all the packages were the same (it even packaged all functions in all function packages), i.e. it just duplicated the service package file (as with "without the individually" option). The new packaging is visible as soon as you also set "webpackIncludeModules".

The second major improvement is, that now every function is compiled individually, which leads to much better optimization, because every function is optimized separately. This will show up, when you enable tree shaking in WebPack.

And last, it prepares to use manual multi-compiles. The next step will be, that you can emit an array of webpack configurations from your webpack.conf, that will automatically trigger the multi-compiler. The new packaging infrastructure will allow for that without having to add any new implementation - only validate has to be touched to make it work.

Additionally, the issues with missing modules and unstable packaging should be solved as the plugin does not change the internal Serverless servicePath anymore, which led to unpredictable behavior.

[johnf]

nm. I had wrong config in serverless.yml. I can now see a different zip file per function.

Some of them look wright and have a single js. Others seem to be 8M in size and have the while source inside.

I'll have more time on the weekend to try and work out what the difference is

[HyperBrain]

You can also try to use "lib.entries" for the webpack entry definition that is available since 2.2.0.
This makes sure that the plugin sets the correct entry points.

// webpack.conf
const slsw = require('serverless-webpack');
module.exports= {
  ...
  entry: slsw.lib.entries,
  ...
  output:
    ...
    filename: '[name].js',
    ...
};

This makes the use of the plugin quite easy and eliminates the chances for errors. It could be that the 8MB issue comes from the old manual declaration.

[HyperBrain]

Switched one of our production services to this branch now. After the latest fixes it packages everything correctly and independently and even works with git dependencies and the serverless-warmup plugin (which creates a separate function dynamically during the package process).

Also works with separate build and deployment (using the --package option) from different servers.

[cwaltken-edrans]

On this branch I get the following error... Adding underscore as a dependency did not solve the issue...

  Reference Error ----------------------------------------
 
  _ is not defined
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.11.1
     Serverless Version:     1.18.0

[HyperBrain]

Fixed in a minute... I checked in the wrong file ;-)

[HyperBrain]

@cwaltken-edrans Can you update and try again?

[cwaltken-edrans]

Thanks for the quick update!

Can you update and try again?

That fixes the lodash error. Still I'm back at this one:

  Type Error ---------------------------------------------
 
  lambda is not a function
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.11.1
     Serverless Version:     1.18.0

Maybe there is still another dependency missing?

[HyperBrain]

I used the invoke local multiple times today and had no problems so far. It might be a specific configuration in your project. The best way to investigate it would be to have a quite simple test project that reproduces the error.
Can you create an issue for the problem? Then we can handle it there and we could put all information we have into it.

[cwaltken-edrans]

I used the invoke local multiple times today and had no problems so far. It might be a specific configuration in your project. The best way to investigate it would be to have a quite simple test project that reproduces the error.
Can you create an issue for the problem? Then we can handle it there and we could put all information we have into it.

I've just checked again. I broke the entry function trying to fix the issues I had previously. I reverted the changes and now I can build my function.

Sadly my initial issue is still there. Functions invoked locally don't have access to files bundled with the file-loader it seems. I'll open an issue on that.

[HyperBrain]

Yes, please do that. I did not do in-depth tests yet with different loaders, so there might be still issues in there that have to be fixed. A big thank you for taking the effort to test 🥇