Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heroku slug size: can node_modules be deleted after build? #491

Closed
nruth opened this issue Oct 5, 2016 · 9 comments
Closed

Heroku slug size: can node_modules be deleted after build? #491

nruth opened this issue Oct 5, 2016 · 9 comments

Comments

@nruth
Copy link
Contributor

nruth commented Oct 5, 2016

3 apps takes the slug size up towards 300MB, and it looks like most of this is in node_modules that shouldn't be necessary after the precompile step creates the javascript and css assets?

Poking around with heroku run bash and du is showing the node_modules directories as being around 200-500MB which will zip to some extent but shouldn't be necessary.

Is there a reason I've not thought of for keeping these around in the final dyno slug, or could we add something to delete the files from the slug once asset compilation succeeds?

There's something similar in the Heroku docs for Java.

@seanpdoyle
Copy link
Contributor

@nruth thanks for bringing this to our attention.

Since Dynos and their filesystems are ephemeral, we cache node_modules and bower_components to speed up future builds.

If it's possible to remove directories from the slug, but keep them in the cache, that would be ideal.

Are you interested in investigating that possibility?

@nruth
Copy link
Contributor Author

nruth commented Oct 5, 2016

Yep that sounds good. I'm currently looking at a build-pack solution and will report back if I get it working. I've found one that removes files after other buildpacks and am trying it out on a staging remote.

@seanpdoyle
Copy link
Contributor

@nruth is the simplest solution introducing a .slugignore file, or am I missing something?

@seanpdoyle
Copy link
Contributor

Ah, I AM missing something:

However, it is not uncommon for there to exist files in the repo that are necessary for the build, but are not required at runtime.

--Lostmyname

@nruth
Copy link
Contributor Author

nruth commented Oct 5, 2016

Yep it seems like slugignore is more for excluding psd or pdf files from docs or artwork directories rather than cleaning up compilation artefacts. Probably needs a V2.

The buildpacks give another way to do this, but it's broken and took some poking around to get a working build.

When it worked, it reduced my slug size from ~296MB to 68MB by deleting the 3 ember project directories. The apps still seem to work after this, which I assume is because the compiled assets are stored somewhere in the rails app's asset pipeline instead of in those subdirectories?

My specific configuration is 3 ember apps as subdirectories (specifically, git submodules with a machine-user key). I've changed the names but it basically looks like this:

ember-cli-rails initializer:

EmberCli.configure do |c|
  c.app :'admin-panel'
  c.app :'member-panel'
  c.app :'reports-dashboard'
end

heroku buildpacks:add --index 3 https://github.com/moravcik/heroku-buildpack-post-build-clean.git#8f7caa624c0d0f9102a6cc6ec47f363a66ea0020

$ heroku buildpacks
=== app Buildpack URLs
1. heroku/nodejs
2. heroku/ruby
3. https://github.com/moravcik/heroku-buildpack-post-build-clean.git#8f7caa624c0d0f9102a6cc6ec47f363a66ea0020

.slug-post-clean containing:

admin-panel
member-panel
reports-dashboard

where each is a subdirectory containing the full ember-cli project and all the leftover node_modules that were causing the problem.

I don't know if dependency caching is working for my app so can't give a definite answer there, but the slug size is down and from what I understood of the buildpack docs the cache of the node or ruby buildpacks shouldn't be affected by another buildpack?

Long-term we're going to move to a redis lightning-deploy type setup because heroku push is getting really really slow now, but short-term I just wanted to get the slug size away from the 300MB red line so we can continue deploying and updating our project without fear of heroku rejecting the changes.

@nruth
Copy link
Contributor Author

nruth commented Oct 5, 2016

I'm not 100% confident this is correct yet, but going to continue with it through to production and see how it goes. It seems simpler than changing the gem.

What do you think of wrapping this up by me making a readme PR for the Heroku deployment instructions that briefly describes the problem and links to this issue and the buildpack?

@seanpdoyle
Copy link
Contributor

What do you think of wrapping this up by me making a readme PR for the Heroku deployment instructions that briefly describes the problem and links to this issue and the buildpack?

Those gains are definitely worth mentioning in the README, maybe worth adding support to.

👍 Thanks for putting in the work!

nruth added a commit to nruth/ember-cli-rails that referenced this issue Oct 6, 2016
nruth added a commit to nruth/ember-cli-rails that referenced this issue Oct 6, 2016
nruth added a commit to nruth/ember-cli-rails that referenced this issue Oct 6, 2016
nruth added a commit to nruth/ember-cli-rails that referenced this issue Oct 6, 2016
@nruth
Copy link
Contributor Author

nruth commented Jan 12, 2017

Just to follow up on my previous hedging, we've been using this in production without issues since my original posts.

@thejchap
Copy link

Just an FYI for anyone else who stumbles on this...this doesn't work if you're using the include_ember_* helpers because ember-cli-rails-assets looks in those directories for a package.json (https://github.com/seanpdoyle/ember-cli-rails-assets/blob/master/lib/ember_cli/assets/paths.rb#L17).

Here's what my .slug-post-clean ended up looking like:

frontend/checkout/node_modules
frontend/checkout/bower_components
frontend/checkout/tmp

frontend/dashboard/node_modules
frontend/dashboard/bower_components
frontend/dashboard/tmp

Samsinite pushed a commit to Samsinite/ember-cli-rails that referenced this issue Mar 10, 2020
Closes tricknotes#491.

[Heroku slug size is limited](https://devcenter.heroku.com/articles/slug-compiler#slug-size). The build process creates artifacts that are not necessary for the server to run, but are included in the deployed Heroku slug.

Omitting these build assets can dramatically reduce slug size.

A build-pack solution for this is discussed in [Issue tricknotes#491][tricknotes#491].

[tricknotes#491]: tricknotes#491
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants