Yarn transition work-in-progress #2659

jywarren · 2018-04-26T20:30:55Z

PublicLabBot · 2018-04-26T20:39:10Z

	1 Message
📖	@jywarren Thank you for your pull request! I’m here to help with some tips and recommendations. Please take a look at the list provided and help us review and accept your contribution! And don’t be discouraged if you see errors – we’re here to help.

Generated by 🚫 Danger

jywarren · 2018-04-26T22:41:23Z

Running bower-away now

jywarren · 2018-04-27T12:20:29Z

travis-ci/travis-ci#6720

jywarren · 2018-04-27T12:29:28Z

Hmmm, Node version 0.10.29 is not supported, please use Node.js 4.0 or higher.

jywarren · 2018-04-27T12:29:46Z

@ryzokuken any ideas?

ryzokuken · 2018-04-28T12:37:37Z

@jywarren who throws this error? Yarn?

If we're on Node 0.x, we should definitely migrate. Node 4 has reached EOL, and ideally, we should jump to atleast Node 8.

jywarren · 2018-04-28T13:10:41Z

Yes I tried to set a higher node version... But it didn't work! See the issue I linked to above for yarn... Thanks!

…

On Sat, Apr 28, 2018, 8:38 AM Ujjwal Sharma ***@***.***> wrote: @jywarren <https://github.com/jywarren> who throws this error? Yarn? If we're on Node 0.x, we should definitely migrate. Node 4 has reached EOL, and ideally, we should jump to atleast Node 8. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ1ELF3PdhiVxx5oEbizxZNpe-2Ezks5ttGKSgaJpZM4TnkO4> .

ryzokuken · 2018-04-28T13:15:06Z

But travis allows you to use yarn. All you need to do is to check-in the yarn lockfile (which we should, and follow a policy similar to Gemfile.lock).

jywarren · 2018-04-28T16:53:27Z

Can you add a commit to this PR? Thanks!

…

On Sat, Apr 28, 2018, 9:15 AM Ujjwal Sharma ***@***.***> wrote: But travis allows you to use yarn. All you need to do is to check-in the yarn lockfile (which we should, and follow a policy similar to Gemfile.lock). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ69__zWTmN60NcPBTFTaz-TrCS3vks5ttGtagaJpZM4TnkO4> .

jywarren · 2018-04-29T21:47:15Z

Trying to do more research in https://blog.travis-ci.com/2016-11-21-travis-ci-now-supports-yarn and add a yarn.lock -- though don't know what that has to do with the node version... For now just trying to set node version more explicitly in .travis.yml -- and failing!

jywarren · 2018-04-29T21:52:34Z

Hmm, looks like nvm is installing node 6 outside the container but not inside?

jywarren · 2018-04-29T22:00:06Z

maybe something like this? https://stackoverflow.com/questions/25899912/install-nvm-in-docker

jywarren · 2018-04-29T23:11:16Z

OK, locally i get:

[1/1] ⡀ phantomjs: Try to install PhantomJS globally
[-/1] ⡀ waiting...
[-/1] ⡀ waiting...
[-/1] ⡀ waiting...
error /home/warren/sites/plots2/node_modules/phantomjs: Command failed.
Exit code: 1
Command: node install.js
Arguments:
Directory: /home/warren/sites/plots2/node_modules/phantomjs
Output:
Considering PhantomJS found at /usr/bin/phantomjs
Found PhantomJS at /usr/bin/phantomjs ...verifying
PhantomJS detected, but wrong version 2.1.1 @ /usr/bin/phantomjs.

jywarren · 2018-04-29T23:23:31Z

Getting closer -- i think we need help from @icarito maybe...

Step 9/23 : RUN rm /bin/sh && ln -s /bin/bash /bin/sh
 ---> Running in f85fe4dccf28
 ---> 6913a578b8d2
Removing intermediate container f85fe4dccf28
Step 10/23 : RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
 ---> Running in 4e881d19f410
 ---> b45713b2c669
Removing intermediate container 4e881d19f410
Step 11/23 : ENV NVM_DIR /usr/local/nvm # or ~/.nvm , depending
 ---> Running in c6392be267d2
 ---> 385ff8918de7
Removing intermediate container c6392be267d2
Step 12/23 : ENV NODE_VERSION 6
 ---> Running in edd3595d4c26
 ---> 98caca356cad
Removing intermediate container edd3595d4c26
Step 13/23 : RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.20.0/install.sh | bash     && . $NVM_DIR/nvm.sh     && nvm install $NODE_VERSION     && nvm alias default $NODE_VERSION     && nvm use default
 ---> Running in 333d9ab389ce
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5086  100  5086    0     0  45420      0 --:--:-- --:--:-- --:--:-- 45819
=> Downloading nvm from git to '/usr/local/nvm # or ~/.nvm , depending'
=> Cloning into '/usr/local/nvm # or ~/.nvm , depending'...
Note: checking out 'v0.20.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b new_branch_name
HEAD is now at 8964cb4... v0.20.0
=> Appending source string to /root/.bashrc
=> Close and reopen your terminal to start using nvm
/bin/sh: /usr/local/nvm: No such file or directory
Service 'web' failed to build: The command '/bin/sh -c curl https://raw.githubusercontent.com/creationix/nvm/v0.20.0/install.sh | bash     && . $NVM_DIR/nvm.sh     && nvm install $NODE_VERSION     && nvm alias default $NODE_VERSION     && nvm use default' returned a non-zero code: 1

icarito · 2018-04-30T00:11:23Z

I would rather we used prepackaged node rather than nvm if possible there are debian packages we can use or ideally a ready made base image. https://hub.docker.com/ is really slow for me but I'd look for an image with the right Node version and then install the ruby packages from the distro.

icarito · 2018-04-30T00:12:53Z

maybe we can switch to one of these: https://hub.docker.com/_/node/

jywarren · 2018-04-30T13:08:03Z

Hmm, interesting:

Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIES
section: 'hoe'

I don't see hoe in either Gemfile or Gemfile.lock...

jywarren · 2018-04-30T13:12:54Z

Bundler version issue maybe? rubygems/bundler#4467

https://stackoverflow.com/questions/34276324/travis-reports-odd-message-of-corrupted-gemfile-lock

icarito · 2018-05-01T03:37:43Z

Thanks for enduring build experiments with me.
So in Debian Stretch we have specific versions that aren't going to change. We could aim at locking into those - this could have the impact of reducing some of our update burden because we can generally depend on Debian to support packages etc and security. Or we can ride the wave and try to keep up with newest.
I found there's a Docker image called ruby-node that tries that last path so I was tempted to try it, but the "maintained version" whatever that means would bump Ruby to 2.5.

Thinking over this again I think the best approach is to return to Ruby base image as that is what ruby-node's Dockerfile does and install the desired Node version on top.

We really shouldn't think of nvm or rvm in containers as containers are disposable by nature and get rebuilt with the exact desired versions, generally no need for version managers except at build time.

jywarren · 2018-05-01T15:07:06Z

That's fine, i'm not wedded to nvm its just what we often use in development. I'm also fine with not being the latest node version. Honestly i'd just be happy to see the build succeed :-P it was a bit harder than I thought, given that the only issue remaining seemed to be the node version!

jywarren · 2018-05-24T16:30:52Z

@icarito -- any suggestions on this?

icarito · 2018-05-25T07:16:39Z

Sooner rather than later we will need to upgrade our container to Stretch.
In this process we can add the stretch-backports repo that serves v8.11.2 LTS.
That would be the simplest path forward. The manual way of installing it looks a bit convulted:
https://github.com/Starefossen/docker-ruby-node/blob/master/2-8/stretch/Dockerfile

jywarren · 2018-05-25T20:46:09Z

Sooner rather than later we will need to upgrade our container to Stretch.

OK, so let's try first opening a new PR for the Stretch port, then return to this. @ryzokuken if you can try this out, just be sure to link back to here so there's a next step.

jywarren · 2018-06-08T21:54:32Z

Seeing:

Node version 0.10.29 is not supported, please use Node.js 4.0 or higher.

ryzokuken · 2018-06-10T19:57:15Z

@jywarren makes sense, it's bad if we're at Node v0.10.x, let's switch to Node v10.x instead.

jywarren · 2018-06-10T20:44:37Z

I'm fine with that! If you can move this forward at all, I'd be very grateful. I'm just trying to move ahead with your excellent suggestion of moving to Yarn, and am stuck here. Happy to accept help from anyone who can get this to pass! Thanks!

…

On Sun, Jun 10, 2018 at 3:57 PM, Ujjwal Sharma ***@***.***> wrote: @jywarren <https://github.com/jywarren> makes sense, it's bad if we're at Node v0.10.x, let's switch to Node v10.x instead. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ7ysjEJQSavzgwBlrm8_JD8k6fBBks5t7XobgaJpZM4TnkO4> .

Souravirus · 2018-09-07T08:49:03Z

Hi @jywarren, @icarito now all the tests are passing here as well as the deployment to unstable branch is working fine. Please review this.

jywarren · 2018-09-07T14:30:08Z

This is great. 🎉 Can you give us a checklist on anything that has to be done on production for this to work? Or do you and @icarito anticipate that it'll run smoothly.

And one more thing - have you been able to double check that the most current bower.json file has been copied over (with accurate version numbers) to package.json?

Souravirus · 2018-09-07T15:16:54Z

Yeah I will check the bower.json file again to see if any libraries are missing. Other than that I have made it to work smoothly in production but it would be great if @icarito can review this once.

jywarren · 2018-09-07T16:14:19Z

super, awesome.

…

On Fri, Sep 7, 2018 at 11:17 AM Sourav Sahoo ***@***.***> wrote: Yeah I will check the bower.json file again to see if any libraries are missing. Other than that I have made it to work smoothly in production but it would be great if @icarito <https://github.com/icarito> can review this once. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ9YTx82qSb6WMsEKv7xl3a8d390dks5uYo3mgaJpZM4TnkO4> .

Souravirus · 2018-09-07T17:25:53Z

All the packages are according to bower.json except the leaflet-environmental-layers as yarn doesn't have 1.0.1 version of it. So, we have to take the 1.0.0 version of it.

icarito

Other than the build order, this looks fine to me! Congratulations, this is a long effort!
Thanks again.

icarito · 2018-09-08T06:24:15Z

.travis.yml

+#language: node_js
+#node_js: '6'
+#cache: yarn # This caches `$HOME/.yarn-cache`
+


Please remove these comments unless they help somehow!

Ok I wiil remove these

icarito · 2018-09-08T06:26:10Z

Makefile

 	docker-compose exec web rake assets:precompile
 	docker-compose exec web rake tmp:cache:clear
 	docker-compose down --remove-orphans
 	rm -f ./tmp/pids/server.pid
 	docker-compose up -d
+	docker-compose exec -T web yarn --ignore-engines --ignore-scripts --modules-folder public/lib 


You've removed the javascript package installation from line 10/11 and put it into line 15. This effectively waits until after rebooting the Rails app to get going with Javascript installation. So if Javascript installation fails, you've already restarted the app possibly broken by missing Javascript dependencies. If you move this line to line 10, then if yarn fails the app is still running fine and no harm was done. If you agree please do so! Thanks!

Ok I will move the yarn installation command to line 11. Thanks

icarito · 2018-09-08T06:26:51Z

Makefile

 	docker-compose run web rake assets:precompile
 	rm -f ./tmp/pids/server.pid
 	docker-compose up -d
+	docker-compose exec -T web yarn --ignore-engines --ignore-scripts --modules-folder public/lib 


Again here, you wait till after deploying the container to install Javascript packages, I think the same applies here. Thanks!

icarito · 2018-09-08T15:16:09Z

@jywarren This morning it came to me that you may find it won't deploy cleanly with redeploy-container method (because in the first part of the process it'll exec yarn but yarn isn't installed in the current image). I guess this is why @Souravirus put this command after restarting the container (because then yarn will be available).
In these cases (like when we upgrade ruby) - when redeploy-container fails, our Makefile has deploy-container task which we can use to deploy from a stopped container status.

Souravirus · 2018-09-08T17:13:44Z

Yeah right @icarito putting yarn before restarting the container is giving error as you can see in https://jenkins.laboratoriopublico.org/job/Plots-Unstable/296/console . So, I guess best is to use the yarn command after starting the container. What do you think @icarito.

icarito · 2018-09-10T06:20:19Z

Thanks @Souravirus for giving it a try! I think this is pinpointing an issue we've been having with deployment to production being somewhat unstable when upgrading stuff like Ruby. We've been using docker-compose exec. I think we should be using docker-compose run --rm. I'll issue commit for this and try it!

icarito · 2018-09-10T06:53:23Z

I think this is working fine in unstable now! @jywarren and the docker-compose bit should make it reliable to upgrade onwards.

jywarren · 2018-09-10T15:10:16Z

OK, shall I merge this in then? SUPER!!!

Souravirus · 2018-09-10T15:21:09Z

Yeah its ready @jywarren

jywarren · 2018-09-10T15:22:12Z

🎉 !!!

jywarren · 2018-09-11T21:54:53Z

This is live!!!!!

Souravirus · 2018-09-11T22:03:33Z

We have to change the documentation for the change from bower to yarn. I guess making a first-timer issue of it will work.

Souravirus · 2018-09-11T22:07:09Z

I guess we can make it a code-in task too. What do you think @jywarren

jywarren · 2018-09-11T22:08:45Z

That'd be great!!!

…

On Tue, Sep 11, 2018 at 6:07 PM Sourav Sahoo ***@***.***> wrote: I guess we can make it a code-in task too. What do you think @jywarren <https://github.com/jywarren> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2659 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ27_B3i3Q_i7nkPVQmPkrfyCLY-Rks5uaDQNgaJpZM4TnkO4> .

* Yarn transition work-in-progress (help-wanted) * switch to yarn attempt * changed root location of packages * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update Dockerfile * Update .travis.yml * Update Dockerfile * Update Dockerfile * Update Dockerfile * Remove reference to unneeeded installations Keep minimal. * Still more tweaking for switching base OS image * Add .dockerignore Trying with ignored Gemfile.lock * Update Dockerfile * Update Dockerfile * Update Dockerfile * Added yarn.lock * Added phantomjs-prebuilt to package.json and yarn.lock * Removed separated installation of phantomjs-prebuilt * Run forced installation of yarn * Removed bower.json and .bowercc * Added noty and jquery-confirm * Modifications in Dockerfile * Added yarn install in .travis.yml * Added lel packages to package.json * Added yarn command in .gitlab.yml * Deleted bower command from .gitlab-ci.yml * Changes in .gitlab-ci.yml * Updated yarn according to bower * Deprecated lel version * Changes in Makefile * Removed bower from Makefile * Added rack-test to Gemfile * Modified Dockerfile according to current version * Changes in Gemfile.lock related to ruby 2.4.4 * Makefile changes related to ruby 2.4.4 * Removed bundle install command in .travis.yml * Added i18n Gem to Gemfile * Changes in Dockerfile related to npm error Hostname/IP doesn't match certificate's altnames * Changes in Dockerfile related to jobs option of bundle install * Removed Gemfile.lock in .dockerignore * Changes in .dockerignore * Changes in .dockerignore * Added Gemfile.lock in Dockerfile * Updated Gemfile.lock * Changes in .dockerignore * Modified way of installing bundler * Updated Dockerfile * Some changes in Dockerfile * Some changes * Changes in package.json * Removed comments in .travis.yml * Redeploy independent container * Use bash command line

jywarren added the in progress label Apr 26, 2018

jywarren mentioned this pull request Apr 26, 2018

Switch from Bower to Yarn #1215

Closed

jywarren assigned icarito Apr 30, 2018

ghost assigned jywarren Jun 8, 2018

Changes in package.json

b493d8e

icarito requested changes Sep 8, 2018

View reviewed changes

Removed comments in .travis.yml

903c330

Souravirus force-pushed the yarn branch from 422a4ba to 903c330 Compare September 8, 2018 21:02

Redeploy independent container

d6cd975

ghost assigned icarito Sep 10, 2018

Use bash command line

e311651

icarito changed the title ~~Yarn transition work-in-progress (help-wanted)~~ Yarn transition work-in-progress Sep 10, 2018

icarito approved these changes Sep 10, 2018

View reviewed changes

jywarren merged commit 5c2bce3 into master Sep 10, 2018

ghost removed the in progress label Sep 10, 2018

Souravirus deleted the yarn branch September 11, 2018 22:02

jywarren mentioned this pull request Oct 24, 2018

Yarn causing to fail to build Docker container (ENOENT lstat) #3727

Closed

Yarn transition work-in-progress #2659

Yarn transition work-in-progress #2659

Conversation

jywarren commented Apr 26, 2018 • edited Loading

PublicLabBot commented Apr 26, 2018 • edited by plotsbot Loading

jywarren commented Apr 26, 2018

jywarren commented Apr 27, 2018

jywarren commented Apr 27, 2018

jywarren commented Apr 27, 2018

ryzokuken commented Apr 28, 2018

jywarren commented Apr 28, 2018 via email

ryzokuken commented Apr 28, 2018

jywarren commented Apr 28, 2018 via email

jywarren commented Apr 29, 2018

jywarren commented Apr 29, 2018

jywarren commented Apr 29, 2018

jywarren commented Apr 29, 2018

jywarren commented Apr 29, 2018

icarito commented Apr 30, 2018

icarito commented Apr 30, 2018

jywarren commented Apr 30, 2018 • edited Loading

jywarren commented Apr 30, 2018

icarito commented May 1, 2018

jywarren commented May 1, 2018 • edited Loading

jywarren commented May 24, 2018

icarito commented May 25, 2018

jywarren commented May 25, 2018

jywarren commented Jun 8, 2018

ryzokuken commented Jun 10, 2018

jywarren commented Jun 10, 2018 via email

Souravirus commented Sep 7, 2018

jywarren commented Sep 7, 2018

Souravirus commented Sep 7, 2018

jywarren commented Sep 7, 2018 via email

Souravirus commented Sep 7, 2018

icarito left a comment

Choose a reason for hiding this comment

icarito Sep 8, 2018

Choose a reason for hiding this comment

Souravirus Sep 8, 2018

Choose a reason for hiding this comment

icarito Sep 8, 2018

Choose a reason for hiding this comment

Souravirus Sep 8, 2018

Choose a reason for hiding this comment

icarito Sep 8, 2018

Choose a reason for hiding this comment

icarito commented Sep 8, 2018

Souravirus commented Sep 8, 2018

icarito commented Sep 10, 2018

icarito commented Sep 10, 2018

jywarren commented Sep 10, 2018

Souravirus commented Sep 10, 2018

jywarren commented Sep 10, 2018

jywarren commented Sep 11, 2018

Souravirus commented Sep 11, 2018

Souravirus commented Sep 11, 2018

jywarren commented Sep 11, 2018 via email

jywarren commented Apr 26, 2018 •

edited

Loading

PublicLabBot commented Apr 26, 2018 •

edited by plotsbot

Loading

jywarren commented Apr 30, 2018 •

edited

Loading

jywarren commented May 1, 2018 •

edited

Loading