no automatic restart when using boot2docker docker volume

[chrishiestand]

nodemon doesn't seem to restart node when using a docker volume, or in this case a boot2docker ( os x -> virtualbox -> docker) volume. nodemon starts okay, but then fails to restart node when a watched file has been modified in the shared OS X volume.

root@760d5d2ff8cf:/src# nodemon --watch index.js 
20 Nov 21:06:58 - [nodemon] v1.2.1
20 Nov 21:06:58 - [nodemon] to restart at any time, enter `rs`
20 Nov 21:06:58 - [nodemon] watching: index.js
20 Nov 21:06:58 - [nodemon] starting `nodejs index.js`
Running on http://localhost:8080
request received
#index.js changed and saved here but output does not change as it should
request received
request received

nodemon -L causes restarts at least once a second even when no files have changed - way too frequently.

forever -w seems to work just fine. That seems like the best workaround for now.

output of dump:

root@b36ab2778133:/src# nodemon --dump
20 Nov 22:08:27 - [nodemon] v1.2.1
20 Nov 22:08:27 - [nodemon] to restart at any time, enter `rs`
20 Nov 22:08:27 - [nodemon] watching: *.*
--------------
node: v0.10.25
nodemon: v1.2.1
command: node /usr/local/bin/nodemon --dump
cwd: /src
OS: linux x64
--------------
{ run: false,
  system: 
   { cwd: '/src',
     useFind: false,
     useWatch: true,
     useWatchFile: false },
  required: false,
  dirs: [ '/src' ],
  timeout: 1000,
  options: 
   { scriptPosition: 0,
     dump: true,
     script: 'index.js',
     args: [],
     ignore: 
      [ '.git',
        'node_modules',
        'bower_components',
        '.sass-cache',
        re: /\.git|node_modules|bower_components|\.sass\-cache/ ],
     watch: [ '*.*', re: /.*\..*/ ],
     restartable: 'rs',
     execMap: { py: 'python', rb: 'ruby' },
     stdin: true,
     verbose: false,
     stdout: true,
     execOptions: 
      { script: 'index.js',
        exec: 'node',
        args: [],
        nodeArgs: undefined,
        ext: 'js',
        env: {},
        execArgs: [] },
     ext: 'js',
     monitor: 
      [ '*.*',
        '!.git',
        '!/src/node_modules/**/*',
        '!bower_components',
        '!.sass-cache' ] },
  load: [Function],
  reset: [Function: reset],
  lastStarted: 0,
  loaded: [],
  command: 
   { raw: { executable: 'node', args: [ 'index.js' ] },
     string: 'node index.js' } }

[chisleu]

When I use boot2docker with my branch off of library/node, when I try to use nodemon /path/to/app --watch, the cpu goes nuts like it is constantly restarting, although I can't verify.

I tried -L, but it also caused the CPU to go nuts.

I changed to pm2 for now, but it has 2-3 seconds of delay before it restarts. Still, better than nothing.

[tanis2000]

+1 this happens to me as well with a mounted volume on boot2docker (OSX + VirtualBox).

[remy]

I'm marking as "help wanted" - I don't have the infrastructure to even begin testing and debugging this.

I'm sure it's linked to clocks being out of whack somehow, but can't replicate myself with simple tests.

[tanis2000]

@remy if I can help you in any way, just let me know what I should do.

[remy]

@tanis2000 if you can pull the project inside of your virtual machine. I'd start by trying to discover why -L sets the CPU insane. The watching logic is all in watch.js - it's a little complicated because there's multiple methods to actually watch.

Really you want to get a handle on what's causing the issue in the first place. I usually console log the crap out of the code when I'm spelunking for errors.

[mattapperson]

still not seeing an automatic restart with a volume mounted nodejs project in b2d

[mrmurphy]

Remy, thanks for nodemon. Did you close this issue because you resolved it? Or did you not have enough helpful feedback to fix?

We've been experiencing the same problems as mentioned above.

[remy]

No, it looks like it was closed by mistake in the associated PR. I'll reopen and ask for help (from you lot!).

[remy]

If anyone wants to push a Docker image that I can test and debug this with – I'd be grateful.

[olalonde]

👍 same problem here

[remy]

@olalonde then same question to you: "[do you have] a Docker image that I can test and debug this with – I'd be grateful.".

[chisleu]

I will make one if someone else doesn't. I can't get to it until tonight though, which is why I had not yet replied.

Assuming Ubuntu 12.04 is ok unless I hear otherwise.

A faster solution would be to use vagrant. If you have vagrant and virtualbox, you can install a VM with

vagrant up ubuntu/precise64 --provider virtualbox
vagrant ssh

Then install node/npm and nodemon. Place a node file in a shared resource (such as /vagrant) and you should find these problems to be easy to reproduce.

I'm not sure about docker as I no longer use docker for development (because of this issue, and because I use MacOSX and the boot to docker image causes CPU and sleep problems for my little Macbook Air.)

It still affects my work in Vagrant and seems to have the same results.

BTW, PM2 suffers a similar fate now in Vagrant. It simply fails to restart at all.

[olalonde]

Ok, I just made a test but can't push it to Github at the moment.

Install boot2docker. Run $(boot2docker shellinit).

Put all the following files in the same directory (e.g. nodemontest).

./Dockerfile

FROM node:0.10-slim
RUN mkdir /app
WORKDIR /app

CMD ["./node_modules/nodemon/bin/nodemon.js", "index.js"]

./index.js

console.log(new Date());

./package.json

{
  "name": "testnodemon",
  "devDependencies": {
    "nodemon": "^1.2.1"
  }
}

Once, you have those files, run the following commands.

npm install;
docker build -t testnodemon .;
docker run -v $(pwd):/app testnodemon;

Edit index.js, nodemon doesn't restart the script.

You can use the following to get a shell in the container:

docker run -i -t -v $(pwd):/app testnodemon /bin/bash;
root@762c18df76ee:/app# mount | grep app
none on /app type vboxsf (rw,nodev,relatime)

Related? https://www.virtualbox.org/ticket/10660

[remy]

@olalonde thanks for this. I'll go through the steps this week to try to replicate. I'm sure it's linked the latest information in this issue #11 but I need to see it first hand.

[olalonde]

The problem I believe is that vboxsf doesn't support inotify notifications. The alternative would be to poll the directory but that could be end up eating all the CPU on large code bases. Another solution which I believe was adopted by Ruby's guard is to have a client on the host machine which sends restart messages to a another process running on the guest machine.

Anyways, I hope you can come up with something better :)

[chisleu]

It is such a problem to resolve that we started running node locally with a remote Cassandra DB (since C* would be the only service in docker.)

[olalonde]

Another solution would be to watch files normally on the host and whenever a file changes, write to .file-changed. The nodemon running in the container could just tail .file-changed and reload the process whenever it receives new data.

[olalonde]

I just submitted a PR with the solution I posted above. It's not elegant, but it's simple and it works.

Basically, nodemon needs to run on the host with the --local flag:

nodemon --local

and the VM/container can listen for file change events by using the --remote flag:

nodemon --remote index.js

The local nodemon communicates with the remote one by toggling a file called .nodemon-ping

[remy]

Can you try the development version of nodemon that I've pushed? It contains a lot of other tweaks, but the main thing I've changed is it'll use find instead of fs.watchFile (which I think uses the broken IO notify...).

To install:

npm install -g nodemon@dev

Can someone quickly give me an idea of whether that works or not? I tried it myself, but I'm still pretty new to docker so wasn't quite sure of what I was doing...

[remy]

(it should be [email protected])

[remy]

Nope, okay, I'm pretty sure [email protected] farts and makes a mess in docker. Anyone else want to confirm that's cool, otherwise I'll take this as a fail.

[aisipos]

I can confirm for myself at least using -L in 1.7.0 in a docker container is working for me. Without it, it doesn't work for me. (I'll note also, with some sadness, that while polling does work it does use a bit of CPU).

[chrishiestand]

@remy Thanks for your care on this issue. I've changed my workflow and haven't used this feature in some time. If other people have reported success with -L that is good enough for me. Thanks!

[pencilcheck]

-L works but the CPU will soar when run in boot2docker vm.

[remy]

@pencilcheck you should also be wary of the number of files its watching.

[d6u]

@remy -L works for me with docker-compose (previously boot2docker) version 1.4.2 on Node 0.10 with Nodemon 1.7.1

[pencilcheck]

In a VM -L is still needed it seems but I have switched to a different van that setup NFS instead of vboxfs, now it runs much much quieter but unfortunately writing speed is not as responsive as native

[thom4parisot]

-L works perfectly, thanks!

[akselnordal]

Nothing of the above worked for me, so I ended up creating a small bash-script that starts my servers, restarts the pm2 instance inside docker from the host OS and tails docker logs. To avoid zombie processes, I'vce included a trap that kills the docker processes started by docker-compose

#!/bin/bash

function clean_up {
  docker-compose kill
  exit
}

trap clean_up SIGHUP SIGINT SIGTERM

docker-compose build
docker-compose start

# tail all logs 
docker-compose logs&

# fswatch is available for linux and osx.
fswatch -0 src | while read -d "" event
  do
    echo "file changed: ${event}"
    image=$(docker ps | grep <docker app-name> | cut -d " " -f1 )
    docker exec -t $image ./node_modules/.bin/pm2 restart <pm2 app-name>
  done

[Omnomios]

I was chasing this issue for a while. Nodemon was catching the change no problem but node never received the signal to terminate. I was looking through the nodemon source and saw that there was an issue with child processes and as a result it was using psTree to get to the children.

The docker image I was using (fedora) didn't come with PS pre installed and so nodemon was using a fallback to pull the process that wasn't working for me.

Installing the 'procps' package in the dockerbuild got it working for me.

Hope this helps someone.

[thalesfsp]

So, what's the final solution and how to, to get nodemon watching for changes in Mac OS X -> Boot2Docker -> Docker Container? It's possible or not? There's some documentation/README.md/Wiki? @remy

[pyper]

@thalesfsp It looks as though there is a few issues possibly causing this. @Omnomios has done some next level bug chasing and found something. For me, I simply had to explicitly set --watch.

[altieres]

The best way I have found to have filesystem notifications visible within the container was as follows:
1- Create two folders, one to map the project and a "mirror"
2- Map the project in the first folder
3- Keep a background script running in the container, rsyncing the project folder to "mirror"
4- Run the project from "mirror"

It may not be the most efficient or most elegant way, but this way was transparent to users of the container. No additional script needs to be run.
Not tested in a larger project, but in my case I did not realize performance issues.

https://github.com/altieres/docker-jekyll-s3

[nettofarah]

I have similar solution to @Omnomios.
I wrote this bash script

#!/bin/bash

image=$(docker ps | grep <container_name> | cut -d " " -f1 )
docker exec -t $image pm2 restart server

and hooked it up to webpack in my gulpfile

gulp.task('watch-backend', function() {
  webpack(backendConfig).watch(100, function(err, stats) {
    onBuild(err, stats)
    restartServer()
  })
})

var exec = require('child_process').exec

function restartServer() {
  exec('./restart-container.sh', function(err, stdout, stderr) {
    if (stdout) {
      console.log('stdout: ' + stdout)
    }

    if (stderr) {
      console.log('stderr: ' + stderr)
    }

    if (err !== null) {
      console.log('exec error: ' + err)
    }
  })
}

It has been working pretty well for me.
You have to run webpack (or whatever other watcher) in the host machine though.

[darrinholst]

I cannot figure out how to get this working either. Shelling into the container and touching a file triggers nodemon...doing the same thing from osx doesn't do anything.

[darrinholst]

whoops...I missed the need for -L. Working once I added that.

[kirkstrobeck]

-L ftw ! 👍

[yordis]

I am using Docker for Mac and it doesn't work either but -L flag fixed it

[mshick]

Using a barebones security distro and was having this issue. @Omnomios solution installing procps worked for me.

[bradcosine]

Only -L fixed the issue for me

[IonelLupu]

Has anyone found a better solution, more official fix (maybe without any parameter)t o this -L parameter?

[mshick]

@jcwolf have you tried the solution that worked for me? procps? I haven't been using legacy mode for over a year now and get automatic restarts.

Here's a snippet of my Dockerfile

FROM node:8.9.4-wheezy
RUN apt-get update
RUN apt-get install -y procps make gcc g++ python libpq-dev

[remy]

@mshick would you be able to add this to the FAQ? Sounds like you've solved a problem that many people run in to! Thanks in advance

[mshick]

@remy Added a note. Credit goes to @Omnomios. Hope it helps!

[ms20hj]

a simple patch would be to change the defaultConnectionProperties.properties inside the jar file.

oracle.jdbc.timezoneAsRegion=false

defaultConnectionProperties.properties file path: oracle.jdbc，you can find it from ojdbc6.jar.
My oracle6.jar version 11.2.0.3