When running inside docker, mongoServer.getConnectionString() hangs #254

dotanrs · 2019-12-23T11:02:49Z

mongodb-memory-server: 6.1.1
node: v12.14.0

I started a docker container that runs node, and executed the following commands:

const MongodbMemoryServer = require('mongodb-memory-server').MongoMemoryServer;
const mongoServer = new MongodbMemoryServer();

(async () => {
  await mongoServer.getConnectionString();
  console.log('Done');
})();

Done is never printed, whereas outside docker this works fine

The text was updated successfully, but these errors were encountered:

hasezoey · 2019-12-23T11:05:30Z

please give information on what your system is (with version if linux distro) and your used docker image

PS: if your image is based on alpine, then sorry, mongodb dosnt have a build for alpine

PPS: i edited your issue's code to be better & faster to understand

dotanrs · 2019-12-23T11:08:52Z

I'm running on mac Mojave
the image is node:12.14

hasezoey · 2019-12-23T11:11:39Z

Information:
node:12.14 is based on debian-stretch

@dotanrs could you provide the output of new MongodbMemoryServer({ debug: true })?

dotanrs · 2019-12-23T12:01:00Z

> new MongodbMemoryServer({ debug: true })
Autostarting MongoDB instance...
Called MongoMemoryServer.start() method:
MongoMemoryServer {
  runningInstance: Promise { <pending> },
  instanceInfoSync: null,
  opts: { debug: true },
  debug: [Function]
}
> Starting MongoDB instance with following options: {"port":41817,"dbName":"cd3b60bd-55b7-46b9-ad32-d699110fb622","ip":"127.0.0.1","uri":"mongodb://127.0.0.1:41817/cd3b60bd-55b7-46b9-ad32-d699110fb622?","storageEngine":"ephemeralForTest","dbPath":"/tmp/mongo-mem-31V51vEtjsYLfb","tmpDir":{"name":"/tmp/mongo-mem-31V51vEtjsYLfb"}}
MongoBinary options: {"downloadDir":"/Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries","platform":"linux","arch":"x64","version":"4.0.3"}
MongoBinary: Mongod binary path: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod
Mongo[41817]: Called MongoInstance._launchKiller(parent: 31, child: 44):
MongoBinary: Download lock removed
Mongo[41817]: STDERR: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod: 2: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod: Syntax error: newline unexpected (expecting ")")

Mongo[41817]: CLOSE: 2
Mongo[41817]: [MongoKiller]: exit - [0,null]

dotanrs · 2019-12-24T11:33:21Z

Tried now with the ubuntu:latest image

$ lsb_release -d
Description: Ubuntu 18.04.2 LTS
$ node -v
v12.14.0
$ npm -v
6.13.4

Same result:

> new MongodbMemoryServer({ debug: true })
Autostarting MongoDB instance...
Called MongoMemoryServer.start() method:
MongoMemoryServer {
  runningInstance: Promise { <pending> },
  instanceInfoSync: null,
  opts: { debug: true },
  debug: [Function]
}
Starting MongoDB instance with following options: {"port":42483,"dbName":"d4d89c7e-7202-4636-b750-e2908419df85","ip":"127.0.0.1","uri":"mongodb://127.0.0.1:42483/d4d89c7e-7202-4636-b750-e2908419df85?","storageEngine":"ephemeralForTest","dbPath":"/tmp/mongo-mem-41901ewleTlosid2","tmpDir":{"name":"/tmp/mongo-mem-41901ewleTlosid2"}}
MongoBinary options: {"downloadDir":"/Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries","platform":"linux","arch":"x64","version":"4.0.3"}
MongoBinary: Mongod binary path: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod
Mongo[42483]: Called MongoInstance._launchKiller(parent: 4190, child: 4213):
Mongo[42483]: STDERR: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod: 2: /Users/dotanreis/src/igor/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.0.3/mongod: Syntax error: newline unexpected (expecting ")")

Mongo[42483]: CLOSE: 2
Mongo[42483]: [MongoKiller]: exit - [0,null]

hasezoey · 2019-12-24T12:21:21Z

@dotanrs because it is exactly the same, i need to ask: are you sure it is in the docker container?
and that the docker container has no linked directorys? (like to the code / node_modules)
-> i ask because the normal mongodb-memory-server package saves the binaries to ./node_modules/.cache/mongodb-binaries

dotanrs · 2019-12-24T14:49:44Z

I'm running in a container, I tried both locally and in Jenkins

However I did use a docker volume, and I guess that's why it happened.

re-installing mongodb-memory-server from inside the docker fixed it (although I had to do apt-get install libssl1.0.0 libssl-dev first)

Thanks for the help!

firstnoises · 2019-12-27T10:58:54Z

I have a similar problem and I believe I find the root cause.

My set up consist of a host with ubuntu, and a docker development environment with centos. I use the same set of source files by mounting it by the docker.

I run my IDE in ubuntu, with unit-test on save enabled. And occasionally I will do a full build inside the docker. However, mongodb-memory-server will only work in one of these two environment at one moment.

This is because the mongod downloaded will only work on one of these distro. If it was downloaded in ubuntu, the tests won't work in centos.

I suggest that appending lsb_release -si -sr to the binaries path would be a good solution to the problem. e.g. ./node_modules/.cache/mongodb-binaries/CentOS 7.4.1708/

hasezoey · 2019-12-27T11:02:46Z

@firstnoises

I suggest that appending lsb_release -si -sr to the binaries path would be a good solution to the problem. e.g. ./node_modules/.cache/mongodb-binaries/CentOS 7.4.1708/

no it wouldnt, because mongodb-memory-server checks if the file already exists when yes, dont download it again, and when you not use -global package, it will be installed into <project root>/node_modules/.cache/mongodb-binaries (global will install it to ~/.cache/mongodb-binaries

so, either install and use the -global package, or clear your node_modules/.cache every time, or dont mount it

firstnoises · 2019-12-27T11:16:47Z

no it wouldnt, because mongodb-memory-server checks if the file already exists when yes, dont download it again, and when you not use -global package, it will be installed into /node_modules/.cache/mongodb-binaries (global will install it to ~/.cache/mongodb-binaries

It will happens for my use case, as I am working on the same set of source files from two distro at the same time. This is very important for me, as my code interacts with other native codes that is distro dependent, so I have to run test in two distro after each code changes.

Yes, It will be solved by clearing my node_modules/.cache every time, by that will also means that I will need to redownload mongod hundreds of time a day!

hasezoey · 2019-12-27T11:18:54Z

@firstnoises like i said, use the mongodb-memory-server-global package to install it in ~/.cache (that isnt in your project folder), but otherwise, i think improvements could be done, like appending mongod-distro-version or something like that

firstnoises · 2019-12-27T11:20:24Z

That make sense, thanks

hasezoey · 2019-12-27T11:21:51Z

@firstnoises made a proposal at #256, will follow up with a PR when i have the time

nodkz · 2019-12-27T14:38:59Z

For those who are using MMS in docker I recommend to build a new image with MongoDB binaries and use env variable MONGOMS_SYSTEM_BINARY=/usr/local/bin/mongod with a proper path (or setup systemBinary property in your package.json)

alexbjorlig · 2020-02-12T19:08:20Z

Hi @nodkz . I am new to Docker, but trying to use MMS in Github Actions with this docker image with Cypress for running e2e tests. How would I build a new image with MongoDB binaries as you propose?

hasezoey · 2020-02-12T19:52:40Z

@alexbjorlig when you know how to make an docker image: (https://docs.docker.com/engine/reference/builder/)
(somewhere in there)

download the specified binary / copy it from host system
copy the config (package.json) to your image / set an environment variable pointing to the binary

example:

COPY /path/to/your/host/binary /usr/local/bin/mongod # COPY FROM TO

# Copy package.json
COPY /path/to/your/project/package.json /path/to/your/project/package.json # COPY FROM TO, probably already included somewhere when building your project
# OR
# set environment variable
ENV MONGOMS_SYSTEM_BINARY=/usr/local/bin/mongod # ENV VARIABLE=VALUE

alexbjorlig · 2020-02-13T07:29:15Z

@hasezoey thanks for the help, but I am unsure how/where to do this in a github actions? Currently my github-action config look like this:

jobs:
  e2eTests:
    runs-on: ubuntu-latest
    container: cypress/browsers:node13.6.0-chrome80-ff72

So my idea right now would be to create a new docker image, from the above image, but including the MMS binaries. To be honest I would prefer not having to maintain a docker image (or configuring docker at all 😁) - so if you have any other ideas for solving the issue please let me know 😊

Update

I added this docker file,

FROM cypress/browsers:node13.6.0-chrome80-ff72

RUN npm install mongodb-memory-server -g

ENV MONGOMS_SYSTEM_BINARY=~/.cache/mongodb-binaries

But I still get the this error when using the docker image in a Github action:

MongoBinary: can't find system binary at ~/.cache/mongodb-binaries. ENOENT: no such file or directory, access '~/.cache/mongodb-binaries'

hasezoey · 2020-02-13T13:46:48Z

@alexbjorlig if you need the global version you need to install mongo-memory-server-global

To be honest I would prefer not having to maintain a docker image (or configuring docker at all grin) - so if you have any other ideas for solving the issue please let me know blush

i know what you mean, but it is quite easy with docker's auto-build (on github pushes / on FROM changes)

alexbjorlig · 2020-02-13T15:44:11Z

@hasezoey thanks for helping out.

With this Docker file:

FROM cypress/browsers:node13.6.0-chrome80-ff72
RUN npm install mongodb-memory-server-global -g
ENV MONGOMS_SYSTEM_BINARY=~/.cache/mongodb-binaries

I still get this error:

console.log node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:81
2020-02-13T15:30:18.6014791Z     MongoBinary: can't find system binary at ~/.cache/mongodb-binaries. ENOENT: no such file or directory, access '~/.cache/mongodb-binaries'

hasezoey · 2020-02-13T16:35:30Z

@alexbjorlig you need to point MONGOMS_SYSTEM_BINARY to the binary, not the directory
btw maybe it installs into the wrong user

alexbjorlig · 2020-02-13T20:05:14Z

I published the container here. When I connect and look for the .cache, this is all I get (no .cache directory) 🤷🏽‍♂️

Where should I look for the binaries then?

hasezoey · 2020-02-13T20:47:51Z

@alexbjorlig just build it myself, the issue is easily identified - the system for your build is "debian10", which dosnt have an "4.0.3" build available, the earliest version i could find is "4.2.1"

alexbjorlig · 2020-02-13T20:50:49Z

@hasezoey thanks for trying it out - I'm trying to figure all this out 🤓What do you mean by "4.0.3" build? Build of what - is this something I can "fix" ?

hasezoey · 2020-02-13T20:54:05Z

@alexbjorlig the version "4.0.3" & "4.2.1" as in "version of mongodb binary" - you can fix this by one of those:

manually, download the wanted binary and add them in an seperate step and point to it
set an global config to download it (never used, so i dont know how to, sorry)
install the local version with "mongo-memory-server" (it will download to node_modules/.cache/) and you can set the config in either package.json or an separate file (i dont know the name at the moment) [Note: you need to ADD your project first, before installing this package]

update: you can set the global config file anywhere, but need to set the environment variable MONGOMS_ENV_CONFIG_PREFIX=/path/to/config

alexbjorlig · 2020-02-13T20:58:29Z

Ok, option 1 sounds like the best one. But I will have to read more about manually downloading the binary, I am unsure about how to do this. But thanks for trying to help me out, so nice of you 🙏🏽

hasezoey · 2020-02-13T21:08:04Z

But I will have to read more about manually downloading the binary, I am unsure about how to do this. But thanks for trying to help me out, so nice of you 🙏🏽

some hints:
to download go to either https://www.mongodb.org/dl/linux or https://www.mongodb.com/download-center/community and download your wanted version
extract it, search for "mongod" and run an ADD for this file
set an ENV pointing to the ADDed mongod

alexbjorlig · 2020-02-14T08:53:54Z

Hi @hasezoey - I think I'm getting close now 😅

My only problem seem to be the ADD command:

FROM cypress/browsers:node13.6.0-chrome80-ff72

# Download binary, debian10 v4.2 (-O emit outputs)
RUN wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-debian10-v4.2-latest.tgz

# Extract the downloaded archive
RUN tar -zxvf mongodb-linux-x86_64-debian10-v4.2-latest.tgz

# Remove the downloaded archive
RUN rm mongodb-linux-x86_64-debian10-v4.2-latest.tgz

# Rename extracted
RUN mv mongodb-* mongodb

# Add mongo binary to docker
ADD mongodb/bin/mongod mongod

# Point env variable to binary
ENV MONGOMS_SYSTEM_BINARY=./mongodb/bin/mongod

It fails with:
ADD failed: stat /var/lib/docker/tmp/docker-builder737024791/mongodb/bin/mongod: no such file or directory

But if I build the docker image without the ADD command, and run the container to inspect the files, the mongodb/bin/mongod is there?

hasezoey · 2020-02-14T09:14:54Z

@alexbjorlig simple, you use RUN which executes inside the image, and you try to ADD from outside the image

alexbjorlig · 2020-02-14T09:20:37Z

@alexbjorlig simple, you use RUN which executes inside the image, and you try to ADD from outside the image

Hmm ok - so does that mean I don't need this command at all `ADD mongodb/bin/mongod mongod?

hasezoey · 2020-02-14T09:44:49Z

@alexbjorlig yes, you dont need an ADD if you execute an RUN download, i assumed to download it on the host machine and add it later
PS: the rm command only add additional space, because the commands are layered, i would recommend combining them in to one RUN command

RUN wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-debian10-v4.2-latest.tgz \
# Extract the downloaded archive
         tar -zxvf mongodb-linux-x86_64-debian10-v4.2-latest.tgz \
# Remove the downloaded archive
         rm mongodb-linux-x86_64-debian10-v4.2-latest.tgz

alexbjorlig · 2020-02-14T10:06:16Z

@hasezoey amazing - it worked 🥳 😂 🎉!

The final docker image is this:

FROM cypress/browsers:node13.6.0-chrome80-ff72

# Download binary, extract and remove archive
RUN wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-debian10-v4.2-latest.tgz \
    && tar -zxvf mongodb-linux-x86_64-debian10-v4.2-latest.tgz \
    && rm mongodb-linux-x86_64-debian10-v4.2-latest.tgz

# Rename extracted
RUN mv mongodb-* /etc/mongodb

# Point env variable to binary
ENV MONGOMS_SYSTEM_BINARY=/etc/mongodb/bin/mongod

And when using this image inside a Github action, I get this good looking log statement 😎

Thanks for helping me out fixing this issue. If there is somehow I can return the favor, please let me know 🙃

jpike88 · 2020-03-02T11:49:53Z

Just upgraded to cypress/browsers:node12.16.1-chrome80-ff73 and this has broken my test builds using Codebuild.

hasezoey · 2020-09-07T07:04:00Z

closing because original issue was solved

4.0.3/mongod: Syntax error: newline unexpected (expecting ")")

which was caused by mounting the host's node_modules

The solution was taken from here: typegoose/mongodb-memory-server#254 (comment)

hasezoey added the Needs More Information label Dec 23, 2019

This comment has been minimized.

Sign in to view

hasezoey mentioned this issue Dec 27, 2019

Add distro and version to the mongod file #256

Closed

alexbjorlig mentioned this issue Feb 12, 2020

Cypress cy.screenshot() timeout during cypress run cypress-io/cypress#5016

Closed

hasezoey closed this as completed Sep 7, 2020

thatkookooguy added a commit to Kibibit/achievibit that referenced this issue Apr 12, 2021

build(devcontainer): support in-memory mongodb for tests

011c4ad

The solution was taken from here: typegoose/mongodb-memory-server#254 (comment)

thatkookooguy added a commit to Kibibit/achievibit that referenced this issue Apr 12, 2021

build(devcontainer): support in-memory mongodb for tests

54e2e33

The solution was taken from here: typegoose/mongodb-memory-server#254 (comment)

thatkookooguy added a commit to Kibibit/achievibit that referenced this issue Apr 18, 2021

build(devcontainer): support in-memory mongodb for tests

b1cad03

The solution was taken from here: typegoose/mongodb-memory-server#254 (comment)

thatkookooguy added a commit to Kibibit/achievibit that referenced this issue Apr 18, 2021

build(devcontainer): support in-memory mongodb for tests

5cda3f8

The solution was taken from here: typegoose/mongodb-memory-server#254 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When running inside docker, mongoServer.getConnectionString() hangs #254

When running inside docker, mongoServer.getConnectionString() hangs #254

dotanrs commented Dec 23, 2019 •

edited by hasezoey

Loading

hasezoey commented Dec 23, 2019 •

edited

Loading

dotanrs commented Dec 23, 2019

hasezoey commented Dec 23, 2019 •

edited

Loading

dotanrs commented Dec 23, 2019

This comment has been minimized.

dotanrs commented Dec 24, 2019 •

edited by hasezoey

Loading

hasezoey commented Dec 24, 2019

dotanrs commented Dec 24, 2019

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019 •

edited

Loading

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019

nodkz commented Dec 27, 2019

alexbjorlig commented Feb 12, 2020

hasezoey commented Feb 12, 2020

alexbjorlig commented Feb 13, 2020 •

edited

Loading

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020 •

edited

Loading

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 14, 2020

hasezoey commented Feb 14, 2020

alexbjorlig commented Feb 14, 2020

hasezoey commented Feb 14, 2020

alexbjorlig commented Feb 14, 2020

jpike88 commented Mar 2, 2020

hasezoey commented Sep 7, 2020

When running inside docker, mongoServer.getConnectionString() hangs #254

When running inside docker, mongoServer.getConnectionString() hangs #254

Comments

dotanrs commented Dec 23, 2019 • edited by hasezoey Loading

hasezoey commented Dec 23, 2019 • edited Loading

dotanrs commented Dec 23, 2019

hasezoey commented Dec 23, 2019 • edited Loading

dotanrs commented Dec 23, 2019

This comment has been minimized.

dotanrs commented Dec 24, 2019 • edited by hasezoey Loading

hasezoey commented Dec 24, 2019

dotanrs commented Dec 24, 2019

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019 • edited Loading

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019

firstnoises commented Dec 27, 2019

hasezoey commented Dec 27, 2019

nodkz commented Dec 27, 2019

alexbjorlig commented Feb 12, 2020

hasezoey commented Feb 12, 2020

alexbjorlig commented Feb 13, 2020 • edited Loading

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020 • edited Loading

alexbjorlig commented Feb 13, 2020

hasezoey commented Feb 13, 2020

alexbjorlig commented Feb 14, 2020

hasezoey commented Feb 14, 2020

alexbjorlig commented Feb 14, 2020

hasezoey commented Feb 14, 2020

alexbjorlig commented Feb 14, 2020

jpike88 commented Mar 2, 2020

hasezoey commented Sep 7, 2020

dotanrs commented Dec 23, 2019 •

edited by hasezoey

Loading

hasezoey commented Dec 23, 2019 •

edited

Loading

hasezoey commented Dec 23, 2019 •

edited

Loading

dotanrs commented Dec 24, 2019 •

edited by hasezoey

Loading

hasezoey commented Dec 27, 2019 •

edited

Loading

alexbjorlig commented Feb 13, 2020 •

edited

Loading

hasezoey commented Feb 13, 2020 •

edited

Loading