Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gemfile.lock not created for second project in Dockerfile #188

Closed
ShockwaveNN opened this issue Feb 2, 2018 · 17 comments · Fixed by #208
Closed

Gemfile.lock not created for second project in Dockerfile #188

ShockwaveNN opened this issue Feb 2, 2018 · 17 comments · Fixed by #208

Comments

@ShockwaveNN
Copy link

Hi, i notice some regression in update from ruby:2.4.1 to ruby:2.4.2
This dockerfile

FROM ruby:2.4.1

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

For ruby 2.4.1 it correctly show presence of /clipboard/Gemfile.lock
But if same Dockerimage created FROM ruby:2.4.2 this cause
ls: cannot access /clipboard/Gemfile.lock: No such file or directory

Probably somehow it related with update from bundler 1.15.4 to bundler 1.16.0 but even if I manully rollback to bundler 1.15.4 in ruby:2.4.2 error is the same.
This is actual for any project, i choose ruby-jwt and cliboard as they pretty small.
@gavinkflam
Copy link

I was suffered from this issue as well.

After hours of extensive research, it originates from an upstream issue of Bundler.

Please refer to rubygems/bundler#6154 and rubygems/bundler#6162.

===

The workaround provided by @lucasmazza is working.

You can add the environment variable BUNDLE_GEMFILE=./Gemfile to revert to the original behaviour.

@gavinkflam
Copy link

Should we introduce BUNDLE_GEMFILE=./Gemfile into all images with bundler ~1.16.0?

Or should we just wait for a fix in the upstream bundler?

@lucasmazza
Copy link

@gavinkflam thanks for mentioning this on this repository. Since we stumbled upon this on our app more and more I believe this should be fixed by removing the environment variables on the image rather than changing bundler - I believe that the image assumes too much of how the people are using the image and "installing" their apps with it.

@deivid-rodriguez
Copy link
Contributor

@lucasmazza I totally agree with the image being too opinionated. I think the whole block I mention here could probably be removed except for the BUNDLE_SILENCE_ROOT_WARNING=1, and use default bundler's values.

@lucasmazza
Copy link

@yosifkit any thoughts on trimming down the environment variables on the images?

@yosifkit
Copy link
Member

Current images with bundler 1.16.1 work fine with equivalent Dockerfile from above.

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ruby:2.4
 ---> dbe4f0ae7c72
Step 2/2 : RUN set -ex; 	git clone https://github.com/jwt/ruby-jwt.git; 	cd ruby-jwt; 		bundle install; 	cd ..; 	git clone https://github.com/janlelis/clipboard.git; 	cd clipboard; 		bundle install; 	cd ..; 	ls -la /clipboard/Gemfile.lock
 ---> Running in e5a0695ade91
+ git clone https://github.com/jwt/ruby-jwt.git
Cloning into 'ruby-jwt'...
+ cd ruby-jwt
+ bundle install
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.22
Installing ffi 1.9.22 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ git clone https://github.com/janlelis/clipboard.git
Cloning into 'clipboard'...
+ cd clipboard
+ bundle install
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ ls -la /clipboard/Gemfile.lock
-rw-r--r-- 1 root root 661 Feb 23 20:23 /clipboard/Gemfile.lock
Removing intermediate container e5a0695ade91
 ---> 9d12bfabeb9b
Successfully built 9d12bfabeb9b

@ShockwaveNN
Copy link
Author

Can confirm that issue is not actual for your variant of Dockerfile:

FROM ruby:2.4

RUN set -ex;\
    git clone https://github.com/jwt/ruby-jwt.git;\
    cd ruby-jwt;\
    bundle install;\
    cd ..;\
    git clone https://github.com/janlelis/clipboard.git;\
    cd clipboard;\
    bundle install;\
    ls -la /clipboard/Gemfile.lock

But issue still actual for me with with my original Dockerfile from first message

FROM ruby:2.4

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

I think issue should be reopened

@tianon
Copy link
Member

tianon commented Feb 26, 2018

I'm not able to reproduce with that exact Dockerfile: 😕

$ cat Dockerfile
FROM ruby:2.4

RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock

$ docker build --pull .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM ruby:2.4
2.4: Pulling from library/ruby
4176fe04cefe: Already exists 
851356ecf618: Already exists 
6115379c7b49: Already exists 
aaf7d781d601: Already exists 
acd6992c338a: Pull complete 
91d9490effb0: Pull complete 
1deabe842a08: Pull complete 
Digest: sha256:eeb289458b9025d9c6c598739ca88e0e0e92e3fa28b8140b0e611f7363961eab
Status: Downloaded newer image for ruby:2.4
 ---> 4b7621478e79
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
 ---> Running in 06f72f0071b9
Cloning into 'ruby-jwt'...
Removing intermediate container 06f72f0071b9
 ---> c8a5341e65b9
Step 3/6 : RUN cd ruby-jwt && bundle install
 ---> Running in 4be1b26aa2b6
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4be1b26aa2b6
 ---> 9e59a5707974
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
 ---> Running in 7dbf43405646
Cloning into 'clipboard'...
Removing intermediate container 7dbf43405646
 ---> 83cc647e79c0
Step 5/6 : RUN cd clipboard && bundle install
 ---> Running in b3a1b93b38f2
Using rake 12.3.0
Using bundler 1.16.1
Using docile 1.1.5
Using json 2.1.0
Using simplecov-html 0.10.2
Using simplecov 0.13.0
Using codacy-coverage 1.1.8
Using codeclimate-test-reporter 1.0.8
Using diff-lcs 1.3
Using ffi 1.9.23
Using jwt 2.1.0 from source at `.`
Using rbnacl 5.0.0
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Using simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container b3a1b93b38f2
 ---> 72c29d7ea676
Step 6/6 : CMD ls -la /clipboard/Gemfile.lock
 ---> Running in ebc57e5b2b3e
Removing intermediate container ebc57e5b2b3e
 ---> e9519654e875
Successfully built e9519654e875

@ShockwaveNN
Copy link
Author

@tianon Do you not forget to run your container? ls -la /clipboard/Gemfile.lock is CMD command, which execute only on docker run, not docker build

@tianon
Copy link
Member

tianon commented Feb 27, 2018

Oh bizarre -- something happens to the files when they get commit into a layer that breaks the second bundle install. There's nothing different between the two invocations except Docker's layering. 😕

@tianon
Copy link
Member

tianon commented Feb 27, 2018

For some reason I haven't figured out yet, when we run the second bundle install in a new layer, bundle env reports:

...
  Set via BUNDLE_GEMFILE: "/ruby-jwt/Gemfile"
...

... even though no such environment variable is or ever was set, and we're in the /clipboard directory, and it works fine if we run both bundle install commands in the same layer without round-tripping through Docker.

@tianon
Copy link
Member

tianon commented Feb 27, 2018

Ok, I've managed to reproduce with @yosifkit's single-layer image as well by adding hash -r in between the two bundle install invocations. What's happening here is that the image-provided /usr/local/bin/bundle script is getting shadowed by a new /usr/local/bundle/bin/bundle script (via binstubs) which is pointing at a specific Gemfile instead.

@lucasmazza
Copy link

@tianon as mentioned on previous comments, this isn't Docker fault by its own, but the combination of how this image configures Bundler (BUNDLE_BIN is always set) and upstream changes on Bundler (see rubygems/bundler#6154 and rubygems/bundler#6162).

@tianon
Copy link
Member

tianon commented Feb 27, 2018

Applying the following simple patch appears to do the trick:

-ENV PATH $BUNDLE_BIN:$PATH
+
+# bundler itself is in /usr/local/bin, and due to https://github.com/docker-library/ruby/issues/188 (and thus https://github.com/bundler/bundler/issues/6162) we need to ensure that we don't get a bundler-binstub'd version of bundler in the PATH before our global bundler (hence the explicit "/usr/local/bin" prefix on this PATH)
+ENV PATH /usr/local/bin:$BUNDLE_BIN:$PATH
+
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM f43db8c0ae0b
 ---> f43db8c0ae0b
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
 ---> Running in 605212cce70f
Cloning into 'ruby-jwt'...
Removing intermediate container 605212cce70f
 ---> b96e0830368a
Step 3/6 : RUN cd ruby-jwt && bundle install
 ---> Running in 1fe833dc0c17
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 1fe833dc0c17
 ---> 9fd2e01b9635
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
 ---> Running in bcda7be744b8
Cloning into 'clipboard'...
Removing intermediate container bcda7be744b8
 ---> 9930c870c723
Step 5/6 : RUN cd clipboard && bundle install
 ---> Running in 4db88df3ebff
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4db88df3ebff
 ---> 953376471d3b
Step 6/6 : RUN ls -la /clipboard/Gemfile.lock
 ---> Running in f7c7cfdd1c8f
-rw-r--r-- 1 root root 661 Feb 27 22:22 /clipboard/Gemfile.lock
Removing intermediate container f7c7cfdd1c8f
 ---> 1cd1366dee09
Successfully built 1cd1366dee09

@tianon tianon mentioned this issue Feb 27, 2018
Closed
@deivid-rodriguez
Copy link
Contributor

To me that patch, or #191, are workarounds to the same root problem: this image shouldn't be setting those environment variables and should instead leave the default bundler environment untouched.

@gavinkflam
Copy link

@deivid-rodriguez @lucasmazza @tianon
I agree that we should be generalizing the image, rather than keep on adding configurations and patching.

@tianon tianon mentioned this issue May 11, 2018
Merged
@yosifkit
Copy link
Member

Given the comment here: rubygems/bundler#6469 (comment). I think the best fix for the Docker image is this. It would keep it future-proof if the steps outlined there are followed.

-ENV PATH $BUNDLE_BIN:$PATH
+ENV PATH $BUNDLE_PATH/gems/bin:$PATH

I have successfully built the following Dockerfiles using this change that fail in the current ruby images:

FROM ruby:2.4-test
RUN set -ex;\
    git clone https://github.com/jwt/ruby-jwt.git;\
    cd ruby-jwt;\
    bundle install
RUN set -ex;\
    git clone https://github.com/janlelis/clipboard.git;\
    cd clipboard;\
    bundle install;\
    ls -la /clipboard/Gemfile.lock


FROM ruby:2.4-test
RUN set -ex;\
    git clone https://github.com/jwt/ruby-jwt.git;\
    cd ruby-jwt;\
    bundle install;\
    cd ..;\
    hash -r; \
    git clone https://github.com/janlelis/clipboard.git;\
    cd clipboard;\
    bundle install;\
    ls -la /clipboard/Gemfile.lock

If we do this, then we could probably revert or adjust the docs added in docker-library/docs#1221.

Could those having this issue try setting this at the top of their Dockerfile to see if it works for them?

# this is the PATH before it is adjusted by the ruby Dockerfile
ENV PATH $BUNDLE_PATH/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

@yosifkit yosifkit mentioned this issue May 15, 2018
Merged
@tianon tianon mentioned this issue May 21, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix PATH to use the fully generic binstubs infosiftr/ruby
6 participants
@lucasmazza @tianon @ShockwaveNN @yosifkit @deivid-rodriguez @gavinkflam