Bundler: freeze lockfile on run, and "normalize" platform on plugin changes #13015

robbavey · 2021-06-21T15:28:47Z

This PR enables the upgrade of bundler to the latest version.

Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > 2.23 making runtime changes to Gemfile.lock (unless the lock file was frozen) based on the specific platform the application was being run on, overriding any platforms (including generic java platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in /usr/share/logstash.

This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic java platform, and not changed to the specific platform for the runtime JVM.

This PR also introduces a new runtime flag, --enable-local-plugin-development. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg

gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka'

to facilitate quick and simple plugin testing.

This PR also sets the silence_root_warning flag to avoid bundler printing out alarming looking warning messages when sudo is used. This warning message was concerning for users - it would be printed out during normal operation of bin/logstash-plugin install/update/remove when run under sudo, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages.

This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests.

Release notes

Updated Bundler to latest version
Ensured that Gemfile.lock are appropriately frozen
Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

How to test this PR locally

Ensure that logstash runs after install, as stand-alone, deb, rpm
Ensure that plugin installation, upgrade, removal and offline-pack creation and installation works, and that logstash runs correctly after each of these operations

The vagrant tests have been updated to include these tests for debian and rpm across multiple flavours of those distributions:

https://logstash-ci.elastic.co/job/elastic+logstash+master+multijob-acceptance/278/

Related issues

Relates: #12958, #12910, #12818.

yaauie

First pass:

Two things aren't clear to me:

What is the new --dev flag that we are adding to the logstash process, and when does it need to be set? I see that it effectively causes bundler to not freeze the lockfile, but I don't understand when it needs to get set and why. I suspect that a more descriptive flag or an environment variable may be a more discoverable path forward, and that we should also document this as a part of the "how to develop Logstash" tutorials.
We moved around a lot of silence_root_warning bits, removing a default from our invoke!, explicitly passing it to invoke! in a lot of places, and using its effective value to set an ENV variable that presumably the underlying bundler uses. Can you explain why we did what we did?

yaauie · 2021-07-26T16:34:00Z

lib/bootstrap/bundler.rb

+    def specific_platform
+      ::Gem.platforms.find {|plat| plat.is_a?(::Gem::Platform) && plat.os=='java' && !plat.cpu.nil?}
+    end
+
+    def genericize_platform
+      output = LogStash::Bundler.invoke!({:add_platform => 'java', :silence_root_warning => true})
+      remove_platform_options = {:remove_platform => specific_platform.to_s, :silence_root_warning => true} unless specific_platform.nil?
+      output << LogStash::Bundler.invoke!(remove_platform_options) unless remove_platform_options.nil?
+    end


While not possible directly with our tooling as-is, it is possible to end up with a Gemfile.lock with more than one specific platform. Since we effectively have a zero-or-more situation, I would prefer to iterate over an enumberable.

Suggested change

def specific_platform

::Gem.platforms.find {|plat| plat.is_a?(::Gem::Platform) && plat.os=='java' && !plat.cpu.nil?}

end

def genericize_platform

output = LogStash::Bundler.invoke!({:add_platform => 'java', :silence_root_warning => true})

remove_platform_options = {:remove_platform => specific_platform.to_s, :silence_root_warning => true} unless specific_platform.nil?

output << LogStash::Bundler.invoke!(remove_platform_options) unless remove_platform_options.nil?

end

def specific_platforms

::Gem.platforms.select {|plat| plat.is_a?(::Gem::Platform) && plat.os=='java' && !plat.cpu.nil?}

end

def genericize_platform

output = LogStash::Bundler.invoke!({:add_platform => 'java', :silence_root_warning => true})

specific_platform.each do |platform_to_remove|

output << LogStash::Bundler.invoke!({:remove_platform => platform_to_remove.to_s, :silence_root_warning => true})

end

end

lib/pluginmanager/bundler/logstash_injector.rb

robbavey · 2021-07-26T17:26:25Z

@yauuie Having the concept of a dev mode/thawed mode enables a logstash developer to make changes to the Gemfile and have them picked up at runtime. An example would be to use a development plugin - prior to this change it was possible to add , :path => '/path/to/plugin' and have it picked up automatically and used, which can be useful when testing plugins. Having the dev flag allowed this to still be possible. It also provides an "escape hatch" if a user somehow manages to find themselves broken and frozen.

I to'ed and fro'ed on the dev flag, what the right name for it was (thaw, unfreeze?), whether an ENV would make sense and whether it was worth including at all, and I'm happy to take suggestions - its one reason why I hadn't documented it yet...

The silence_root_warning is set to avoid this error, which is output during any interaction with bin/logstash-plugin when used with sudo, which is required in a deb/rpm environment.:

Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.

As far as using the ENV, I could probably do with investing some more effort to see if that is avoidable - I'd run into somewhat of a brick wall trying to get the setting to take effect and that did the trick.

removing a default from our invoke!,
Which default did I remove?

yaauie

I will give more thought to the --dev flag. I worry that a short flag that is very easy to type will lend itself to becoming overloaded or divorced from its original intent.

In my mind it is a rare feature that doesn't need to be frequently typed, and would benefit from having a longer, more-descriptive name. Maybe --enable-local-plugin-development? When this flag is not enabled, and a user tries to use a local path-defined plugin, can we reliably detect this and suggest adding the flag?

yaauie · 2021-07-26T19:20:26Z

lib/bootstrap/bundler.rb

@@ -91,7 +91,7 @@ def setup!(options = {})
    # @option options [Array] :without  Exclude gems that are part of the specified named group (default: [:development])
    # @return [String, Exception] the installation captured output and any raised exception or nil if none
    def invoke!(options = {})
-      options = {:max_tries => 10, :clean => false, :install => false, :update => false, :local => false,
+      options = {:silence_root_warning => false, :max_tries => 10, :clean => false, :install => false, :update => false, :local => false,


This is where your diff effectively removes the default value, since our options becomes the result of merging the given options onto the hard-coded hash.

🤦🏼 this doesn't remove a default. How did I read it that way?

I've removed the variable, so it is set across the board anyway - I don't think there is a place that seeing the bundler error would be useful or actionable for users, so I'm not sure what value not silencing it would bring

robbavey · 2021-07-27T21:43:06Z

jenkins test this please

robbavey · 2021-07-29T17:33:27Z

@yaauie Ready for another round, I think

yaauie

The bundler changes look great.

I've left a couple comments about reducing repetition in specs by using a helper method, and noted a lack of clarity in changes to the QA matrix and supporting code.

yaauie · 2021-07-29T20:28:13Z

lib/bootstrap/bundler.rb

+      # to update plugins when logstash is run as a service
+      # Note: Using an `ENV` here, as ::Bundler.settings.set_local(:silence_root_warning, true)
+      # does not work (set_global *does*, but that seems too drastic a change)
+      ENV["BUNDLE_SILENCE_ROOT_WARNING"] = "true"


Truly a nitpick, since we exit the process before doing anything else that will be affected by this, but it may be good to clean up after ourselves so we don't pollute the ENV for longer than we need to.

def with_env(modifications) backup_env = ENV.to_hash ENV.replace(backup_env.merge(modifications)) yield ensure ENV.replace(backup_env) end

Which would allow us to do:

with_env("BUNDLE_SILENCE_ROOT_WARNING" => "true") do if !debug? # Will deal with transient network errors execute_bundler_with_retry(options) else options[:verbose] = true execute_bundler(options) end end

Yeah, I like that. Not a fan of unintended side effects. Even if it's not a problem now, its the kind of subtle thing that can bite later

yaauie · 2021-07-29T20:31:41Z

lib/pluginmanager/bundler/logstash_injector.rb

@@ -93,6 +93,11 @@ def inject(gemfile_path, lockfile_path, dependencies)

          builder.eval_gemfile("bundler file", gemfile.generate())
          definition = builder.to_definition(lockfile_path, {})
+          # Remove the specific platform, and leave the generic platform
+          LogStash::Bundler.specific_platforms.each do |specific_platform|


🤔 is there a reason we don't just invoke LogStash::Bundler::genericise_platform here? Is it possible that only removing the specific platforms and not adding a generic platform could leave us with no platforms?

This particular code path is only used during the offline pack installation, and doesn't call out to the command-line bundler and the invoke! method that the genericize_platform uses. However, I have made a change here to call the specific_platforms filter on the platforms returned from definition.platforms, rather than making a call to Gem::platforms

yaauie · 2021-07-29T20:38:27Z

logstash-core/locales/en.yml

+          Allow Gemfile to be manipulated directly to facilitate simpler local plugin development.
+          This is an advanced setting, intended only for use by logstash developers, and should
+          not be used in production.


Since this is included in command-line --help output that doesn't play nicely with wrapping, we should reformat to keep the individual lines short (<40char).

Suggested change

Allow Gemfile to be manipulated directly to facilitate simpler local plugin development.

This is an advanced setting, intended only for use by logstash developers, and should

not be used in production.

Allow Gemfile to be manipulated directly

to facilitate simpler local plugin

development.

This is an advanced setting, intended

only for use by Logstash developers,

and should not be used in production.

yaauie · 2021-07-29T20:40:51Z

logstash-core/src/main/java/org/logstash/Logstash.java

+    private static final String MUTATED_GEMFILE_ERROR = "Logstash was unable to start due to an unexpected Gemfile change.\n" +
+            "If you are a user, this is a bug.\n" +
+            "If you are a logstash developer, please try restarting logstash with the " +
+            "`--enable-local-plugin-development` flag set.";


👍🏼 for useful only-when-needed guidance

yaauie · 2021-07-29T20:44:22Z

qa/acceptance/spec/shared_examples/cli/logstash-plugin/install.rb

+            logstash.start_service
+            Stud.try(40.times, RSpec::Expectations::ExpectationNotMetError) do
+              expect(logstash).to be_running
+            end
+            logstash.stop_service


here, we are adding a secondary validation that Logstash can start up after performing the primary task successfully.

Since we are doing it repeatedly, I think we should break this out into a helper method.

yaauie · 2021-07-29T20:47:14Z

qa/acceptance/spec/shared_examples/cli/logstash-plugin/update.rb

@@ -33,23 +33,39 @@

    before do
      logstash.run_command_in_path("bin/logstash-plugin install --no-verify --version #{previous_version} #{plugin_name}")
+      logstash.run_command_in_path("bin/logstash-plugin list")
      # Logstash won't update when we have a pinned version in the gemfile so we remove them


this comment appears to go with a line that was moved.

It is not clear to me why we need to do a gemfile replacement to un-pin, and whether that is supposed to affect the result of the have_installed validation that is now occurring before the un-pinning.

I've moved the comment around.

It's a pretty hacky test. We install a named, older version of the test plugin, and do a have_installed check to ensure the pre-condition that the plugin has installs as expected. The sed ugliness that is done on the Gemfile is to remove the pin - bin/logstash-plugin update will not upgrade pinned plugins (in this case a plugin that has been installed at a specific version). Whether that is desired behaviour, or whether it is something we should allow to be bypassed via a --force option is a valid question

qa/rspec/commands.rb

yaauie · 2021-07-29T20:51:08Z

qa/config/platforms.json

@@ -6,12 +6,8 @@
    "centos-7": { "box": "elastic/centos-7-x86_64", "type": "redhat" },
    "oel-6": { "box": "elastic/oraclelinux-6-x86_64", "type": "redhat" },
    "oel-7": { "box": "elastic/oraclelinux-7-x86_64", "type": "redhat" },
-    "fedora-28": { "box": "elastic/fedora-28-x86_64", "type": "redhat", "experimental": true },


are these changes to drop old platforms in scope? If so, I'd like to know why. I would assume that we would merely include later versions, instead of dropping these entirely.

They're not. I'll reinstate them, and bring the removal up in a separate PR, or at least spark a discussion as to whether we want to keep them - they're not run in CI right now, and I was trying to figure it if it was the cause behind test failures. (Actually hashicorp/vagrant#12265 is the cause)

robbavey · 2021-07-30T17:41:33Z

Adding @karenzone for doc update review

robbavey · 2021-08-16T15:28:50Z

@yaauie Ready for another round - the CI failures look unrelated, see: #13136

yaauie

I'm largely comfortable merging as-is. This is wide-ranging, but sensible and merely a side-effect of us hooking bundler in such a way.

I have left a few nitpicks and also have a question about needing to ensure that a generic java is in the platform list inside the "logstash injector" when we are removing the specific javas.

lib/pluginmanager/bundler/logstash_injector.rb

lib/pluginmanager/install.rb

yaauie · 2021-08-16T16:47:47Z

qa/acceptance/spec/spec_helper.rb

+    Stud.try(40.times, RSpec::Expectations::ExpectationNotMetError) do
+      yield
+    end


This certainly works for all of the specs you have introduced in this PR, but it may prove to be a mismatch when we introduce other specs that check other things with the running service.

It may be helpful to only yield once when we know that logstash is actually running. I know this double-checks in the cases of our current use, but it also ensures that we don't retry other expectations 40x in this helper.

Suggested change

Stud.try(40.times, RSpec::Expectations::ExpectationNotMetError) do

yield

end

Stud.try(40.times, RSpec::Expectations::ExpectationNotMetError) do

expect(logstash).to be_running

end

yield

Your suggestion didn't quite work - expect(logstash).to be_running doesn't work when a specific jdk_path is used, but I've added the equivalent to the helper method

qa/docker/spec/spec_helper.rb

Co-authored-by: Ry Biesemeyer <[email protected]>

robbavey · 2021-08-16T20:27:37Z

@yaauie Ready for another round, integration tests running at https://logstash-ci.elastic.co/job/elastic+logstash+master+multijob-acceptance/284/

yaauie

LGTM.

Thanks for sticking through all the bonus review cycles ❤️

…malize" platform on plugin changes Backport PR #13015 to 7.15 branch. Original Message: This PR enables the upgrade of bundler to the latest version. Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`. This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM. This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg ``` gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka' ``` to facilitate quick and simple plugin testing. This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages. This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests. * Updated Bundler to latest version * Ensured that `Gemfile.lock` are appropriately frozen * Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment (cherry picked from commit 4707cb)

…d "normalize" platform on plugin changes Backport PR elastic#13015 to 7.x branch. Original Message: This PR enables the upgrade of bundler to the latest version. Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in elastic#12782, which prevented the logstash user writing to files in `/usr/share/logstash`. This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM. This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg ``` gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka' ``` to facilitate quick and simple plugin testing. This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages. This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests. * Updated Bundler to latest version * Ensured that `Gemfile.lock` are appropriately frozen * Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment (cherry picked from commit 4707cb)

…nd "normalize" platform on plugin changes Backport PR elastic#13015 to 7.15 branch. Original Message: This PR enables the upgrade of bundler to the latest version. Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in elastic#12782, which prevented the logstash user writing to files in `/usr/share/logstash`. This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM. This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg ``` gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka' ``` to facilitate quick and simple plugin testing. This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages. This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests. * Updated Bundler to latest version * Ensured that `Gemfile.lock` are appropriately frozen * Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment (cherry picked from commit 4707cb)

#13141) * Backport PR #13015 to 7.15: Bundler: freeze lockfile on run, and "normalize" platform on plugin changes Backport PR #13015 to 7.15 branch. Original Message: This PR enables the upgrade of bundler to the latest version. Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`. This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM. This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg ``` gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka' ``` to facilitate quick and simple plugin testing. This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages. This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests. * Updated Bundler to latest version * Ensured that `Gemfile.lock` are appropriately frozen * Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment (cherry picked from commit 4707cb) * Remove code pinning bundler to `~> 1.17` Also updates lockfile to update `BUNDLED WITH` to latest

#13140) * Backport PR #13015 to 7.x: Bundler: freeze lockfile on run, and "normalize" platform on plugin changes Backport PR #13015 to 7.x branch. Original Message: This PR enables the upgrade of bundler to the latest version. Prior to this PR, the ability to do so was blocked by bundler.setup in versions of bundler > `2.23` making runtime changes to `Gemfile.lock` (unless the lock file was `frozen`) based on the specific platform the application was being run on, overriding any platforms (including generic `java` platform) set during build time. This was in conflict with changes made in #12782, which prevented the logstash user writing to files in `/usr/share/logstash`. This PR will freeze the lockfile when logstash is run, and unfreeze it when manipulating plugins (install, update, remove, install from offline pack) to allow new plugins to be added. While unfrozen, changes are also made to ensure that the platform list remains as the generic `java` platform, and not changed to the specific platform for the runtime JVM. This PR also introduces a new runtime flag, `--enable-local-plugin-development`. This flag is intended for use by Logstash developers only, and enables a mode of operation where a Gemfile can be manipulated, eg ``` gem "logstash-integration-kafka", :path => '/users/developer/code/plugins/logstash-integration-kafka' ``` to facilitate quick and simple plugin testing. This PR also sets the `silence_root_warning` flag to avoid bundler printing out alarming looking warning messages when `sudo` is used. This warning message was concerning for users - it would be printed out during normal operation of `bin/logstash-plugin install/update/remove` when run under `sudo`, which is the expected mode of operation when logstash is installed to run as a service via rpm/deb packages. This PR also updates the vagrant based integration tests to ensure that Logstash still runs after plugin update/install/remove operations, fixes up some regular expressions that would cause test failures, and removes some dead code from tests. * Updated Bundler to latest version * Ensured that `Gemfile.lock` are appropriately frozen * Added new developer-only flag to facilitate local plugin development to allow unfrozen lockfile in a development environment (cherry picked from commit 4707cb) * Remove code pinning bundler to ~> 1.17

* master: Update releases list (elastic#13149) Bundler: freeze lockfile on run, and "normalize" platform on plugin changes (elastic#13015) Doc: Move shared region tags for breaking changes to 8.0.0 content (elastic#13131) Update Snakeyaml version to 1.29 (elastic#13129) Doc: Add breaking changes to breaking changes page (elastic#13130) Added faraday-* and ruby2_keywords notices to licences list (elastic#13126) Release notes draft for 8.0.0-alpha1 (elastic#13098) valid variable expansion for the batch files (elastic#12912) Doc: Enhance and expand DLQ docs (elastic#12959) Update releases list with 7.13.4 (elastic#13088) Doc:Fix typo and adjust keystore text (elastic#12779) doc: add pipeline.ecs_compatibility docs (elastic#12421) ecs_compatibility: revert breaking change; keep `disabled` as default for 8.0.0 (elastic#13080) Doc: Add kafka schema registry setting to troubleshooting (elastic#13073) [Test] Fix Unix acceptance tests (elastic#13071) Move retrieval of Stack version from Gradle's configuration to execution (elastic#13042) Update releases list with 6.8.17 and 7.13.3 (elastic#13041) Docs: keep elastic_app_search output meta-data (elastic#13048) Fix LS benchmarking tool to work with releases >= 7.10.0 (elastic#13052) Revert "Change Gradle's :logstash-integration-tests:integrationTests task to depends on copyES (elastic#12847)" (elastic#13045)

maltewhiite · 2021-12-16T12:04:31Z

How can I fix this? https://discuss.elastic.co/t/fatal-error-after-updating-to-7-16-1/291959

MarcoMartins86 · 2021-12-16T16:02:32Z

@maltewhiite I've also encountered this problem. I'm using the logstash official docker and I was adding gems to the Gemfile manually which triggers the error. The tricky part was to run the embedded bundle to install the gems, not sure if there is an easier way to do it without having hardcoded paths that might change in the future. This solved for me
/usr/share/logstash/bin/ruby -S $(/usr/share/logstash/bin/ruby -S gem env gemdir 2>&1 | grep /usr/share/logstash/vendor/bundle/)/bin/bundle add redis, if you're not adding gems manually but still see the error maybe try to do a bundle lock before running logstash.

maltewhiite · 2021-12-17T11:42:20Z

@MarcoMartins86 - Thanks a lot!!

Could you explain what happens here?
$(/usr/share/logstash/bin/ruby -S gem env gemdir 2>&1 | grep /usr/share/logstash/vendor/bundle/)/bin/bundle

What is 2>&1?
What is $( ) ?

MarcoMartins86 · 2021-12-17T15:14:11Z

@maltewhiite

$( ): basically evaluates the expression inside the parenthesis
2>&1 | : redirects all output (stdout, stderror) from previous call /usr/share/logstash/bin/ruby -S gem env gemdir to grep

Finally, the whole thing is just to give you the root path for the gem dir /usr/share/logstash/vendor/bundle/jruby/2.5.0, so in the future, the version can change and I don't need to change it manually.

roaksoax added the status:work-in-progress label Jun 21, 2021

robbavey force-pushed the bundler_update branch 7 times, most recently from c07d595 to ff036c6 Compare June 25, 2021 14:06

robbavey force-pushed the bundler_update branch 2 times, most recently from bb6829c to b932c7b Compare July 1, 2021 21:09

robbavey force-pushed the bundler_update branch 2 times, most recently from b4420b9 to 97063eb Compare July 26, 2021 14:16

yaauie reviewed Jul 26, 2021

View reviewed changes

robbavey force-pushed the bundler_update branch 3 times, most recently from 35958a2 to fd4d669 Compare July 27, 2021 21:12

robbavey force-pushed the bundler_update branch 3 times, most recently from de845c2 to 8ac6138 Compare July 28, 2021 19:04

robbavey marked this pull request as ready for review July 29, 2021 17:32

robbavey changed the title ~~[wip] Bundler: freeze lockfile on run, and "normalize" platform on pl…~~ Bundler: freeze lockfile on run, and "normalize" platform on plugin changes Jul 29, 2021

roaksoax added status:needs-review and removed status:work-in-progress labels Jul 29, 2021

yaauie reviewed Jul 29, 2021

View reviewed changes

robbavey requested a review from karenzone July 30, 2021 17:40

robbavey force-pushed the bundler_update branch from 4c0a135 to f73021a Compare August 16, 2021 13:51

yaauie reviewed Aug 16, 2021

View reviewed changes

robbavey and others added 4 commits August 16, 2021 13:38

Remove debugging information

8c24027

Co-authored-by: Ry Biesemeyer <[email protected]>

whitespace snip

f002c38

Co-authored-by: Ry Biesemeyer <[email protected]>

Always add generic java platform when injecting dependencies

3732b9f

Fix test to allow for custom jdk location

48889fd

yaauie approved these changes Aug 17, 2021

View reviewed changes

robbavey merged commit 4707cbd into master Aug 17, 2021

This was referenced Aug 17, 2021

Backport PR #13015 to 7.x: Bundler: freeze lockfile on run, and "norm… #13140

Merged

Backport PR #13015 to 7.15: Bundler: freeze lockfile on run, and "nor… #13141

Merged

robbavey mentioned this pull request Aug 18, 2021

Revert bundler version #12910

Closed

5 tasks

kares mentioned this pull request Sep 1, 2021

Release notes for 7.15.0 #13139

Merged

roaksoax removed the status:approved label Oct 1, 2021

kares added the v7.15.0 label Nov 25, 2021

kares mentioned this pull request Nov 25, 2021

official private gem repositories docs needs to be updated #13451

Open

kares mentioned this pull request Dec 13, 2021

mutating a Gemfile should have better error message using bin/logstash-plugin #13505

Open

jsvd deleted the bundler_update branch June 2, 2023 12:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler: freeze lockfile on run, and "normalize" platform on plugin changes #13015

Bundler: freeze lockfile on run, and "normalize" platform on plugin changes #13015

robbavey commented Jun 21, 2021 •

edited

Loading

yaauie left a comment

yaauie Jul 26, 2021

robbavey commented Jul 26, 2021 •

edited

Loading

yaauie left a comment

yaauie Jul 26, 2021

yaauie Jul 27, 2021

robbavey Jul 28, 2021

robbavey commented Jul 27, 2021

robbavey commented Jul 29, 2021

yaauie left a comment

yaauie Jul 29, 2021

robbavey Jul 30, 2021

yaauie Jul 29, 2021

robbavey Jul 30, 2021

yaauie Jul 29, 2021

yaauie Jul 29, 2021

yaauie Jul 29, 2021

yaauie Jul 29, 2021

robbavey Jul 30, 2021

yaauie Jul 29, 2021

robbavey Jul 30, 2021

robbavey commented Jul 30, 2021

robbavey commented Aug 16, 2021

yaauie left a comment

yaauie Aug 16, 2021

robbavey Aug 16, 2021

robbavey commented Aug 16, 2021

yaauie left a comment

maltewhiite commented Dec 16, 2021

MarcoMartins86 commented Dec 16, 2021

maltewhiite commented Dec 17, 2021

MarcoMartins86 commented Dec 17, 2021

Bundler: freeze lockfile on run, and "normalize" platform on plugin changes #13015

Bundler: freeze lockfile on run, and "normalize" platform on plugin changes #13015

Conversation

robbavey commented Jun 21, 2021 • edited Loading

Release notes

Checklist

Author's Checklist

How to test this PR locally

Related issues

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Jul 26, 2021 • edited Loading

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Jul 27, 2021

robbavey commented Jul 29, 2021

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Jul 30, 2021

robbavey commented Aug 16, 2021

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Aug 16, 2021

yaauie left a comment

Choose a reason for hiding this comment

maltewhiite commented Dec 16, 2021

MarcoMartins86 commented Dec 16, 2021

maltewhiite commented Dec 17, 2021

MarcoMartins86 commented Dec 17, 2021

robbavey commented Jun 21, 2021 •

edited

Loading

robbavey commented Jul 26, 2021 •

edited

Loading