vendor ruby #404

xu-cheng · 2016-06-27T14:11:56Z

Add install-vendor command to install vendor ruby/git
The vendor Ruby/Git will be put inside
Library/Homebrew/vendor/{ruby,git}/<version>, with a symlink
Library/Homebrew/vendor/{ruby,git}/opt pointed to it.
In addition, a Library/Homebrew/vendor/{ruby-version,git-version} will
track the latest version of vendor binaries.
It is important to notice that for most of users they will continue to use system ruby/git. vendor version is only used for old systems.
add --homebrew=fail-on-old-vendor-version flag for ENV/scm/git to help auto detect outdated vendor version.
set minimal git version to 1.8, which supports -c flag during git clone.
current vendored ruby/git is for test purpose only.

To see vendor git, master...xu-cheng:vendor-bak

MikeMcQuaid · 2016-06-27T15:45:32Z

Could we make this PR just for Ruby for now to slim the diff and add documentation for how to build and upload the Ruby as part of this PR? Thanks!

xu-cheng · 2016-06-28T07:11:27Z

I make the PR only for ruby now.

add documentation for how to build and upload the Ruby as part of this PR?

I have move the portable formulae repo under Homebrew's organization: https://github.com/Homebrew/homebrew-portable
I think document should go to that repo.

There are already some basic document here. However, I may not find enough free time lately to work on automatic building/uploading tool. If any maintainers are interesting to work on that, feel free to do so.

xu-cheng · 2016-06-28T07:13:22Z

cc @mistydemeo @sjackman for suggesting on automatic building PowerPC and Linux vendor tools.

mistydemeo · 2016-06-28T07:17:09Z

Library/Homebrew/cmd/install-vendor.sh

+  then
+    curl_args="-#fLA"
+  else
+    curl_args="-sfLA"


In a followup PR, I'd like to look into vendoring curl as well; older platforms won't be able to fetch from all hosts without.

This is on my watch list. However, I haven't figured out how to download vendor curl itself. Maybe a separate fetching strategy using system curl with --insecure flag?

It's a bit of a chicken and egg problem, huh? Yes, maybe...

Assuming they got Homebrew from some trusted source, maybe the way to address this would be to ignore transport security and only rely on the SHA-256 to validate the integrity of the download? (I guess this pretty much boils down to system curl with the --insecure flag.)

UniqMartin · 2016-06-28T16:02:21Z

Add install-vendor command to install vendor ruby/git

Can we rename this to vendor-install or something similar? The problem here is that this has a common prefix with the very popular install command and will annoy people that are used to the install command being tab-completed (including the trailing space). We can of course also address this by excluding install-vendor from tab completion, but this adds unnecessary complication.

The vendor Ruby/Git will be put inside Library/Homebrew/vendor/{ruby,git}/<version>, with a symlink Library/Homebrew/vendor/{ruby,git}/opt pointed to it.

I think re-using opt here (I assume inspired from HOMEBREW_PREFIX/opt) feels a bit confusing to me, simply because the directory structure is a different one (<package>/opt vs. opt/<package>). I believe active, current, or latest (in this order of preference) would be a better name here. opt isn't self-explanatory if it appears in a directory listing next to a bunch of version numbers, but e.g. active is.

MikeMcQuaid · 2016-06-28T21:34:54Z

I make the PR only for ruby now.

Thanks. I still see e.g. git_URL in here, though?

There are already some basic document here. However, I may not find enough free time lately to work on automatic building/uploading tool. If any maintainers are interesting to work on that, feel free to do so.

Would be good to add how/where to upload to that document, even if it's not automatic or terminal commands but links to e.g. Bintray. Also, rather than documenting multiple commands just making a simple script that runs them and could be part of that tap feels like it'll be easier e.g. brew portable-install ruby that just runs those commands listed in the Ruby and checks the exit codes.

Can we rename this to vendor-install or something similar? The problem here is that this has a common prefix with the very popular install command and will annoy people that are used to the install command being tab-completed (including the trailing space).

👍

I think re-using opt here (I assume inspired from HOMEBREW_PREFIX/opt) feels a bit confusing to me, simply because the directory structure is a different one (/opt vs. opt/). I believe active, current, or latest (in this order of preference) would be a better name here. opt isn't self-explanatory if it appears in a directory listing next to a bunch of version numbers, but e.g. active is.

👍 to current but 👍 to this point in general.

MikeMcQuaid · 2016-06-28T21:35:49Z

Library/Homebrew/cmd/install-vendor.sh

+
+  if [[ -n "$HOMEBREW_VERBOSE" ]]
+  then
+    curl_args="-fLA"


Could turn the shared args into a shared variable. Also, can you use the long form of these flags to make it more readable.

xu-cheng · 2016-06-29T08:49:48Z

I still see e.g. git_URL in here, though?

Can we keep it as is for now, which can help me to reduce conflicting in git rebase. Please note I want to ship vendoring Ruby and git at the same time, or at least in short time between two. As I have stated in the old PR, we need to make sure vendoring system work for both Ruby and Git(as they are invoked in different ways). It will create a lot of implemention as well as maintenance problems if we fail to make sure them in the first time.

As for auto-building, I am considering to ultilize test-bot in #410. But there are several things need to be done. One simple script for all won't work, because there are a few manually tests invoked. I will need to tweak test and linkage command to make auto test possible.

MikeMcQuaid · 2016-06-29T18:55:37Z

Can we keep it as is for now, which can help me to reduce conflicting in git rebase. Please note I want to ship vendoring Ruby and git at the same time, or at least in short time between two. As I have stated in the old PR, we need to make sure vendoring system work for both Ruby and Git(as they are invoked in different ways). It will create a lot of implemention as well as maintenance problems if we fail to make sure them in the first time.

I'm pretty distrustful of any system that involves us having to get it perfect first time. As a result, I'd like to consider making this opt-in using an environment variable so we can get wider testing with it in master before we roll it out to everyone to build confidence (as we did with the Bash updater which was a similarly dramatic change).

I'm 👎 on shipping Git and Ruby at the same time or having Git references in this PR. We should ship just Ruby first and shipping Git/Curl once we've had vendored Ruby working in production for users for e.g. a few weeks. Making the PR as minimal as possible will make review better and faster and introduce less changes that could break things.

As for auto-building, I am considering to ultilize test-bot in #410. But there are several things need to be done. One simple script for all won't work, because there are a few manually tests invoked. I will need to tweak test and linkage command to make auto test possible.

We don't need autobuild or autotesting but there's definitely room to script some of the instructions you have there and e.g. run commands and tell the developers to manually inspect them. Documentation gets easily outdated, scripts less so.

xu-cheng · 2016-06-30T07:03:15Z

As a result, I'd like to consider making this opt-in using an environment variable so we can get wider testing with it in master before we roll it out to everyone to build confidence (as we did with the Bash updater which was a similarly dramatic change).

We can do it.

having Git references in this PR

Sorry, but I fail to understand why a simple line git_URL would cause any problem. It's very frustrated for me as you are basically asking me to repeatedly manually fix git conflict for me to test both vendor Ruby and Git locally whenever I refresh this PR. I'd like to choose the road which makes my development easier.

I'm 👎 on shipping Git and Ruby at the same time

OK, maybe I should rephrase my word. I won't ship any PR before I have confidence this vendor system can work for both git and ruby, which means it should at lease pass my local tests. As I have kept repeatedly stated, some part of details in system cannot be easily changed afterwards.

I'm pretty distrustful of any system that involves us having to get it perfect first time.

I am just trying to make it as perfect as possible in the first time. I don't think you are against this right?

xu-cheng · 2016-06-30T13:27:28Z

I think I have addressed all comments.

MikeMcQuaid · 2016-06-30T22:03:45Z

We can do it.

Great 👍

Sorry, but I fail to understand why a simple line git_URL would cause any problem. It's very frustrated for me as you are basically asking me to repeatedly manually fix git conflict for me to test both vendor Ruby and Git locally whenever I refresh this PR. I'd like to choose the road which makes my development easier.

Sorry, I don't think resolving conflicts in your local development environment justifies adding code to a Homebrew pull request. It won't cause problems but it's dead code that implies a Git feature that's not reviewed yet or a hard requirement for 1.0 (like Ruby is). I'd suggest you focus on just building the Ruby feature for now and don't rebase the Git work until this PR is merged.

OK, maybe I should rephrase my word. I won't ship any PR before I have confidence this vendor system can work for both git and ruby, which means it should at lease pass my local tests. As I have kept repeatedly stated, some part of details in system cannot be easily changed afterwards.

Why does this system need to work for Git for us to ship the Ruby PR?

I am just trying to make it as perfect as possible in the first time. I don't think you are against this right?

Nope, but I do think it's important to focus on Ruby for now and Git later.

xu-cheng · 2016-07-01T06:28:52Z

Why does this system need to work for Git for us to ship the Ruby PR?

This is where you missed. We do need to make vendor system including update detection works for both Ruby and Git. On the one hand, I don't like the idea that we will have two different systems to vendor them, if we find it fails for Git. On the other hand, it is extremely important that we should make sure users won't stuck on old vendor version. Because otherwise, we are creating a huge security hole here. And more improtantly, things like file structure cannot be esaily changed in the future PR at all.

If any help, I can remove the Git mention only before we are shipping this PR. But before then, I will keep it. I don't want to be shortsighted on robutness and security.

MikeMcQuaid · 2016-07-01T15:20:26Z

If any help, I can remove the Git mention only before we are shipping this PR.

Yes, that's fine.

MikeMcQuaid · 2016-07-11T07:27:47Z

Is there any ETA on this PR? There's a few Ruby 1.8 things in the wings that mean I'm wondering how long it's going to be until we're Ruby 2 only. Could you consider shipping this code (without the Git stuff) nearly as-is but just not enabling it without an environment variable so we can test it locally? I think this is higher priority than pretty much any other feature/cleanup in Homebrew at the moment. Another option would be creating this on a non-fork branch so multiple people can work on it.

The vendor Ruby will be put inside `Library/Homebrew/vendor/portable-ruby/<version>`, with a symlink `Library/Homebrew/vendor/portable-ruby/current` pointed to it. In addition, a `Library/Homebrew/vendor/portable-ruby-version` will track the latest version of vendor binaries. This gives us version control on vendor Ruby and enables us to bump vendor Ruby whenever needed such as security update.

MikeMcQuaid · 2016-07-11T13:10:09Z

Library/Homebrew/cmd/vendor-install.sh

@@ -0,0 +1,195 @@
+#:  * `vendor-install` [<target>]:
+#:     Install vendor version of Homebrew dependencies.


Think we can hide this from the manpage (for now at least).

e.g. with #: @hide_from_man_page as the first line.

MikeMcQuaid · 2016-07-11T13:11:10Z

@xu-cheng One final comment then: and we can improve documentation etc. afterwards.

* Use vendor Ruby if it's present * Install vendor Ruby for system without Ruby 2.x

The name of the formula is not extracted correctly when the URL includes a hyphen.

BrewTestBot added the in progress Maintainers are working on this label Jun 27, 2016

xu-cheng added the discussion Input solicited from others label Jun 27, 2016

tdsmith mentioned this pull request Jun 27, 2016

--insecure arg added back #403

Closed

mistydemeo reviewed Jun 28, 2016
View reviewed changes

MikeMcQuaid reviewed Jun 28, 2016
View reviewed changes

UniqMartin mentioned this pull request Jun 28, 2016

keg_relocate: Fix hardlink behavior for ruby-macho #400

Closed

5 tasks

MikeMcQuaid changed the title ~~vendor ruby/git~~ vendor ruby Jul 11, 2016

MikeMcQuaid reviewed Jul 11, 2016
View reviewed changes

xu-cheng added 3 commits July 11, 2016 21:12

add vendor-install command

a35e3f1

brew.sh: new HOMEBREW_RUBY_PATH resolution logic

534bf08

* Use vendor Ruby if it's present * Install vendor Ruby for system without Ruby 2.x

ruby.sh: add test flag

dff6bfd

xu-cheng merged commit 7508da2 into Homebrew:master Jul 11, 2016

xu-cheng deleted the vendor branch July 11, 2016 13:16

BrewTestBot removed the in progress Maintainers are working on this label Jul 11, 2016

iMichka pushed a commit to iMichka/brew that referenced this pull request Jun 22, 2017

BottleLoader: Fix installing a bottle from an URL (Homebrew#404)

2e12154

The name of the formula is not extracted correctly when the URL includes a hyphen.

Homebrew locked and limited conversation to collaborators May 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vendor ruby #404

vendor ruby #404

xu-cheng commented Jun 27, 2016 •

edited

Loading

MikeMcQuaid commented Jun 27, 2016

xu-cheng commented Jun 28, 2016 •

edited

Loading

xu-cheng commented Jun 28, 2016

mistydemeo Jun 28, 2016

xu-cheng Jun 28, 2016

mistydemeo Jun 28, 2016

UniqMartin Jun 28, 2016 •

edited

Loading

UniqMartin commented Jun 28, 2016

MikeMcQuaid commented Jun 28, 2016

MikeMcQuaid Jun 28, 2016

xu-cheng commented Jun 29, 2016

MikeMcQuaid commented Jun 29, 2016

xu-cheng commented Jun 30, 2016

xu-cheng commented Jun 30, 2016

MikeMcQuaid commented Jun 30, 2016

xu-cheng commented Jul 1, 2016 •

edited

Loading

MikeMcQuaid commented Jul 1, 2016

MikeMcQuaid commented Jul 11, 2016

MikeMcQuaid Jul 11, 2016

MikeMcQuaid Jul 11, 2016

MikeMcQuaid commented Jul 11, 2016

		@@ -0,0 +1,195 @@
		#: * `vendor-install` [<target>]:
		#: Install vendor version of Homebrew dependencies.

vendor ruby #404

vendor ruby #404

Conversation

xu-cheng commented Jun 27, 2016 • edited Loading

MikeMcQuaid commented Jun 27, 2016

xu-cheng commented Jun 28, 2016 • edited Loading

xu-cheng commented Jun 28, 2016

mistydemeo Jun 28, 2016

Choose a reason for hiding this comment

xu-cheng Jun 28, 2016

Choose a reason for hiding this comment

mistydemeo Jun 28, 2016

Choose a reason for hiding this comment

UniqMartin Jun 28, 2016 • edited Loading

Choose a reason for hiding this comment

UniqMartin commented Jun 28, 2016

MikeMcQuaid commented Jun 28, 2016

MikeMcQuaid Jun 28, 2016

Choose a reason for hiding this comment

xu-cheng commented Jun 29, 2016

MikeMcQuaid commented Jun 29, 2016

xu-cheng commented Jun 30, 2016

xu-cheng commented Jun 30, 2016

MikeMcQuaid commented Jun 30, 2016

xu-cheng commented Jul 1, 2016 • edited Loading

MikeMcQuaid commented Jul 1, 2016

MikeMcQuaid commented Jul 11, 2016

MikeMcQuaid Jul 11, 2016

Choose a reason for hiding this comment

MikeMcQuaid Jul 11, 2016

Choose a reason for hiding this comment

MikeMcQuaid commented Jul 11, 2016

xu-cheng commented Jun 27, 2016 •

edited

Loading

xu-cheng commented Jun 28, 2016 •

edited

Loading

UniqMartin Jun 28, 2016 •

edited

Loading

xu-cheng commented Jul 1, 2016 •

edited

Loading