keg_relocate: cache files rewritten during brew bottle #1253

jawshooah · 2016-10-10T01:17:09Z

Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your changes? Here's an example.
Have you successfully run brew tests with your changes locally?

brew bottle replaces instances of the Homebrew prefix, cellar, and repository with placeholders in all text files. Cache these files in INSTALL_RECEIPT.json so that we don't have to check every single text file for placeholders on install.

Also add a couple of other optimizations to speed up building bottles.

Fixes #1250.

jawshooah · 2016-10-10T01:51:02Z

@zmwangx This cuts the install time for mysql on my machine down from ~100s to ~10s.

MikeMcQuaid · 2016-10-10T08:05:37Z

Library/Homebrew/formula_installer.rb

@@ -777,13 +777,18 @@ def pour
    end

    keg = Keg.new(formula.prefix)
+    tab_file = formula.prefix.join(Tab::FILENAME)
+    # Skip the cache since the receipt will be rewritten


Not sure I understand this comment, can you elaborate a bit?

Sure. Tab.from_file and Tab.for_keg both cache the resulting Tab instance so that when either is called with the same path, the same instance is returned. Tab.from_file_content bypasses the cache. This is desirable here because keg.relocate_text_files will rewrite the tab file, and we want to load in the rewritten file before modifying it and writing it out again a few lines down.

I think it would be clearer to nuke the cache before rereading rather than use a method that's not using the cache (and should probably be private as a result).

MikeMcQuaid · 2016-10-10T08:06:18Z

Library/Homebrew/keg_relocate.rb

-  def relocate_text_files(old_prefix, new_prefix, old_cellar, new_cellar,
-                          old_repository, new_repository)
-    files = text_files | libtool_files
+  def relocate_text_files(old_prefix, new_prefix, old_cellar, new_cellar, # rubocop:disable Metrics/ParameterLists


Maybe worth passing in a hash instead of disabling this cop.

Agreed, I'll clean this up.

MikeMcQuaid · 2016-10-10T08:06:52Z

Library/Homebrew/keg_relocate.rb

@@ -75,8 +78,8 @@ def text_files
      path.find do |pn|
        next if pn.symlink? || pn.directory?
        next if Metafiles::EXTENSIONS.include? pn.extname
-        if Utils.popen_read("/usr/bin/file", "--brief", pn).include?("text") ||
-           pn.text_executable?
+        if pn.text_executable? ||


Good idea on flipping these 👍

MikeMcQuaid · 2016-10-10T08:07:25Z

Library/Homebrew/metafiles.rb

 class Metafiles
  # https://github.com/github/markup#markups
  EXTENSIONS = %w[
    .adoc .asc .asciidoc .creole .html .markdown .md .mdown .mediawiki .mkdn
    .org .pod .rdoc .rst .rtf .textile .txt .wiki
-  ].freeze
+  ].to_set.freeze


If this is a frozen array I'm not sure I see the advantages of making it a set?

O(1) lookup versus O(n). If order doesn't matter and there are no duplicates, why use an array when you could use a set?

Keep in mind that Keg#text_files queries this collection once for every file, so bringing the complexity of that search down is worthwhile if we care about brew bottle runtime.

The algorithmic speedup is always nice but given Sets aren't strictly interchangeable with arrays I figure it's usually worth using arrays unless profiling shows using sets produces a measurable speedup. Based on your previous comments it sounds like it does so 👍

MikeMcQuaid · 2016-10-10T08:08:02Z

Library/Homebrew/dev-cmd/bottle.rb

@@ -205,6 +206,7 @@ def bottle_formula(f)
        tab.poured_from_bottle = false
        tab.HEAD = nil
        tab.time = nil
+        tab.changed_files = changed_files


My only concern with putting it in the tab is that this may be a very long list. How many values are there for e.g. mysql?

Here's my INSTALL_RECEIPT.json for mysql:

{ "used_options": [], "unused_options": [ "--with-test", "--with-embedded", "--with-archive-storage-engine", "--with-blackhole-storage-engine", "--with-local-infile", "--with-debug" ], "built_as_bottle": true, "poured_from_bottle": true, "changed_files": [ "INSTALL_RECEIPT.json", "bin/mysql_config", "bin/mysqld_multi", "bin/mysqld_safe", "homebrew.mxcl.mysql.plist", "include/mysql/my_config.h", "lib/pkgconfig/mysqlclient.pc", "mysql-test/Makefile", "mysql-test/suite/ndb/include/have_java.inc", "share/doc/mysql/INFO_BIN", "share/man/man1/myisam_ftdump.1", "share/man/man1/mysql.server.1", "share/man/man1/mysql_config.1", "share/man/man1/mysql_install_db.1", "share/man/man1/mysqld_multi.1", "share/man/man1/mysqld_safe.1", "share/man/man1/mysqldumpslow.1", "support-files/mysql-log-rotate", "support-files/mysql.server", "support-files/mysqld_multi.server" ], "time": 1476101424, "source_modified_time": 1472126415, "HEAD": null, "stdlib": "libcxx", "compiler": "clang", "runtime_dependencies": [], "source": { "path": "/usr/local/Library/Taps/homebrew/homebrew-core/Formula/mysql.rb", "tap": "homebrew/core", "spec": "stable", "versions": { "stable": "5.7.15", "devel": null, "head": null, "version_scheme": 0 } } }

Not too bad, I think. Would you rather output to a separate file?

Nah that seems fine, thanks 👍

MikeMcQuaid · 2016-10-10T08:08:12Z

Loving the work here so far, great job @jawshooah!

reitermarkus · 2016-10-10T13:26:26Z

Library/Homebrew/keg_relocate.rb

-        if Utils.popen_read("/usr/bin/file", "--brief", pn).include?("text") ||
-           pn.text_executable?
+        if pn.text_executable? ||
+           Utils.popen_read("/usr/bin/file", "--brief", pn).include?("text")


Could it be faster running this only once for all files?

files = path.find.reject { |pn| pn.symlink? || pn.directory || Metafiles::EXTENSIONS.include?(pn.extname) } text_file_indices = Utils.popen_read("/usr/bin/file", "--no-dereference", "--brief", *files) .lines .map.with_index { |line, i| line.include?("text") ? i : nil } .compact text_files = files.values_at(*text_file_indices)

@reitermarkus Some local testing looks like it should be, good catch. There's probably a maximum number of arguments we can specify but batching them may well produce a speedup 👍

I tried to get rid of

brew/Library/Homebrew/cask/lib/hbc/artifact/uninstall_base.rb

Line 11 in c9c7f46

PATH_ARG_SLICE_SIZE = 500

once before by using getconf ARG_MAX, but somehow got sidetracked. 😄

jawshooah · 2016-10-14T20:37:59Z

@MikeMcQuaid are you still interested in this now that #1258 and friends have been merged? It still offers a significant boost in install time, ~4x from my rough testing.

MikeMcQuaid · 2016-10-15T09:56:12Z

It still offers a significant boost in install time, ~4x from my rough testing.

This was going to be my next question! In that case: yes, definitely! Please also check a bottle with/without this data can still be poured/relocated correctly both without/with this change (i.e. that all four combinations work fine).

Great work here!

MikeMcQuaid · 2016-10-23T08:34:08Z

Any news here @jawshooah? Shout if I can help at all.

MikeMcQuaid · 2016-10-23T08:37:48Z

Library/Homebrew/keg_relocate.rb

-                          old_repository, new_repository)
-    files = text_files | libtool_files
+  def replace_locations_with_placeholders
+    relocate_dynamic_linkage(HOMEBREW_PREFIX.to_s, PREFIX_PLACEHOLDER,


Tiny nit: might want to make the two prefix args a key/value of a hash or something? Doesn't really matter but I always get a little 😨 about things that rely on arg ordering like this.

In fact: like you did below with replacements 😆

How about a Struct? f798ba4

jawshooah · 2016-10-23T14:37:41Z

@MikeMcQuaid commented on Oct 23, 2016, 4:34 AM EDT:

Any news here @jawshooah? Shout if I can help at all.

Just need to get around to re-verifying that everything works in the scenarios you mentioned. Nothing blocking me but time 😄

`brew bottle` replaces instances of the Homebrew prefix, cellar, and repository with placeholders in all text files. Cache these files in INSTALL_RECEIPT.json so that we don't have to check every single text file for placeholders on install.

Replace relocate_text_files with three methods that clarify intent: replace_locations_with_placeholders, replace_placeholders_with_locations and replace_text_in_files, the first two calling the third.

jawshooah · 2016-10-24T20:44:34Z

@MikeMcQuaid I've verified that there are no problems installing with/without the data and with/without the change. Merging now as CI is 💚 and you've already approved the changes.

scpeters · 2016-10-25T00:08:58Z

this may have broken the mercurial bottle for me, still investigating...

scpeters · 2016-10-25T00:10:59Z

$ head -1 /usr/local/opt/mercurial/bin/hg
#!@@HOMEBREW_PREFIX@@/opt/python/bin/python2.7

scpeters · 2016-10-25T00:34:08Z

I reverted to e6bce5e, and the problem went away, so I think it's an issue with this PR.

Here's what I see when I install mercurial:

brew install mercurial
==> Downloading https://homebrew.bintray.com/bottles/mercurial-3.9.2.yosemite.bottle.tar.gz
Already downloaded: /Users/jenkins/Library/Caches/Homebrew/mercurial-3.9.2.yosemite.bottle.tar.gz
==> Pouring mercurial-3.9.2.yosemite.bottle.tar.gz
Error: No such file or directory - /usr/local/opt/mercurial/bin/hg

There is a file at that location though:

$ head /usr/local/opt/mercurial/bin/hg
#!@@HOMEBREW_PREFIX@@/opt/python/bin/python2.7
#
# mercurial - scalable distributed SCM
#
# Copyright 2005-2007 Matt Mackall <[email protected]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

import os

zmwangx · 2016-10-25T05:49:12Z

There's a bug here with deleting pyc files in the wrong order. Causing a failure: https://bot.brew.sh/job/Homebrew%20Core/10080/version=el_capitan/testReport/junit/brew-test-bot/el_capitan/bottle_buku/. Fix in #1375.

formula_installer: fix regression in #1253

BrewTestBot added the in progress Maintainers are working on this label Oct 10, 2016

MikeMcQuaid reviewed Oct 10, 2016

View reviewed changes

reitermarkus reviewed Oct 10, 2016

View reviewed changes

jawshooah mentioned this pull request Oct 10, 2016

keg_relocate: run /usr/bin/file in batches #1258

Merged

5 tasks

ilovezfs added cask Homebrew Cask and removed cask Homebrew Cask labels Oct 11, 2016

MikeMcQuaid approved these changes Oct 23, 2016

View reviewed changes

jawshooah and others added 7 commits October 24, 2016 16:21

metafiles: convert EXTENSIONS and BASENAMES from Array to Set

ce33f59

metafiles: use Set.new instead of Array#to_set

9628a61

metafiles: convert Metafiles class to module

5b64fa6

keg_relocate: refactor relocate_text_files

9c519bb

Replace relocate_text_files with three methods that clarify intent: replace_locations_with_placeholders, replace_placeholders_with_locations and replace_text_in_files, the first two calling the third.

keg_relocate: brew style fixes

adc4b1f

keg_relocate: wrap relocation locations in struct

fab2cff

jawshooah merged commit 79e8cdd into Homebrew:master Oct 24, 2016

jawshooah deleted the perf/relocate-text-files branch October 24, 2016 20:44

BrewTestBot removed the in progress Maintainers are working on this label Oct 24, 2016

scpeters mentioned this pull request Oct 25, 2016

Major regression in #1253 #1372

Closed

3 tasks

nuhuxi mentioned this pull request Oct 25, 2016

/usr/local/Homebrew migration tracking issue #987

Closed

This was referenced Oct 25, 2016

bottle: fix nonexistent pyc bug #1375

Merged

buku: add beautifulsoup4 and cryptography dependencies Homebrew/homebrew-core#6241

Closed

ilovezfs added a commit that referenced this pull request Oct 25, 2016

Merge pull request #1376 from jawshooah/fix-relocate-regression

a4f654f

formula_installer: fix regression in #1253

Homebrew locked and limited conversation to collaborators May 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

keg_relocate: cache files rewritten during brew bottle #1253

keg_relocate: cache files rewritten during brew bottle #1253

jawshooah commented Oct 10, 2016 •

edited

Loading

jawshooah commented Oct 10, 2016

MikeMcQuaid Oct 10, 2016

jawshooah Oct 10, 2016

MikeMcQuaid Oct 10, 2016

MikeMcQuaid Oct 10, 2016

jawshooah Oct 10, 2016

MikeMcQuaid Oct 10, 2016

MikeMcQuaid Oct 10, 2016

MikeMcQuaid Oct 10, 2016

jawshooah Oct 10, 2016

jawshooah Oct 10, 2016 •

edited

Loading

MikeMcQuaid Oct 10, 2016

MikeMcQuaid Oct 10, 2016

jawshooah Oct 10, 2016

MikeMcQuaid Oct 10, 2016

MikeMcQuaid commented Oct 10, 2016

reitermarkus Oct 10, 2016

MikeMcQuaid Oct 10, 2016

reitermarkus Oct 10, 2016 •

edited

Loading

jawshooah commented Oct 14, 2016 •

edited

Loading

MikeMcQuaid commented Oct 15, 2016

MikeMcQuaid commented Oct 23, 2016

MikeMcQuaid Oct 23, 2016

MikeMcQuaid Oct 23, 2016

jawshooah Oct 23, 2016

jawshooah Oct 24, 2016

jawshooah commented Oct 23, 2016

jawshooah commented Oct 24, 2016

scpeters commented Oct 25, 2016

scpeters commented Oct 25, 2016

scpeters commented Oct 25, 2016

zmwangx commented Oct 25, 2016

keg_relocate: cache files rewritten during brew bottle #1253

keg_relocate: cache files rewritten during brew bottle #1253

Conversation

jawshooah commented Oct 10, 2016 • edited Loading

jawshooah commented Oct 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jawshooah Oct 10, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MikeMcQuaid commented Oct 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reitermarkus Oct 10, 2016 • edited Loading

Choose a reason for hiding this comment

jawshooah commented Oct 14, 2016 • edited Loading

MikeMcQuaid commented Oct 15, 2016

MikeMcQuaid commented Oct 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jawshooah commented Oct 23, 2016

jawshooah commented Oct 24, 2016

scpeters commented Oct 25, 2016

scpeters commented Oct 25, 2016

scpeters commented Oct 25, 2016

zmwangx commented Oct 25, 2016

jawshooah commented Oct 10, 2016 •

edited

Loading

jawshooah Oct 10, 2016 •

edited

Loading

reitermarkus Oct 10, 2016 •

edited

Loading

jawshooah commented Oct 14, 2016 •

edited

Loading