brew info should display download size

[osalbahr]

Verification

• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

I want to know the total download/install size when installing a package.

What is the motivation for the feature?

I am using a limited storage system (16 GB). Also boredom and I want to contribute to Homebrew again.

How will the feature be relevant to at least 90% of Homebrew users?

Sort of? It would be nice. But probably not needed. Storage is cheap. It will only affect people with pay-per-use data plans.

What alternatives to the feature have been considered?

brew deps then somehow find tarball size for each package. This is good enough for my use.

Example (only first step):

$ echo $(brew deps fastfetch)
binutils gcc gmp isl libmpc lz4 mpfr xz zlib zstd

Then a bash for-each, since formula names are guaranteed to not have spaces (I think).

But would be nice if it's integrated into Homebrew under an undocumented flag. I would appreciate pointers on how to start this, such as existing open issues/discussions. Thx!

More info:

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: No developer tools installed.
Install Clang or run `brew install gcc`.
$ brew config
HOMEBREW_VERSION: 4.3.23-53-g29c22e0
ORIGIN: https://github.com/Homebrew/brew
HEAD: 29c22e0ab3fd9826cefd4b14c143266dea3bc391
Last commit: 2 days ago
Core tap JSON: 22 Sep 11:36 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_EDITOR: /usr/bin/nano
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.4 => /var/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.4_1/bin/ruby
CPU: octa-core 64-bit sandybridge
Clang: N/A
Git: 2.46.1 => /bin/git
Curl: 8.6.0 => /bin/curl
Kernel: Linux 6.10.10-200.fc40.x86_64 x86_64 GNU/Linux
OS: Fedora release 40 (Forty)
Host glibc: 2.39
/usr/bin/gcc: N/A
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

[osalbahr]

Here is a minimal way to find the total tarball sizes:

for P in $(brew deps fastfetch)
do
    brew cat "$P" | \
    grep -m1 url | \
    awk '{print $2}' | \
    xargs wget -qO- | \
    wc -c
done

Note: tapping and downloading is required by the above for loop which (sort of) defeats the points, but this is merely a proof of concept. I think there should be a "size" field right below the sha256 value in a formula. Also theoretically a "security" measure to check both the shasum and size, but that's a later decision.

[carlocab]

I think there should be a "size" field right below the sha256 value in a formula.

That's a bit tricky since the bottle contains a copy of the formula, so changing the formula changes the size of the bottle too.

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest, so in principle this should be straightforward to recover with curl and jq. I don't think it contains source tarball size, but most users probably don't really care about this (because they install formulae from bottles).

Or, in a pinch, if you only care about compressed size:

brew fetch foo
wc -c "$(brew --cache foo)" # bottle size

brew fetch --build-from-source foo
wc -c "$(brew --cache --build-from-source foo)" # source tarball size

This makes it so that you don't need to download foo all over again if you want to install it or build it from source.

[osalbahr]

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest, so in principle this should be straightforward to recover with curl and jq. I don't think it contains source tarball size, but most users probably don't really care about this (because they install formulae from bottles).

I agree. The size of bottle is what I was looking for rather than the tarballs.

That said, I think the bottle size (both compresses and uncompressed) is available in the annotations in the manifest

That sounds like what I was looking for. Is there perhaps some sort of API that lets me "fetch" only the size, without first downloading the bottle?

And wc -c "$(brew --cache foo)" does seem to print the intended "Download Size" of a specific formula. But to my understanding I have to have the bottle cached locally first.

[carlocab]

That sounds like what I was looking for. Is there perhaps some sort of API that lets me "fetch" only the size, without first downloading the bottle?

Yes. The manifest can be fetched separately from the bottle. (In fact, brew downloads the manifest first before downloading the bottle.)

And wc -c "$(brew --cache foo)" does seem to print the intended "Download Size" of a specific formula. But to my understanding I have to have the bottle cached locally first.

Yes, that's why you need to brew fetch first.

[osalbahr]

Where can I read more about getting only the manifest? I couldn't find it documented in https://docs.brew.sh.

[carlocab]

It's not documented, but you can see what brew does to fetch it by doing brew fetch --verbose:

❯ brew fetch --force --verbose libuv
==> Downloading https://ghcr.io/v2/homebrew/core/libuv/manifests/1.48.0
/usr/bin/env /opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.3.23-54-gde9848d\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 15.0\)\ curl/8.7.1 --header Accept-Language:\ en --fail --retry 3 --header Accept:\ application/vnd.oci.image.index.v1+json --header Authorization:\ Bearer\ QQ== --remote-time --output /Users/carlocab/Library/Caches/Homebrew/downloads/8ee1d27fb604f55e3c4415b96e34dc9c7f557996832c372d984c8162b29a4ad2--libuv-1.48.0.bottle_manifest.json.incomplete --location https://ghcr.io/v2/homebrew/core/libuv/manifests/1.48.0

[osalbahr]

Now I'm thinking I perhaps should first work on a brew fetch --manifest-only foo. What do you think?

[carlocab]

Seems to have a relatively niche/obscure use-case at the moment, so it's probably not a good fit.

[osalbahr]

Ok. I'll see what I can do. Thx!

[umnikos]

One time I tried to install a calculator with brew. It took a weirdly long amount of time, and then I saw this:

Installing libqalculate dependency: xorg-server

I think I'd personally benefit from brew calculating some kind of size for the operation (download size should be easy) and then giving the user a yes/no prompt if that size is large.

[osalbahr]

I am no longer interested in solely working on a PR. It turned out to be more complicated than I expected as there is no API to fetch the download/install size.

But happy to help. @umnikos if you (or someone else) are interested, lmk, and I will re-open the issue. Happy to help you debug too, just @ me.

I will also consider working on this given enough thumbs up.

Best proxy for interest I could find is a 2-year-old StackOverflow post with 7 upvotes: How to make Brew show the size of the formula before installing it?.

That post also has workarounds that seem to no longer work but might be close enough. Most notably, it talks about the Bearer Token which is needed for /usr/bin/env /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Linuxbrew/4.3.23-56-g9160445\ \(Linux\;\ x86_64\ Ubuntu\ 22.04.5\ LTS\)\ curl/7.81.0 --header Accept-Language:\ en --fail --retry 3 --header Accept:\ application/vnd.oci.image.index.v1+json --header Authorization:\ Bearer\ QQ== --remote-time --output /home/linuxbrew/.cache/Homebrew/downloads/0f02a3a463ce4e72f92871751d9ba7b872ca8090348074d46ffb523fd67e1c7b--xz-5.6.2.bottle_manifest.json.incomplete --location http (the token is the main reason why simply curl doesn't work, but I do not understand the rest of that one-liner).

[MikeMcQuaid]

brew calculating some kind of size for the operation (download size should be easy)

@umnikos if you read the whole thread you'll see why this is not easy. If you disagree: we'd welcome said easy PR 😉

t turned out to be more complicated than I expected as there is no API to fetch the download/install size.

@osalbahr as mentioned by @carlocab above: said API is "download the manifest first and use that to figure out the size of the formula".

The longer-term solution here would be to consider if we could e.g. download all these manifests daily and provide a JSON file of the rough sizes that could be incorporated into the formulae.brew.sh JSON API. I'd imagine ~24h outdated information here would be more useful than none at all.

[osalbahr]

@osalbahr as mentioned by @carlocab above: said API is "download the manifest first and use that to figure out the size of the formula".

I tried to work on that, but the path for going from libuv (for example) to the long curl one-liner seemed intimidating. My approach is usually to hack something up in a language I know, like bash, then figure out how to put that in a language I am not familiar with, like Ruby. However I think I might need to go at this issue in Ruby directly.

Could you point me at the Ruby function that creates the curl command that downloads the manifest? It is difficult to figure that out since I couldn't find info on the manifest in the technical docs.

[apainintheneck]

It's not exactly what you're looking for but @cho-m opened a proof of concept PR (#18172) a couple weeks ago to add formula size information in brew info. The size information gets pulled from the manifest which seems similar to what you all are talking about here. Of course, it doesn't incorporate that information into brew install but maybe there's some logic that could be shared from that PR if/when it gets merged in. At the very least it seems to be tangentially related to this discussion.

[cho-m]

I did have another related P.O.C. (local idea, not a PR) of creating a summed size. Mainly was experimenting with it as part of --dry-run (and also hacking that into some user prompting feature), e.g. rough mock up for install size (download size would just be summing f.bottle.bottle_size)

--- a/Library/Homebrew/install.rb
+++ b/Library/Homebrew/install.rb
@@ -284,17 +285,27 @@ module Homebrew
           end
         end

-        if dry_run
+        if dry_run || prompt
           if (formulae_name_to_install = formulae_to_install.map(&:name))
             ohai "Would install #{Utils.pluralize("formula", formulae_name_to_install.count,
                                                   plural: "e", include_count: true)}:"
             puts formulae_name_to_install.join(" ")

+            formulae = Set.new
             formula_installers.each do |fi|
               print_dry_run_dependencies(fi.formula, fi.compute_dependencies, &:name)
+              formulae.add(fi.formula)
+              formulae.merge(fi.compute_dependencies.flatten.map(&:to_formula))
             end
+            installed_sizes = formulae.map { |f| f.bottle.installed_size }
+            ohai "Total size: #{disk_usage_readable(installed_sizes.sum)}" if installed_sizes.all?(&:positive?)
+          end
+          return if dry_run

---

Anyway, will try to get around to my original PR first. Hopefully will have some more time.

EDIT: Also, should now have more data to work with as all Sequoia bottles will have size information in manifest. Older versions won't.

[osalbahr]

@cho-m awesome! Let me know if you'd like some help, feel free to @ me in the current or a new PR. We're not doing the exact same, but we can build off of a common ground. I like @MikeMcQuaid's idea of adding bottle sizes in JSON API. What is your current hack, and do you have a long-term plan?

@apainintheneck thx for linking the related PR!

[osalbahr]

Oh, disk_usage_readable looks like a cool function. But I first need to figure out how to get the size_in_bytes.

@cho-m btw, how did you get installed_sizes.sum? I can't find it in the main fork or your fork (or global GitHub search). Perhaps you haven't pushed it yet or it's in a private fork.

Edit: ok, just realized it's a variable in the diff you pasted. I should probably read a Ruby book or do an online course (or ask ChatGPT). I welcome recommendations. All I'm getting so far is that the absolute values are some way of Ruby to do a lambda-style for-each and then the sizes are somehow added up. Looks like you're almost there!

[MikeMcQuaid]

@cho-m @osalbahr Note: if/when we're adding more information to the API here: it may be nice to also include sh.brew.path_exec_files from the tab/manifest, too; that'll make https://github.com/homebrew/homebrew-command-not-found much nicer.

[osalbahr]

Since I am just learning Ruby, I think brew info is an easier milestone.

@cho-m what do you think?

---

This is something I wanted to know today, just out of curiosity, on WSL (Ubuntu) brew install ruby is how much more/less storage compared to apt/snap install ruby.

$ brew info ruby
==> ruby: stable 3.3.5 (bottled), HEAD
Powerful, clean, object-oriented scripting language
https://www.ruby-lang.org/
Installed
/home/linuxbrew/.linuxbrew/Cellar/ruby/3.3.5 (19,856 files, 59MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-10-13 at 12:55:08
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/r/ruby.rb
License: Ruby
==> Dependencies
Build: autoconf ✘, pkg-config ✘, rust ✘
Required: libyaml ✔, openssl@3 ✔, gperf ✔, libffi ✔, libxcrypt ✔, zlib ✔
==> Options
--HEAD
        Install HEAD version
==> Caveats
By default, binaries installed by gem will be placed into:
  /home/linuxbrew/.linuxbrew/lib/ruby/gems/3.3.0/bin

You may want to add this to your PATH.

Emacs Lisp files have been installed to:
  /home/linuxbrew/.linuxbrew/share/emacs/site-lisp/ruby
==> Analytics
install: 59,960 (30 days), 186,061 (90 days), 711,848 (365 days)
install-on-request: 22,271 (30 days), 74,546 (90 days), 313,322 (365 days)
build-error: 13 (30 days)

[MikeMcQuaid]

Since I am just learning Ruby, I think brew info is an easier milestone.

@osalbahr Feel free to start with just that.

[osalbahr]

Since I am just learning Ruby, I think brew info is an easier milestone.

@osalbahr Feel free to start with just that.

I appreciate the green light. It lets me know that I am moving in the right direction. Though I don't mind if someone beats me to it.

I created a repo learn Ruby. I will come back to adding the brew info feature later. Or, I might script a working solution in Bash first. I will see.

[osalbahr]

I am trying to add a line under ==> Analytics that says bottle-size: 0 as a placeholder. However I can't figure out where the lines under Analytics come from. Could you help by pointing at which Ruby file is responsible for the list?

I looked at info.rb and I am not sure where the delegation happens.

$ brew info fastfetch | tail
==> Options
--HEAD
        Install HEAD version
==> Caveats
Bash completion has been installed to:
  /home/linuxbrew/.linuxbrew/etc/bash_completion.d
==> Analytics
install: 13,275 (30 days), 31,864 (90 days), 65,133 (365 days)
install-on-request: 13,275 (30 days), 31,864 (90 days), 65,133 (365 days)
build-error: 1 (30 days)

[ZhongRuoyu]

That happens here:

brew/Library/Homebrew/utils/analytics.rb

Line 327 in ad9ea1c

output_analytics(json, args:)

Invoked from here in info.rb:

brew/Library/Homebrew/cmd/info.rb

Line 360 in ad9ea1c

Utils::Analytics.formula_output(formula, args:)

[stephbowie]

/start