gettext should properly install its m4 macros

[dimpase]

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

install gettext package with its autoconf macros such as AM_ICONV

What happened (include command output)

its autoconf macros don't get installed to a location known to autotools,
even after running

brew link --force gettext

they remain in /usr/local/opt/gettext/share/aclocal/

What you expected to happen

the m4 files get installed in a location known to autotools, e.g.
/usr/local/share/aclocal/

Step-by-step reproduction instructions (by running brew install commands)

This would need a small autotools project to demonstrate an error message such as

configure:666666: error: possibly undefined macro: AM_ICONV

Running brew link --force gettext merely prints

Warning: Refusing to link macOS provided/shadowed software: gettext
If you need to have gettext first in your PATH run:
  echo 'export PATH="/usr/local/opt/gettext/bin:$PATH"' >> ~/.bash_profile

For compilers to find gettext you may need to set:
  export LDFLAGS="-L/usr/local/opt/gettext/lib"
  export CPPFLAGS="-I/usr/local/opt/gettext/include"

giving no clue regarding the fate of m4 files.
While one may go and manually create symlinks to the files in /usr/local/opt/gettext/share/aclocal/ at
/usr/local/share/aclocal/, this ought to happen automatically.

[dimpase]

@mkoeppe

[dimpase]

I'm told that one may add to env. var. ACLOCAL_PATH that /usr/local/opt/gettext/share/aclocal/
-- perhaps that's what can be printed as a hint by brew link --force gettext

[kumowoon1025]

I think this is a problem that should be tackled earlier on the the build system of whatever project, autoreconf, or maybe needs to be libtoolized in a way that makes it more portable.

Because according to GNU autotools docs, you should not expect that there is a location that is universally known by autotools on all systems. It actually uses gettext as an example.

File: automake.info,  Node: Extending aclocal

6.3.3 Writing your own aclocal macros
-------------------------------------
The ‘aclocal’ program doesn’t have any built-in knowledge of any macros,
so it is easy to extend it with your own macros.

   This can be used by libraries that want to supply their own Autoconf
macros for use by other programs.  For instance, the ‘gettext’ library
supplies a macro ‘AM_GNU_GETTEXT’ that should be used by any package
using ‘gettext’.  When the library is installed, it installs this macro
so that ‘aclocal’ will find it.

[dimpase]

I think this is a problem that should be tackled earlier on the the build system of whatever project, autoreconf, or maybe needs to be libtoolized in a way that makes it more portable.

Because according to GNU autotools docs, you should not expect that there is a location that is universally known by autotools on all systems. It actually uses gettext as an example.

While the quoted docs use gettext m4 macros as an example, it does not mean that there should be no easy way to use already installed gettext.

Indeed, how one would go about finding said installation, unless it is in a standard location, and without a specific hint ACLOCAL_PATH? At the moment we have developer after developer hitting this problem on Homebrew. Printing a hint about PATH, CFLAGS, etc, and not printing one for ACLOCAL_PATH is misleading the user into thinking that m4 macros are properly installed.

One can argue that a tip about ACLOCAL_PATH must be printed by brew link --force whenever
it refuses the installation, and prints hints instead, and so this issue illustrates a more general problem.

[kumowoon1025]

While the quoted docs use gettext m4 macros as an example, it does not mean that there should be no easy way to use already installed gettext.

Indeed, how one would go about finding said installation, unless it is in a standard location, and without a specific hint ACLOCAL_PATH? At the moment we have developer after developer hitting this problem on Homebrew. Printing a hint about PATH, CFLAGS, etc, and not printing one for ACLOCAL_PATH is misleading the user into thinking that m4 macros are properly installed.

One can argue that a tip about ACLOCAL_PATH must be printed by brew link --force whenever
it refuses the installation, and prints hints instead, and so this issue illustrates a more general problem.

Well, frankly I find myself in agreement often, where it would be more convenient if autoconf found gnu gettext automatically.

But I think you are missing the point when you say that there should be an easy way to use the already installed gettext.

Which one is that??

The reason gettext is keg-only in the first place is because the system provides bsd gettext by default. Just installing (even if you brew link --force) gettext shouldn't suddenly make its macros available everywhere all the time. gettext is basically a wrapper around a library, if you configure your project to use it, the included macros should be available. But if you follow the instructions in the caveats, gettextize, and then you still have to add /usr/local/opt/gettext/share/aclocal/ to ACLOCAL_PATH manually for things to work, that probably means you hardcoded something else that you shouldn't have.

But again, I am just repeating what apparently "should" be done, and kind of what I feel like is the "correct" or "intended" approach, though it might sound pedantic and irrelevant in most cases, based on a section that almost reads like a petition written by the authors with full knowledge of how very few will listen.

6.3.6 The Future of ‘aclocal’
-----------------------------

‘aclocal’ is expected to disappear.  This feature really should not be
offered by Automake.  Automake should focus on generating ‘Makefile’s;
dealing with M4 macros really is Autoconf’s job.  The fact that some
people install Automake just to use ‘aclocal’, but do not use ‘automake’
otherwise is an indication of how that feature is misplaced.

   The new implementation will probably be done slightly differently.
For instance, it could enforce the ‘m4/’-style layout discussed in *note
Local Macros::.

   We have no idea when and how this will happen.  This has been
discussed several times in the past, but someone still has to commit to
that non-trivial task.

   From the user point of view, ‘aclocal’’s removal might turn out to be
painful.  There is a simple precaution that you may take to make that
switch more seamless: never call ‘aclocal’ yourself.  Keep this guy
under the exclusive control of ‘autoreconf’ and Automake’s rebuild
rules.  Hopefully you won’t need to worry about things breaking; when
‘aclocal’ disappears, because everything will have been taken care of.
If otherwise you used to call ‘aclocal’ directly yourself or from some
script, you will quickly notice the change.

   Many packages come with a script called ‘bootstrap’ or ‘autogen.sh’,
that will just call ‘aclocal’, ‘libtoolize’, ‘gettextize’ or
‘autopoint’, ‘autoconf’, ‘autoheader’, and ‘automake’ in the right
order.  Actually this is precisely what ‘autoreconf’ can do for you.  If
your package has such a ‘bootstrap’ or ‘autogen.sh’ script, consider
using ‘autoreconf’.  That should simplify its logic a lot (less things
to maintain, yum!), it’s even likely you will not need the scriptly
anymore, and more to the point you will not call ‘aclocal’ directlyies
anymore.

   For the time being, third-party packages should continue to install
public macros into ‘/usr/share/aclocal/’.  If ‘aclocal’ is replaced by
another tool it might make sense to rename the directory, but supporting
‘/usr/share/aclocal/’ for backward compatibility should be really easy
provided all macros are properly written (*note Extending aclocal::).

Also, in macOS's case, I think more prudence is necessary when deciding to add such advice even as a note in caveats, because someone installing gnu gettext just to build one project might follow its instructions, or some unpredictable variant of it blindly. It's a lot harder to "break" the system by doing something like this nowadays, but I was somewhat traumatized after trying to transplant gnu tools back in place of llvm after Apple switched the default and an xserve was fubar until a clean reinstall.

So yeah. about 80% of the time I want gnu gettext, closer to 100% if in an autotools built project, which is pretty much all of them except a handful of meson. But when I do, I can usually find the wrappers around it (using brew, for example) and I can't really think of a situation where following the second half of the caveats (adding compiler/linker flags) or adding paths for aclocal to look in would be the optimal approach.

[dimpase]

We merely want to call AM_ICONV in ./configure, of our multiplatform project sagemath.org.

In particular it is meant to build on Linux, in Conda, in Homebrew, etc.

Stand in our shoes for a moment, please, and tell us how are we supposed to figure out what to add to ACLOCAL_PATH, without going and reading what you have in Homebrew formula for gettext, if you keep refusing to even print a hint about ACLOCAL_PATH, leave alone installing m4 macros in a convenient location?

[dimpase]

For the time being, third-party packages should continue to install
public macros into ‘/usr/share/aclocal/’.

hear, hear!

[kumowoon1025]

I'm saying that shouldn't be your problem. Especially since you're using automake, why should you have to know where gettext is installed on every single user's system?

Even automake passes the buck to autoconf, but says it is including aclocal for the time being because it isn't going to be replaced anytime soon. (Or not. I was trying to find some examples for you and found a lot more projects switching to meson from autotools for some reason)

If you're using brew, add gettext as a dependency and run something that will set up the build system for whatever machine you're doing this on, autoreconf, or maybe you have an autogen.sh or other script that will run autopoint from gettext, which will copy or symlink the files.

If this doesn't work, you probably missed a step or two when configuring your project's build system. Did you have m4 installed first, before anything else, then install autoconf, gettext and automake, then configure your project to use each tool? (Because it sounds like it does use all of them) Is there any sign of gettextize ever being run in the build root in the master branch history? I'm starting to doubt if it was.

For the time being, third-party packages should continue to install
public macros into ‘/usr/share/aclocal/’.

hear, hear!

Yeah, if backwards compatibility was the only concern I'm sure that's what would have been done by default for all installs. But there's a huge interoperability issue for macOS users, which compose a majority of brew users, I'm guessing.

   For the time being, third-party packages should continue to install
public macros into ‘/usr/share/aclocal/’.  If ‘aclocal’ is replaced by
another tool it might make sense to rename the directory, but supporting
‘/usr/share/aclocal/’ for backward compatibility should be really easy
provided all macros are properly written (*note Extending aclocal::).

[kumowoon1025]

If you're building with brew, run brew uses gettext and then brew cat <every-other-package-that-uses-gettext> for examples. If not, I would be very confused what the actual issue is.

[dimpase]

Especially since you're using automake

No, we are not. We are using AM_ICONV to check for an installation of iconv.

[dimpase]

If you're building with brew, run brew uses gettext and then brew cat <every-other-package-that-uses-gettext> for examples. If not, I would be very confused what the actual issue is.

In more detail, we want to use, or can use, some packages from brew, and build some other packages ourselves. Thus, in particular, we need to detect the presense of iconv from our ./configure. The most natural way to do this is to use AM_ICONV macros, which happens to come with gettext. And everywhere, expect Homebrew, installing gettext makes AM_ICONV known to autotools. On Homebrew, to the contrary, it's not only not installed in a location known to autotools, Homebrew does not reveal to you where all these m4 macros went to. Clearly there this something to be improved on Homebrew side, right?

[dimpase]

I really do not understand why refusing to reveal to the users how to use the macros you install is the way to go, instead inviting then basically reverse engineer what to do (by reading Ruby code) may be the way to go.

[kumowoon1025]

Okay, so you're one of the people who "install Automake just to use ‘aclocal’, but do not use ‘automake’
otherwise"? Is the project configured to use gettext or are you just expecting the macros to be available? (Is autopoint being run at least?)

Homebrew already tells you the reason for not installing the package fully, because the system gettext provided is the (very different) BSD gettext. If you're using brew on linux, maybe you're right, but I don't know if this is the right place for this issue (maybe Homebrew/linuxbrew-core?)

[kumowoon1025]

I really do not understand why refusing to reveal to the users how to use the macros you install is the way to go, instead inviting then basically reverse engineer what to do (by reading Ruby code) may be the way to go.

You're supposed to read the gettext documentation to find out how to use the macros, what do you think all those binaries are for that come with gettext?

[dimpase]

I really do not understand why refusing to reveal to the users how to use the macros you install is the way to go, instead inviting then basically reverse engineer what to do (by reading Ruby code) may be the way to go.

You're supposed to read the gettext documentation to find out how to use the macros, what do you think all those binaries are for that come with gettext?

Do you think that every developer that uses an autotools project must have read the gettext documentation? Certainly this is not the case, people usually merely want the soft being built, they are ready to read hints provided by brew, but not to do reverse engineering to get things running.

Needless to say, I presume that there is no breakege in using Homebrew's gettext m4 macros with Homebrew's gettext, iconv, etc.

Our ./configure is created by https://github.com/sagemath/sage/blob/develop/bootstrap, which copies config.rpath from whereever gettext installs them ( this is possible as soon as gettext is in PATH, and no, I did not write this, the guy who did it is AWOL :-))
and then it calls aclocal, automake, and autoconf. All this works provided gettext's m4 macros are found.

[kumowoon1025]

Do you think that every developer that uses an autotools project must have read the gettext documentation? Certainly this is not the case, people usually merely want the soft being built, they are ready to read hints provided by brew, but not to do reverse engineering to get things running.

Maybe not everyone who builds your software, but the project maintainer definitely should read the docs if he or she does not know how to use gettext and wants to use it in the project.

Needless to say, I presume that there is no breakege in using Homebrew's gettext m4 macros with Homebrew's gettext, iconv, etc.

Our ./configure is created by https://github.com/sagemath/sage/blob/develop/bootstrap, which copies config.rpath from whereever gettext installs them ( this is possible as soon as gettext is in PATH, and no, I did not write this, the guy who did it is AWOL :-))
and then it calls aclocal, automake, and autoconf. All this works provided gettext's m4 macros are found.

Well you don't need to use Homebrew's gettext, or any gettext at all, if all you need is libiconv. macOS ships with gnu iconv (I think, not 100% sure about this but I think it still ships with BSD gettext and GNU libiconv) If you do need gettext, instead of manually copying config.rpath (which I don't understand how your script does with just gettext in the PATH? Is there a function to print the path to config.rpath??) autopoint should replace the one already in the directory, as it should have been created/copied when the project was gettextized. autopoint is installed to the same directory as gettext, so I guess first try replacing the config.rpath copy step with, or appending an autopoint step afterwards if you want to be able to revert more easily.

Getting gettext and friends included in PATH should be done just by adding gettext as a dependency. I didn't realize this, but maybe you (or your AWOL dev responsible for maintaining the build system...) should maybe heed the warning about aclocal disappearing one day a little more seriously. I wasn't aware (I don't maintain any projects with portable buildsystems) but even glib is using meson.

[dimpase]

Thanks for a pointer, we should look into autopoint. What's discouraging is need for specific versions. Hopefully one can just get away with

AM_GNU_GETTEXT([external])
AM_GNU_GETTEXT_VERSION(0.18)

in configure.ac ?

We currently get the location of config.rpath by scraping content of gettextize (which is merely a script).

Presumably we can do the same for the m4 macros, in particular as gettextize computes
m4dir:

# Check in which directory the *.m4 macros belong.
macrodirs=`"$func_trace" AC_CONFIG_MACRO_DIR_TRACE "$configure_in"`
if test -z "$macrodirs"; then
  macrodirs=`"$func_trace" AC_CONFIG_MACRO_DIR "$configure_in"`
fi
for arg in $macrodirs; do
  m4dir="$arg"
  break
done

@mkoeppe - any thoughts on this?

---

PS. We have about 4K lines of m4 macros, 70% of it written by us, not just copied. And there are templates for Makefiles, etc, too, that I don't count. This is a bit too much to just go and convert to another build system.

[kumowoon1025]

Thanks for a pointer, we should look into autopoint. What's discouraging is need for specific versions. Hopefully one can just get away with

AM_GNU_GETTEXT([external])
AM_GNU_GETTEXT_VERSION(0.18)

in configure.ac ?

We currently get the location of config.rpath by scraping content of gettextize (which is merely a script).

I did not know they were literally shell script wrappers, but doesn't that just feel wrong 😬
I was trying to understand how the project built but I got lost at the bootstrap file, where it suddenly has a branch where it downloads a tarball, and then found out you have an entire library written in python no less, for just building the project.
Can you explain what you mean by "need for specific versions"? The macro needing [external] past a certain point? or that you need a certain version of gettext?

Presumably we can do the same for the m4 macros, in particular as gettextize computes
m4dir:

# Check in which directory the *.m4 macros belong.
macrodirs=`"$func_trace" AC_CONFIG_MACRO_DIR_TRACE "$configure_in"`
if test -z "$macrodirs"; then
  macrodirs=`"$func_trace" AC_CONFIG_MACRO_DIR "$configure_in"`
fi
for arg in $macrodirs; do
  m4dir="$arg"
  break
done

@mkoeppe - any thoughts on this?

PS. We have about 4K lines of m4 macros, 70% of it written by us, not just copied. And there are templates for Makefiles, etc, too, that I don't count. This is a bit too much to just go and convert to another build system.

You have a lot more than that, I agree it's a lot to convert to a non-proprietary build system, but unless you can get the original dev who concocted it to work on it, it's something that you probably want to do sooner than later if you want it to remain manageable and probably more importantly, if you want it to be maintainable by anyone other than the original author. (At least it's working in most linux distros at the moment right?)

I think the issue is that you are trying to use automake macros that come with gettext, when you use neither. (On a side note, apparently Gnu libiconv and gettext have a circular dependence on one another, so you have to build at least one of them twice on system that runs libtoolize on the project and commits/pushes to origin master.) You could just find the system libiconv and headers for Mac, or add it as a dependency and find it using homebrew.

[dimpase]

Can you explain what you mean by "need for specific versions"?

$ autopoint 
autopoint: *** Missing version: please specify in configure.ac through a line 'AM_GNU_GETTEXT_VERSION(x.yy.zz)' the gettext version the package is using
autopoint: *** Stop.

[dimpase]

you have an entire library written in python no less, for just building the project.

I don't know what made you think like this. Sagemath mainly provides interactive capabilities, via Python shell, to a range of C/C++ libraries (e.g. gmp, mprf, NTL, arb, flint - and a range of more obsure ones Brew doesn't have, such as Symmetrica) as well as to "interactive" computer algebra systems such as Maxima, Singular, GAP, Polymake (not in Brew, I guess) etc.
Many interfaces are to libraries and done via Cygwin, a dialect of Python which compiles to C, for efficiency.
Most (but not all) of these systems are built with autotools, few are built with cmake and scons, none is built with meson.

[kumowoon1025]

you have an entire library written in python no less, for just building the project.

I don't know what made you think like this. Sagemath mainly provides interactive capabilities, via Python shell, to a range of C/C++ libraries (e.g. gmp, mprf, NTL, arb, flint - and a range of more obsure ones Brew doesn't have, such as Symmetrica) as well as to "interactive" computer algebra systems such as Maxima, Singular, GAP, Polymake (not in Brew, I guess) etc.
Many interfaces are to libraries and done via Cygwin, a dialect of Python which compiles to C, for efficiency.
Most (but not all) of these systems are built with autotools, few are built with cmake and scons, none is built with meson.

This was as far as I got.

For the version, you can just add the minimum version IIRC, but as a requirement, (e.g.AM_GNU_GETTEXT_REQUIRE_VERSION(...)) so as to make autopoint copy over the latest version, unless only older ones were installed.

P.S.

Many interfaces are to libraries and done via Cygwin, a dialect of Python which compiles to C, for efficiency.

I'm assuming that was a typo/autocorrect and you meant Cython there right?

[dimpase]

you have an entire library written in python no less, for just building the project.

I don't know what made you think like this. Sagemath mainly provides interactive capabilities, via Python shell, to a range of C/C++ libraries (e.g. gmp, mprf, NTL, arb, flint - and a range of more obsure ones Brew doesn't have, such as Symmetrica) as well as to "interactive" computer algebra systems such as Maxima, Singular, GAP, Polymake (not in Brew, I guess) etc.
Many interfaces are to libraries and done via Cygwin, a dialect of Python which compiles to C, for efficiency.
Most (but not all) of these systems are built with autotools, few are built with cmake and scons, none is built with meson.

This was as far as I got.

This is a very minor and small convenience module - we do have our own packaging system (oh, don't ask why a 15-years old project has this :-)) and it turns out that some configuration things for building the main Python module are best packaged this way.

In fact, the majority of our autoconf code is akin to Brew formulae, only in m4, not Ruby. As we don't have a luxury of everything else uniformly packaged (apart from Python libs, but even they are not 100% uniform), it's very ad hoc, e.g. https://github.com/sagemath/sage/blob/develop/build/pkgs/gf2x/spkg-configure.m4

For the version, you can just add the minimum version IIRC, but as a requirement, (e.g.AM_GNU_GETTEXT_REQUIRE_VERSION(...)) so as to make autopoint copy over the latest version, unless only older ones were installed.

P.S.

Many interfaces are to libraries and done via Cygwin, a dialect of Python which compiles to C, for efficiency.

I'm assuming that was a typo/autocorrect and you meant Cython there right?

Oh yes, sorry - even though we do support Cygwin too (or at least try to).

[kumowoon1025]

Oh wow yeah, no I didn't realize it was almost all in python. I'm in way over my head if you're building a python project with autotools and not in conjunction with setuptools (or is it used later on?), I'm guessing it needs to because it's an api to other C, C++, etc api's.

One last fyi I've got is that brew sh will set up an environment like the one you get when you build using brew, you could probably do that instead of sourcing the homebrew environment script. If that's for using brew packages while not using brew for your actual build, you can find things a lot easier if you use brew irb and basically do what you would in an automated build.

But I think we've been off-topic with this issue for quite some time, though I'm curious what direction this will progress in, you'll get better advice than I've got from people more experienced with maintaining projects and their build systems to be portable on everything from Aix to windows elsewhere, maybe the autotools mailing list, or SO, maybe an issue on the project in question, for when your dev shows up again.

[SMillerDev]

I think this isn't a homebrew related issue, closing

[dimpase]

building a python project with autotools and not in conjunction with setuptools (or is it used later on?),

what's built with autotools are mostly libraries to be used in Python extensions (which are built with the usual Python toolchain).

[dimpase]

Sean, "Lees maar, er staat niet wat er staat" :-)

[Bo98]

I think some of the original issue discussed here should happen to be addressed by #53489.