Add column.exe to the installed files #92

jankatins · 2016-01-06T09:30:38Z

column.exe is included for the benefit of https://github.com/tj/git-extras
which uses column.exe in git line-summary, git summary and git ignore-io.

It's a 25kb file with no additional dependencies (msys-intl-8.dll,
msys-iconv-2.dll and msys-2.0.dll are all included already). From the
util-linux package...

Closes: git-for-windows/git#586

column.exe is included for the benefit of https://github.com/tj/git-extras which uses column.exe in git line-summary, git summary and git ignore-io. It's a 25kb file with no additional dependencies (msys-intl-8.dll, msys-iconv-2.dll and msys-2.0.dll are all included already). From the util-linux package... Closes: git-for-windows/git#586 Signed-off-by: Jan Schulz <[email protected]>

jankatins · 2016-01-06T10:36:19Z

I built a portable installer and column.exe is included. On the other hand, I had to copy git-subtree.html to the right dir (git-sdk-64\mingw64\share\doc\git-doc) first. Without this file ./portable/release.sh 1.7.0-test wouldn't produce an installer :-(. Not sure why this wasn't there.

jankatins · 2016-01-06T12:27:30Z

Just to understand what I did wrong here:

# install sdk, in the msys2 console
cd /usr/src/git
make install 
make install-html # no git-subtree in /mingw64/share/doc/git-doc
cd ../build-extra
 ./portable/release.sh 1.7.0-test

As far as I understand the subtree problem:

The mingw-w64-x86_64-git-doc-html package specially installs the subtree docs into the git-doc dir: https://github.com/git-for-windows/MINGW-packages/blob/master/mingw-w64-git/PKGBUILD#L187

From there the make-file-list.sh picks it up:

$ pacman -Ql mingw-w64-x86_64-git-doc-html |grep subtree
mingw-w64-x86_64-git-doc-html /mingw64/share/doc/git-doc/git-subtree.html
mingw-w64-x86_64-git-doc-html /mingw64/share/doc/git-doc/howto/using-merge-subtree.html
mingw-w64-x86_64-git-doc-html /mingw64/share/doc/git-doc/howto/using-merge-subtree.txt

On the other hand, neither the Documentation makefile nor the normal install procedure seem to call the contrib/subtree makefile, which installs the docs there.

Does this need a

prefix=make -C contrib/subtree install-doc

after the make install-html or should that be included in the install-doc target?

$ git diff
diff --git a/Makefile b/Makefile
index cee4c67..1c9bc31 100644
--- a/Makefile
+++ b/Makefile
@@ -2372,6 +2372,7 @@ install-man:

 install-html:
        $(MAKE) -C Documentation install-html
+       $(MAKE) -C contrib/subtree install-html

 install-info:
        $(MAKE) -C Documentation install-info

dscho · 2016-01-06T12:39:11Z

I had to copy git-subtree.html to the right dir (git-sdk-64\mingw64\share\doc\git-doc) first.

That should not be necessary if you have mingw-w64-x86_64-git-doc-html installed (which should be the default).

jankatins · 2016-01-06T12:45:18Z

Then I think the problem is, that Documentation/install-webdoc.sh deletes all not-in-Documentation files as the last step:

# after manually installing mingw64/share/doc/git-doc/git-subtree.html
$ make install-html
make -C Documentation install-html
make[1]: Entering directory '/usr/src/git/Documentation'
make[2]: Entering directory '/usr/src/git'
make[2]: 'GIT-VERSION-FILE' is up to date.
make[2]: Leaving directory '/usr/src/git'
'/bin/sh' ./install-webdoc.sh /mingw64/share/doc/git-doc
# rm -f /mingw64/share/doc/git-doc/git-subtree.html
make[1]: Leaving directory '/usr/src/git/Documentation'

dscho · 2016-01-06T13:11:54Z

Then I think the problem is, that Documentation/install-webdoc.sh deletes all not-in-Documentation files as the last step:

Whoa. Indeed. Bummer... I thought that I did not even install the documentation as part of the installation of the SDK...

dscho · 2016-01-06T13:12:58Z

And indeed: I don't. Only the binaries are installed, the documentation is left as-is, so after installing a Git for Windows SDK, things should be ready to go... 😄

Add column.exe to the installed files

dscho · 2016-01-06T13:15:26Z

Thanks!

jankatins · 2016-01-06T14:26:36Z

But e.g. the wiki page for the portable installer recommends to run both make install and make install-html and will therefore fail. https://github.com/git-for-windows/git/wiki/Making-a-portable-Git

column.exe will be available in the next git release: git-for-windows/build-extra#92

column.exe will be available in the next git release: git-for-windows/build-extra#92 Also convert tabs to spaces... Was mixed up to now :-/

dscho · 2016-01-06T14:57:54Z

But e.g. the wiki page for the portable installer recommends to run both make install and make install-html and will therefore fail. https://github.com/git-for-windows/git/wiki/Making-a-portable-Git

Ouch. Would you terribly mind fixing that wiki page?

jankatins · 2016-01-06T14:59:40Z

No, no problem. I can also fix the makefile:

$ git diff
diff --git a/Makefile b/Makefile
index cee4c67..1c9bc31 100644
--- a/Makefile
+++ b/Makefile
@@ -2372,6 +2372,7 @@ install-man:

 install-html:
        $(MAKE) -C Documentation install-html
+       $(MAKE) -C contrib/subtree install-html

 install-info:
        $(MAKE) -C Documentation install-info

dscho · 2016-01-06T17:33:03Z

I can also fix the makefile

No, that would be incorrect. subtree is in contrib/ for a reason: it is not officially supported by upstream Git. We just happen to bundle it with Git for Windows, is all, but the Makefile should stay as close as possible to upstream, unless there are very good reasons not to.

jankatins · 2016-01-06T21:30:00Z

Ok, updated the wiki: https://github.com/git-for-windows/git/wiki/Making-a-portable-Git. That page could probably be renamed to "Making a git package (portable or with installer)", at least the only difference I found was that I had to call /installer/release.sh <version>-test

dscho · 2016-01-07T13:42:11Z

That page could probably be renamed to "Making a git package (portable or with installer)"

I refrained from doing that because I'd rather risk redundant information than to confuse users. Seeing as we target Windows users, and seeing as we use quite a bit of Unix-y workflows, it is confusing enough as it is. I hope you agree?

dscho · 2016-01-07T13:42:52Z

BTW what I meant by "redundant information" is that we have the description how to make an installer in a different wiki page: https://github.com/git-for-windows/git/wiki/Making-an-installer

jankatins · 2016-01-07T14:27:43Z

Oh, i didn't find that page (probably because I just found the "Making a portable installer" as a last entry in the right side box). I will update that page as well... [Update: ok, just found your edit. Thanks! :-)]

Regarding the split: this is actually a good point to merge the pages: only one place to add the info, not multiple places with duplicated instructions. But just correcting it there is probably easy enough. :-)

An idea for merging both pages would be this:

The process to build installer is made easy via the Git for Windows SDK. The steps are basically:

Download and install the SDK, which installs the sources for git, additional mysy and mingw packages and helpers to build the installer and provides the Git for Windows SDK MSys shell
[optional] Hack in the changes you want and make these changes available to the installer.
Call the right helper tools to build the package you want.

Installing the SDK

Download the SDK and run it. You will end up with a console window running the Git for Windows SDK MSys shell, which you can use for the next steps. Git is already checked out and the latest version build and installed, so if you just want to build an installer from latest master, you are ready to go.

You can open the Git for Windows SDK MSys shell by double clicking msys2_shell.bat in the install folder (i.e. not the MinGW 32-bit nor the MinGW 64-bit one).

As the installer is made by putting together files from the Git SDK, it is advisable to update those files first. To update the packages, call pacman -Syu (caveats apply).

Hacking on git

If you only want an installer of the latest version of git (master), then you can skip this step.

The sources of git are checked out in /usr/src/git. Hack in (and test) the changes you want...

Afterwards you need to install git to make it available to the installer build: cd /usr/src/git && make install.

If you made any documentation changes, you need to install the documentation too: make install-html && prefix=/mingw64 make -C contrib/subtree install-html (the second make invocation installs the git-subtree.html which is otherwise missing and leads to an error by the portable installer)

Building an installer

You can build both a normal setup installer and a portable one.

The installers are basically created by collecting files from the Git for Windows SDK Msys environment, not from any checked out source code! It's therefore important to make any changed files you made available to the installer (the make calls in the "Hacking on git" step)!

As a one time step, you need to prepare the extra files which contain the helper scripts:

cd /usr/src/build-extra
git fetch
git checkout master

The next step depends on whether you want a normal setup-based installer or a portable installer:

To build a setup based one:

./installer/release.sh <version>-test

where <version> is the Git version.

To build a portable installer, use:

./portable/release.sh <version>-test

The last line of the log shows where the installer packages were created (usually in your user directory: c:\Users\<login-name>).

See also: Updating the SDK

jankatins · 2016-01-07T14:36:40Z

BTW: as a python user who had to deal with building installer with native code and native external dependencies, I was pleasantly surprised how easy and painless the git build process was...

dscho · 2016-01-07T16:33:52Z

Hmm. Your suggested merged page definitely has more information. On the other hand, it lacks the simplicity of the current two pages.

Besides, I re-worked the way I build the installers and portable Gits myself, by introducing a higher-level script. My idea was to extend said script to perform common tasks, in the future including those that are typically performed by contributors.

For example, a future revision of that script could add support to install the SDK and to build the installer in one go. Something like please.sh make-installer or some such.

Another idea is to extend it to be able to test Pull Requests quickly: please.sh build-pull-request git-for-windows/MSYS2-packages#16. Of course, this would require quite a bit of work and I am not sure whether it is worth it.

jankatins · 2016-01-07T20:11:53Z

With my short experience here, I would have been happier if such a "overview" document like above (+ the overview how the Mingw/Msys packages have to be updated to get into the installer: didn't figure that out yet) would be linked from the "do you want to contribute" page.

re please.sh: I don't think you need more than the above. please.sh seems to be fit to your release work, not a user level function (e.g. I had a hard time figuring out what the individual commands do -> does install git mean that it installs git in my windows system or just a package in the msys environment? I can send a PR for please.sh make-installer, but IMO the instructions are already short enought for a newcomer :-)

I think it's more worth to document how all the bits fit together (that's why I included the overview and the background explanations above) so that someone newcomer has a guideline how to make something work...

jankatins · 2016-01-07T22:29:56Z

Regarding building PRs (and to come full circle): git-extras has a git pr command, which checks out PRs. Not sure if that helps in your workflow. The command is this

#!/usr/bin/env bash
# Based on https://gist.github.com/gnarf/5406589 and https://gist.github.com/jhnns/d654d9d6da6d3b749986

if test "$1" = "clean"; then
  git for-each-ref refs/heads/pr/* --format='%(refname)' | while read ref; do
    git branch -D ${ref#refs/heads/}
  done
  exit 0
elif [[ $1 =~ ^(https?://[^/]+/(.+))/pull/([0-9]+).*$ ]]; then
  remote=${BASH_REMATCH[1]}.git
  id=${BASH_REMATCH[3]}
  branch=pr/$id
else
  test -z $1 && echo "pr number required." 1>&2 && exit 1
  remote=${2:-origin}
  id=$1
  branch=pr/$id
fi
git fetch -fu $remote refs/pull/$id/head:$branch && git checkout $branch

dscho · 2016-01-08T15:53:36Z

git-extras has a git pr command, which checks out PRs. Not sure if that helps in your workflow.

In Git for Windows' context, it gets much more complicated, though. Imagine for example a PR to msys2-runtime. To build that PR,

first of all /usr/src/MSYS2-packages/ has to be checked out.
Then the MSys2 (not the MinGW) shell needs to run.
Then /usr/src/MSYS2-packages/msys2-runtime/PKGBUILD has to be patched to pick up the PR.
Then the package needs to be built.
Then it has to be installed (but quit the MSys2 shell first, otherwise you will have a hard time quitting it when the MSys2 runtime changed underneath it; you will want to use pacman.exe from a cmd to install msys2-runtime for that reason).
Then you have to build the installer (or a portable Git) and of course install it somewhere to test.

I highly doubt that git-extras is up to this task...

dscho · 2016-01-08T15:57:40Z

With my short experience here, I would have been happier if such a "overview" document like above (+ the overview how the Mingw/Msys packages have to be updated to get into the installer: didn't figure that out yet) would be linked from the "do you want to contribute" page.

Oh, by all means, go wild on the wiki! Should I have ideas how to improve on your work, I will present them here. Deal?

I think it's more worth to document how all the bits fit together (that's why I included the overview and the background explanations above) so that someone newcomer has a guideline how to make something work...

Makes sense!

jankatins · 2016-01-08T18:02:42Z

I highly doubt that git-extras is up to this task..

No, it doesn't do that, only fetching and checking out a PR from github into the current git repo.

Oh, by all means, go wild on the wiki!

Ok, I will add my above versions (with links to the individual version, if necessary) to the wiki and try to get a link to that doc from the installer page. Looking at your comment about updating packages, it seems that would also be worthwhile to add to the wiki.

jankatins · 2016-01-08T22:37:57Z

Ok, I've created a "getting started" page: https://github.com/git-for-windows/git/wiki/Getting-started:-building-your-own-git-package. IMO it would be nice if that could be added to the https://git-for-windows.github.io/#contribute section:

[pseudo diff]

<p>The <i>Git for Windows SDK</i> is a build environment that includes all the tools necessary for developers who want to contribute by writing code for Git for Windows. 
+ Please look at the 
+ <a href="https://github.com/git-for-windows/git/wiki/Getting-started:-building-your-own-git-package>
+getting started documentation</a> to get an overview about the git for windows packaging 
+and how to include your changes in your own installer packages.</p>
                            <a name="download-sdk" /><a class="button" href="https://github.com/git-for-windows/build-extra/releases/latest" target="_blank">Download <span class="gittext">Git for Windows SDK</span></a>

Also another question: shouldn't the launch ssh agent page content be replaced by

Use `start-ssh-agent` at the start of your session or include that in your bash startup file.

At least my installer warned my that I have to kill ssh-agent.exe before the install could continue.

dscho · 2016-01-11T09:17:41Z

I had to rename the page so I could check it out on NTFS (the file name contained a colon), and I added the link.

shouldn't the launch ssh agent page content be replaced by

Use start-ssh-agent at the start of your session or include that in your bash startup file.

I think you're correct.

jankatins · 2016-01-11T21:44:29Z

Updated https://github.com/git-for-windows/git/wiki/Auto-launching-ssh-agent-when-git-starts

column.exe will be available in the next git release: git-for-windows/build-extra#92 Also convert tabs to spaces... Was mixed up to now :-/

This was referenced Jan 6, 2016

include column.exe from util-linux package git-for-windows/git#586

Closed

Make the win installer more robust tj/git-extras#496

Merged

jankatins force-pushed the column.exe branch from 5051d1d to 2fc568a Compare January 6, 2016 10:31

dscho added a commit that referenced this pull request Jan 6, 2016

Merge pull request #92 from JanSchulz/column.exe

089ba48

Add column.exe to the installed files

dscho merged commit 089ba48 into git-for-windows:master Jan 6, 2016

jankatins added a commit to jankatins/git-extras that referenced this pull request Jan 6, 2016

Update information about column.exe

93ea173

column.exe will be available in the next git release: git-for-windows/build-extra#92

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add column.exe to the installed files #92

Add column.exe to the installed files #92

jankatins commented Jan 6, 2016

jankatins commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

dscho commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 7, 2016

dscho commented Jan 7, 2016

jankatins commented Jan 7, 2016

jankatins commented Jan 7, 2016

dscho commented Jan 7, 2016

jankatins commented Jan 7, 2016

jankatins commented Jan 7, 2016

dscho commented Jan 8, 2016

dscho commented Jan 8, 2016

jankatins commented Jan 8, 2016

jankatins commented Jan 8, 2016

dscho commented Jan 11, 2016

jankatins commented Jan 11, 2016

Add column.exe to the installed files #92

Add column.exe to the installed files #92

Conversation

jankatins commented Jan 6, 2016

jankatins commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

dscho commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 6, 2016

jankatins commented Jan 6, 2016

dscho commented Jan 7, 2016

dscho commented Jan 7, 2016

jankatins commented Jan 7, 2016

Installing the SDK

Hacking on git

Building an installer

jankatins commented Jan 7, 2016

dscho commented Jan 7, 2016

jankatins commented Jan 7, 2016

jankatins commented Jan 7, 2016

dscho commented Jan 8, 2016

dscho commented Jan 8, 2016

jankatins commented Jan 8, 2016

jankatins commented Jan 8, 2016

dscho commented Jan 11, 2016

jankatins commented Jan 11, 2016