Use git submodules for tilesets, mods and languages

[int-ua]

Problem
Without using --depth option current repository clone already takes almost 2GiB of space. And most of this space is in the .git folder.

Solution
AFAIU its' growth speed can be reduced significantly if tilesets and mods (and optionally l10n files for languages) will be moved into separate repositories by using git submodule. This way when cloning and building yourself you will only download what you actually need. Cons may include additional steps during automated builds.

Edit: and of course setting this all up is additional work. Posting this suggestion just in case dev team haven't considered this solution or didn't have a chance to reconsider lately.

[FONZACUS]

somewhat relevant, id also like to suggest the downloads be smaller as well. like to only ship with english and the current default tileset. if possible along with excluding mods and fonts too.

im not trying to be offensive in the slightest, but i feel sorry for the narc.ro owners, i dont know how much they spend on bandwidth.

[Leland]

This way when cloning and building yourself you will only download what you actually need

FWIW you can just take a shallow clone when you do this

[ZhilkinSerg]

i dont know how much they spend on bandwidth.

http://dev.narc.ro/awstats/awstats.pl?month=03&year=2019&output=downloads&config=dev.narc.ro

[int-ua]

FWIW you can just take a shallow clone when you do this

I'm already doing it for building upstream, of course. But I had problems pushing upstream changes into my development fork from local shallow clone. Do you have a Is there any way to sync local shallow clone of my development fork with upstream?

[mlangsdorf]

git fetch --deepen $NUM will deepen your repo's commit history by $NUM commits.

[int-ua]

git fetch --deepen $NUM will deepen your repo's commit history by $NUM commits.

Tried that . But maybe it wasn't deep enough. Will try again later, thanks.

[mlangsdorf]

You can repeatedly call fetch with deepen and it will deepen it by another $NUM commits each time.

[kevingranade]

I've considered it before, I'm not a fan of using git submodule, but simply moving tiles elsewhere is a possibility.
Also a possibility is splitting the tilesets to a seperate archive for releases. I'm not interested in splitting out the language files.

However, AFAIK this isn't a major problem at this time since we can simply shallow copy as people have outlined.

[int-ua]

Trying to update my fork that is 1336 commits behind:

$ git pull --depth 1400 upstream master
remote: Enumerating objects: 507086, done.
remote: Counting objects: 100% (507086/507086), done.
remote: Compressing objects: 100% (96477/96477), done.
Отримання об’єктів:  62% (313268/503799), 1.31 GiB | 4.69 MiB/s

Looks like --depth here didn't work :/ Am I doing something wrong?

Update:

$ git rev-list --count --first-parent refs/remotes/upstream/master
20980
$ git rev-list --count --first-parent refs/heads/master
20980

This is --depth 1 clone of my fork after syncing with upstream with these commands:

$ git reflog expire --expire=now --all && git repack -ad && git prune
$ du -sh .git
2,2G    .git

Looks like the problem was that I first executed git pull without depth and stopped it, but git saved the unwanted depth already.

[int-ua]

This came up again briefly in #43986 and just for anyone looking here in the future: CI needs this change for git submodules support

--- a/.github/workflows/matrix.yml
+++ b/.github/workflows/matrix.yml
@@ -36,9 +36,10 @@ jobs:
         SANITIZE: ${{ matrix.sanitize }}
     steps:
     - name: checkout repository
-      uses: actions/checkout@v1
+      uses: actions/checkout@v2
       with:
         fetch-depth: 10
+        submodules: true
     - name: install dependencies
       run: |
           sudo apt-get update

[int-ua]

No pressure, just a relevant link: https://gist.github.com/korya/9047870
Blocker: both filter-branch and filter-repo lose track of folder once it was moved at a commit.
Solved: newren/git-filter-repo#182