Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: Speed up third party libraries cloning #28567

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
+66 −40
Open

Build: Speed up third party libraries cloning #28567

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
84e535a
build: speed up third party libraries cloning by using `no-deps-git-h…
difhel Oct 23, 2024
2154526
docs: update docs
difhel Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 51 additions & 40 deletions Telegram/build/prepare/prepare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ def nativeToolsError():
'qt6',
'skip-release',
'build-stackwalk',
'no-deps-git-history'
]
options = []
runCommand = []
Expand Down Expand Up @@ -326,6 +327,17 @@ def run(commands):
else:
return subprocess.run("set -e\n" + commands, shell=True, env=modifiedEnv).returncode == 0

no_deps_git_history = "no-deps-git-history" in options
if no_deps_git_history:
print("Notice: Some dependency repositories will be downloaded without full history (via git clone --depth 1) to reduce time and disk space.")

def git(command, args):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does what I asked, but not the way other things are done here. It looks to me that more consistent with how the script already works would be another approach:

  • if 'no-deps-git-history' is passed as an option you add something like 'DEPTH' -> '--depth 1' to the 'environment' map, otherwise set it to an empty string
  • in all places necessary just add $DEPTH in the git command so that the '--depth 1' is added if necessary..

If this way it works as expected, then I'd like that approach to be implemented. If somehow I missed something and this doesn't work then we'll proceed with the one you proposed.

result = f"git {command}"
if no_deps_git_history:
result += " --depth 1"
result = result + " " + args
return result

# Thanks https://stackoverflow.com/a/510364
class _Getch:
"""Gets a single character from standard input. Does not echo to the
Expand Down Expand Up @@ -500,9 +512,9 @@ def runStages():
del jom.zip
""", 'ThirdParty')

stage('depot_tools', """
stage('depot_tools', f"""
mac:
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
{git('clone', 'https://chromium.googlesource.com/chromium/tools/depot_tools.git')}
cd depot_tools
./update_depot_tools
""", 'ThirdParty')
Expand All @@ -529,18 +541,18 @@ def runStages():
make $MAKE_THREADS_CNT
""", 'ThirdParty')

stage('lzma', """
stage('lzma', f"""
win:
git clone https://github.com/desktop-app/lzma.git
{git('clone', 'https://github.com/desktop-app/lzma.git')}
cd lzma\\C\\Util\\LzmaLib
msbuild -m LzmaLib.sln /property:Configuration=Debug /property:Platform="$X8664"
release:
msbuild -m LzmaLib.sln /property:Configuration=Release /property:Platform="$X8664"
""")

stage('xz', """
stage('xz', f"""
!win:
git clone -b v5.4.5 https://github.com/tukaani-project/xz.git
{git('clone', '-b v5.4.5 https://github.com/tukaani-project/xz.git')}
cd xz
sed -i '' '\\@check_symbol_exists(futimens "sys/types.h;sys/stat.h" HAVE_FUTIMENS)@d' CMakeLists.txt
CFLAGS="$UNGUARDED" CPPFLAGS="$UNGUARDED" cmake -B build . \\
Expand Down Expand Up @@ -574,8 +586,8 @@ def runStages():
make install
""")

stage('mozjpeg', """
git clone -b v4.1.5 https://github.com/mozilla/mozjpeg.git
stage('mozjpeg', f"""
{git('clone', '-b v4.1.5 https://github.com/mozilla/mozjpeg.git')}
cd mozjpeg
win:
cmake . ^
Expand Down Expand Up @@ -611,8 +623,8 @@ def runStages():
cmake --install build
""")

stage('openssl3', """
git clone -b openssl-3.2.1 https://github.com/openssl/openssl openssl3
stage('openssl3', f"""
{git('clone', '-b openssl-3.2.1 https://github.com/openssl/openssl openssl3')}
cd openssl3
win32:
perl Configure no-shared no-tests debug-VC-WIN32 /FS
Expand Down Expand Up @@ -732,9 +744,9 @@ def runStages():
make install
""")

stage('gas-preprocessor', """
stage('gas-preprocessor', f"""
win:
git clone https://github.com/FFmpeg/gas-preprocessor
{git('clone', 'https://github.com/FFmpeg/gas-preprocessor.git')}
cd gas-preprocessor
echo @echo off > cpp.bat
echo cl %%%%%%** >> cpp.bat
Expand Down Expand Up @@ -810,7 +822,7 @@ def runStages():
""")

stage('openh264', """
git clone -b v2.4.1 https://github.com/cisco/openh264.git
""" + git('clone', '-b v2.4.1 https://github.com/cisco/openh264.git') + """
cd openh264
win32:
SET "TARGET=x86"
Expand Down Expand Up @@ -870,8 +882,8 @@ def runStages():
lipo -create build.aarch64/libopenh264.a build.x86_64/libopenh264.a -output ${USED_PREFIX}/lib/libopenh264.a
""")

stage('libavif', """
git clone -b v1.0.4 https://github.com/AOMediaCodec/libavif.git
stage('libavif', f"""
{git('clone', '-b v1.0.4 https://github.com/AOMediaCodec/libavif.git')}
cd libavif
win:
cmake . ^
Expand Down Expand Up @@ -901,8 +913,8 @@ def runStages():
cmake --install . --config MinSizeRel
""")

stage('libde265', """
git clone -b v1.0.15 https://github.com/strukturag/libde265.git
stage('libde265', f"""
{git('clone', '-b v1.0.15 https://github.com/strukturag/libde265.git')}
cd libde265
win:
cmake . ^
Expand Down Expand Up @@ -939,7 +951,7 @@ def runStages():
""")

stage('libwebp', """
git clone -b v1.4.0 https://github.com/webmproject/libwebp.git
""" + git('clone', '-b v1.4.0 https://github.com/webmproject/libwebp.git') + """
cd libwebp
win:
nmake /f Makefile.vc CFG=debug-static OBJDIR=out RTLIBCFG=static all
Expand Down Expand Up @@ -978,8 +990,8 @@ def runStages():
cmake --install build
""")

stage('libheif', """
git clone -b v1.17.6 https://github.com/strukturag/libheif.git
stage('libheif', f"""
{git('clone', '-b v1.17.6 https://github.com/strukturag/libheif.git')}
cd libheif
win:
%THIRDPARTY_DIR%\\msys64\\usr\\bin\\sed.exe -i 's/LIBHEIF_EXPORTS/LIBDE265_STATIC_BUILD/g' libheif/CMakeLists.txt
Expand Down Expand Up @@ -1030,8 +1042,8 @@ def runStages():
cmake --install . --config MinSizeRel
""")

stage('libjxl', """
git clone -b v0.10.3 --recursive --shallow-submodules https://github.com/libjxl/libjxl.git
stage('libjxl', f"""
{git('clone', '-b v0.10.3 --recursive --shallow-submodules https://github.com/libjxl/libjxl.git')}
cd libjxl
""" + setVar("cmake_defines", """
-DBUILD_SHARED_LIBS=OFF
Expand Down Expand Up @@ -1080,11 +1092,10 @@ def runStages():
cmake --install . --config MinSizeRel
""")

stage('libvpx', """
git clone https://github.com/webmproject/libvpx.git
stage('libvpx', f"""
{git('clone', '-b v1.14.1 https://github.com/webmproject/libvpx.git')}
depends:patches/libvpx/*.patch
cd libvpx
git checkout v1.14.1
win:
for /r %%i in (..\\patches\\libvpx\\*) do git apply %%i

Expand Down Expand Up @@ -1149,7 +1160,7 @@ def runStages():

stage('liblcms2', """
mac:
git clone -b lcms2.16 https://github.com/mm2/Little-CMS.git liblcms2
""" + git('clone', '-b lcms2.16 https://github.com/mm2/Little-CMS.git liblcms2') + """
cd liblcms2

buildOneArch() {
Expand All @@ -1174,17 +1185,17 @@ def runStages():
lipo -create build.arm64/liblcms2.a build/liblcms2.a -output ${USED_PREFIX}/lib/liblcms2.a
""")

stage('nv-codec-headers', """
stage('nv-codec-headers', f"""
win:
git clone -b n12.1.14.0 https://github.com/FFmpeg/nv-codec-headers.git
{git('clone', '-b n12.1.14.0 https://github.com/FFmpeg/nv-codec-headers.git')}
""")

stage('regex', """
git clone -b boost-1.83.0 https://github.com/boostorg/regex.git
stage('regex', f"""
{git('clone', '-b boost-1.83.0 https://github.com/boostorg/regex.git')}
""")

stage('ffmpeg', """
git clone -b n6.1.1 https://github.com/FFmpeg/FFmpeg.git ffmpeg
""" + git('clone', '-b n6.1.1 https://github.com/FFmpeg/FFmpeg.git ffmpeg') + """
cd ffmpeg
win:
depends:patches/ffmpeg.patch
Expand Down Expand Up @@ -1362,10 +1373,10 @@ def runStages():
make install
""")

stage('openal-soft', """
stage('openal-soft', f"""
version: 3
win:
git clone -b wasapi_exact_device_time https://github.com/telegramdesktop/openal-soft.git
{git('clone', '-b wasapi_exact_device_time https://github.com/telegramdesktop/openal-soft.git')}
cd openal-soft
cmake -B build . ^
-A %WIN32X64% ^
Expand All @@ -1390,14 +1401,14 @@ def runStages():
""")

if 'build-stackwalk' in options:
stage('stackwalk', """
stage('stackwalk', f"""
mac:
git clone https://chromium.googlesource.com/breakpad/breakpad stackwalk
cd stackwalk
git checkout dfcb7b6799
depends:patches/breakpad.diff
git apply ../patches/breakpad.diff
git clone -b release-1.11.0 https://github.com/google/googletest src/testing
{git('clone', '-b release-1.11.0 https://github.com/google/googletest src/testing')}
git clone https://chromium.googlesource.com/linux-syscall-support src/third_party/lss
cd src/third_party/lss
git checkout e1e7b0ad8e
Expand All @@ -1407,13 +1418,13 @@ def runStages():
xcodebuild -project processor.xcodeproj -target minidump_stackwalk -configuration Release build
""")

stage('breakpad', """
stage('breakpad', f"""
git clone https://chromium.googlesource.com/breakpad/breakpad
cd breakpad
git checkout dfcb7b6799
depends:patches/breakpad.diff
git apply ../patches/breakpad.diff
git clone -b release-1.11.0 https://github.com/google/googletest src/testing
{git('clone', '-b release-1.11.0 https://github.com/google/googletest src/testing')}
win:
SET "PYTHONUTF8=1"
SET "FolderPostfix="
Expand Down Expand Up @@ -1539,7 +1550,7 @@ def runStages():
stage('qt_' + qt, """
git clone -b v$QT-lts-lgpl https://github.com/qt/qt5.git qt_$QT
cd qt_$QT
git submodule update --init --recursive --progress qtbase qtimageformats qtsvg
""" + git('submodule update', '--init --recursive --progress qtbase qtimageformats qtsvg') + """
depends:patches/qtbase_""" + qt + """/*.patch
cd qtbase
win:
Expand Down Expand Up @@ -1631,9 +1642,9 @@ def runStages():
else: # qt > '6'
branch = 'v$QT' + ('-lts-lgpl' if qt < '6.3' else '')
stage('qt_' + qt, """
git clone -b """ + branch + """ https://github.com/qt/qt5.git qt_$QT
""" + git('clone', '-b ' + branch + ' https://github.com/qt/qt5.git qt_$QT') + """
cd qt_$QT
git submodule update --init --recursive qtbase qtimageformats qtsvg
""" + git('submodule update', '--init --recursive qtbase qtimageformats qtsvg') + """
depends:patches/qtbase_""" + qt + """/*.patch
cd qtbase
mac:
Expand Down
5 changes: 5 additions & 0 deletions docs/building-mac.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ Go to ***BuildPath*** and run
git clone --recursive https://github.com/telegramdesktop/tdesktop.git
./tdesktop/Telegram/build/prepare/mac.sh

> [!IMPORTANT]
> You might not need Telegram Desktop's git dependency history, in which case you can run this command **instead of the last one** and save disk space and speed up cloning quite a bit:
>
> ./tdesktop/Telegram/build/prepare/mac.sh no-deps-git-history

### Building the project

Go to ***BuildPath*/tdesktop/Telegram** and run (using [your **api_id** and **api_hash**](#obtain-your-api-credentials))
Expand Down
5 changes: 5 additions & 0 deletions docs/building-win-x64.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ Open **x64 Native Tools Command Prompt for VS 2022.bat**, go to ***BuildPath***
git clone --recursive https://github.com/telegramdesktop/tdesktop.git
tdesktop\Telegram\build\prepare\win.bat

> [!IMPORTANT]
> You might not need Telegram Desktop's git dependency history, in which case you can run this command **instead of the last one** and save disk space and speed up cloning quite a bit:
>
> tdesktop\Telegram\build\prepare\win.bat no-deps-git-history

## Build the project

Go to ***BuildPath*\\tdesktop\\Telegram** and run (using [your **api_id** and **api_hash**](#obtain-your-api-credentials))
Expand Down
5 changes: 5 additions & 0 deletions docs/building-win.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ Open **x86 Native Tools Command Prompt for VS 2022.bat**, go to ***BuildPath***
git clone --recursive https://github.com/telegramdesktop/tdesktop.git
tdesktop\Telegram\build\prepare\win.bat

> [!IMPORTANT]
> You might not need Telegram Desktop's git dependency history, in which case you can run this command **instead of the last one** and save disk space and speed up cloning quite a bit:
>
> tdesktop\Telegram\build\prepare\win.bat no-deps-git-history

## Build the project

Go to ***BuildPath*\\tdesktop\\Telegram** and run (using [your **api_id** and **api_hash**](#obtain-your-api-credentials))
Expand Down