Bounty $$: Termux ReactNative Integration issue support

[simbadMarino]

Hello termux community! I hope this is the right place to ask for your help. I'm running some issues while trying to integrate the termux app into a React Native app.

Looks like the installation of packages is done OK however I'm getting this error when lauching TermuxActivity:
exec("/data/data/com.justshare/files/usr/bin/login"): No such file or directory

with the following output in the Run window:

I/com.justshare: Background concurrent copying GC freed 5842(2286KB) AllocSpace objects, 0(0B) LOS objects, 17% free, 27MB/33MB, paused 960us total 540.557ms
I/com.justshare: Background young concurrent copying GC freed 89594(5243KB) AllocSpace objects, 0(0B) LOS objects, 15% free, 28MB/33MB, paused 8.651ms total 268.872ms
W/com.justshare: type=1400 audit(0.0:145): avc: granted { execute } for name="login" dev="vdc" ino=82904 scontext=u:r:untrusted_app_27:s0:c140,c256,c512,c768 tcontext=u:object_r:app_data_file:s0:c140,c256,c512,c768 tclass=file app=com.justshare
W/com.justshare: Accessing hidden field Ljava/io/FileDescriptor;->descriptor:I (greylist, reflection, allowed)

I'm attaching a brief gif showing the problem:

I have tried lots of different alternatives but I'm getting out of ideas honestly as I'm not a mobile dev expert neither totally understand termux code (even after weeks of study lol)

I'm offering a $40 US dollar bounty in crypto ( Monero, Ripple, TRX, BTT or any TRON token) to whoever finds a solution and publish it here first.

I think it should be a really easy to solve issue for you but honestly cannot see the light now and I'm a bit desperate to find a solution now lol.

Link to the complete repo: https://github.com/simbadMarino/dCloud

Let me know if you need additional info. Thanks and I hope I can your support :)

[agnostic-apollo]

You are directly using bootstrap.zip from termux-packages releases. Termux binaries are hardcoded to use and must be installed under the /data/data/com.termux/file/usr/ $PREFIX path. You cannot directly use termux compiled binaries in your app. You need to compile them specially for your app with the /data/data/com.justshare/files/usr prefix. Check following for how to build packages for yourself.

https://github.com/termux/termux-packages/wiki/Building-packages

https://github.com/termux/termux-packages/wiki/Termux-file-system-layout

You are currently also targeting sdk 28 so termux will work for now, but if you plan to publish on playstore or target sdk 29 plus then you will face other issues as well which you fine info for at the following link. In that case using android-10 branch app will be wiser.

https://github.com/termux/termux-packages/wiki/Termux-and-Android-10

[simbadMarino]

Hello @agnostic-apollo , thanks for your support! I've been learning and trying and following the wikis. I'm using your docker termux build environment and managed to create some .deb packages with a customized $PREFIX by running the build-package.sh script. Then I learned that I needed to run the generate-bootstraps.sh script in order to generate the zip file containing the minimal working environment. I edited properties.sh to change the $PREFIX to com.justshare and ran the script, however the script is only generating: etc, tmp and var folders. I was expecting a complete environment like the one available in the termux-app repository (bin, etc, include, lib, libexec, share, tmp, var, SYMLINKs.txt) and the same containing the login script which is the one currently failing to be executed in my app.

As a side note I didn't fully understand the following sentence about having an apt repository in the web server:

I used the https://dl.bintray.com/termux/termux-packages-24 repository as it looked to have more than enough packages for my purposes but not sure if running the script like: ./scripts/generate-bootstraps.sh --repository "https://dl.bintray.com/termux/termux-packages-24" should be enough as that's why I did to generate my bootstrap packages

Would you please advice on how to generate such a complete environment using my customized $PREFIX? I could not identify any section in the wiki talking about that.
Thanks a lot in advance!!

[ghost]

@simbadMarino

1. Compile packages with your custom prefix.
2. Create apt repository and push it to either local or remote web server. (you can use aptly to generate apt repository from given deb files)
3. Run generate-bootstraps.sh script with custom configuration (prefix & url).

It is not possible to use generate-bootstraps.sh without apt repository because it needs some metadata. It is not possible to use official packages to generate bootstrap archives with custom prefix. All three steps shown above are important.

[simbadMarino]

Thanks a lot! I will try those steps now I understood :) Sent from myMail for iOS Saturday, 3 April 2021 10:52 -0600 from ***@***.*** ***@***.***>: ***@***.***Marino

…

* Compile packages with your custom prefix. * Create apt repository and push it to either local or remote web server. * Run generate-bootstraps.sh script with custom configuration (prefix & url). It is not possible to use generate-bootstraps.sh without apt repository because it needs some metadata. It is not possible to use official packages to generate bootstrap archives with custom prefix. All three steps shown above are important. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or unsubscribe .

[agnostic-apollo]

@simbadMarino

Or you can try the following experimental build-bootstraps.sh script to locally cross-compile only the bootstrap package debs and create bootstrap zips. I wrote this a while ago and some stuff has changed in termux build systems by now. However, I have updated the script to work with current system so that you can use it, and it is working for a different PREFIX. There could of course be issues but likely solvable, I only ever did basic testing, didn't get time for more, although openssh works. On-device (termux in mobile) builds are not supported currently, so use PC (cross-compile in docker).

Firstly, you need to pull the latest changes from termux-packages repo, I just committed some stuff. (package source urls got outdated since old versions were removed by respective hosters)

Then

1. Set the TERMUX_APP_PACKAGE value in properties.sh to your app's package name.
   2. Place the build-bootstraps.sh in termux-packages/scripts directory.
2. Run cd termux-packages
3. Run ./scripts/run-docker.sh ./scripts/build-bootstraps.sh &> build.log to compile for all archs. You should find the bootstrap-*.zip in the termux-packages directory. If you want to compile only for a specific arch like aarch64, then run ./scripts/run-docker.sh ./scripts/build-bootstraps.sh --architectures aarch64 &> build.log. You can pass additional comma-separated list of packages that should be included in the bootstrap in addition to the default ones with the -add option.
4. Replace/place the zips in termux-app/app/src/main/cpp.
5. Add a return statement at start of downloadBootstrap() function in termux-app/app/build.gradle so that the custom bootstrap-*.zip files don't get replaced with termux ones.
6. Then build an APK. Untested, but should hopefully work.

It would be better if you ran ./clean.sh before building for a different prefix to start fresh, or pass -f option to build-bootstraps.sh script at least once (for each arch).

If you get curl: (18) transfer closed with x bytes remaining to read. while running build-bootstraps.sh when its downloading a package source, then just run the script again, it should hopefully work.

If you get curl: (22) The requested URL returned error: 410 Gone or something like 404 Not Found while running build-bootstraps.sh when its downloading a package source, then you will have to update the TERMUX_PKG_VERSION, TERMUX_PKG_SRCURL, etc in the build.sh file for the respective package, and fix any *.patch files or apply additional ones if compilation fails, or open an issue in termux-packages.

Good-luck!

build-bootstraps.zip

Edit:

build-bootstrap.sh is officially available in termux-packages repo, check https://github.com/termux/termux-packages/wiki/For-maintainers#build-bootstrap-archives.

[simbadMarino]

@agnostic-apollo I gotta say THANK YOU!!! The bootstrap package was successfully created and then I managed to make it work in my app. @agnostic-apollo , @xeffyr
Kindly share your Crypto address so I can transfer you what I promised (preferible !

I ran into another issue (Hopefully this is the last time I bother you) I got the following error when trying to install a package using dpkg (I built this package using the build-package.sh script and custom $PREFIX):

I even analyzed the dpkg binary in my bootstrap and could not find any hard-coded com.termux directory. I tried modifying different paths in the manifest and some other java classes but still could not find the root cause of the issue.
Do you know what could be causing that?

P.S. If you prefer I could become a Backer instead of the crypto bounty for some time to support the further development of termux, let me know what do you prefer :)

THANKS AGAIN!!

[ghost]

I got the following error when trying to install a package using dpkg

Few possible causes:

• You have built with the default Termux prefix.
• That's all due to lib-dpkg-path-remove.c.patch which still has hardcoded com.termux.
• Something went wrong when building bootstraps (e.g. script picked official deb files).

Run grep -r 'com.termux' $PREFIX to see affected binaries.

[agnostic-apollo]

@xeffyr I don't think the hard coded com.termux is part of the patch in lib-dpkg-path-remove.c.patch, right?

Anyways, @simbadMarino you likely compiled the package with the wrong package name. In my default bootstrap, with additional openssh and sshpass, I compiled with build-package.sh -a aarch64 git and installed git and pcre2 (dependency of git) with dpkg -i pcre2_10.36_aarch64.deb git_2.31.1_aarch64.deb and it installed just fine, even git init works. Moreover, I couldn't fine any hard coded com.termux strings in dpkg and git deb files. So try compiling again I guess.

And you are very welcome! :) I am glad to hear at least the bootstrap worked to start Termux app fork.

About the bounty, the help is usually free on my part so its not an obligation to give it. But if you still wanna give it, you can check out my donation links here. Currently, I haven't set any cryto though, not sure how I will get it in my bank even, so if one of the payment channels in the link work for you, then great, otherwise no issue. As for being a backer for termux, prioritize backing @xeffyr or @Grimler91 since they do a lot more maintenance than me and have done it for far longer.

Thanks

[ghost]

I don't think the hard coded com.termux is part of the patch

Ah, just looked over patches with grep, so assumed that was the case.

Then issue perhaps remains that one or more packages are still compiled with wrong prefix, either dpkg or git package which was attempted for installation with dpkg. All debs have files archived with absolute path (or relative to the Android rootfs).

[agnostic-apollo]

No worries, I was using a text editor.

Yeah, likely, a clean build should hopefully solve the issue.

[simbadMarino]

Hello @xeffyr I sent the XMR rewards this morning ;) (I had to leave my wallet sync for hours lol) and already sent an email to discuss details to you @agnostic-apollo

In regards of the issue, I removed all my current .deb files in the /deb folder and ran the bootstrap build script again, will check the results today afternoon and will come back with the results.

Thanks!

[ghost]

I sent the XMR rewards this morning

I see, thanks.

[agnostic-apollo]

In regards of the issue, I removed all my current .deb files in the /deb folder and ran the bootstrap build script again, will check the results today afternoon and will come back with the results.

Clearing debs is not enough. Already compiled source files will just be repacked as debs again. You need to run clean.sh before building or pass -f to build script so that already built files are cleared first. My build-bootstrap.sh script also supports the -f option, forgot to add text to its --help output.

[simbadMarino]

yep, I also cleaned everything with clean.sh and just being paranoiac added the -f option ;) . Let's see if it created the file succesfully later today :) (fingers crossed)

[agnostic-apollo]

Ah, okay, that's fine then. Good luck. May the C be with you :)

[simbadMarino]

Hey @agnostic-apollo @xeffyr I got the same error when I tried again, however my final use case is to execute an specific binary file and I could finally do it today so I can live with that issue for a while :)

Just in case you notice something in the build log I'm attaching it for you to take a look whenever you have some free time:
build.log

Thanks both again for your great support! I hope I can somehow contribute to termux later on with fixes, mirrors, etc...

[ghost]

From the log file I see that prefix is custom (/data/data/com.justshare/files/usr), so bootstraps are not the problem. But if you still can't install deb files, try to view their content with dpkg-deb -c filename.deb to verify that deb file has correct prefix.

[agnostic-apollo]

@simbadMarino When you built the bootstrap again after clean.sh, did you also install it in your app or just tried to install the git package? The dpkg is part of bootstrap and if if it was initially built with com.termux prefix, then that could be the reason for the error (considering it's working for me). The dpkg as a dependency of apt is one the first packages that is built, so it may have been built with the wrong prefix, while the rest of the bootstrap packages were correct like bash. This is probably an unlikely scenario, not sure if a rebuild would have been triggered by change in install paths, but just putting it out there. Basically build bootstrap again and install it again after clearing prefix, or reinstall app.

If you already reinstalled the bootstrap again and inspecting contents like xeffyr suggested didn't give anything useful, the maybe compile strace package and extract it manually to prefix since dpkg can't be used for installation. I'm not sure if a static version is built as well or not, which would make extracting easier, otherwise get it from here. then run strace dpkg i git.deb and see if that gives any clues.

I am running dpkg in a background shell and not an interactive shell like you are, so there are probably differences in behaviours.

And you are welcome. And thanks for considering future support. Glad your use case at least works.

[simbadMarino]

Hello @xeffyr , @agnostic-apollo , so at the end looks like I'll need to solve the dpkg issue cuz I need to compile on device to get my binaries hehe.
I ran dpkg-deb -c git_2.30.1_x86_64.deb and got the following... (clearly something went wrong when building this package)

drwxr-xr-x builder/builder 0 2021-02-09 05:36 ./
drwxr-xr-x builder/builder 0 2021-02-09 05:36 ./data/
drwxr-xr-x builder/builder 0 2021-02-09 05:36 ./data/data/
drwxr-xr-x builder/builder 0 2021-02-09 05:36 ./data/data/com.termux/
drwxr-xr-x builder/builder 0 2021-02-09 05:36 ./data/data/com.termux/files/
drwx------ builder/builder 0 2021-02-09 05:36 ./data/data/com.termux/files/usr/
drwx------ builder/builder 0 2021-02-09 05:36 ./data/data/com.termux/files/usr/bin/
.....
.....

I will re-run the deb building and try again . Will keep you posted thanks!

[simbadMarino]

Good news!! I rebuild and could install git and pcre:
Looks like indeed cleaning the build environment worked :)

downloads $ dpkg -i pcre2_10.36_x86_64.deb    (Reading database ... 4855 files and directories currently installed.)                          Preparing to unpack pcre2_10.36_x86_64.deb ...  Unpacking pcre2 (10.36) ...                     Setting up pcre2 (10.36) ...                    ~/downloads $ dpkg -i git_2.31.1_x86_64.deb     (Reading database ... 5067 files and directories currently installed.)                          Preparing to unpack git_2.31.1_x86_64.deb ...   Unpacking git (2.31.1) over (2.31.1) ...        Setting up git (2.31.1) ...
~/downloads $ git

Another question... what if I want to manually build for example "clang" but there is no clang build.sh file?

simbad@debian:~/Documents/temp/termux-packages$ ./scripts/run-docker.sh ./build-package.sh -a x86_64 clang
Running container 'termux-package-builder' from image 'termux/package-builder'...
ERROR: No build.sh script at package dir /home/builder/termux-packages/packages/clang!

Is there a way to add the build script to avoid rebuilding the whole bootstrap? (I'll do that anyway in the near future but just for testing purposes )

[ghost]

what if I want to manually build for example "clang" but there is no clang build.sh file?

clang is a subpackage of libllvm.

[agnostic-apollo]

Good news!! I rebuild and could install git and pcre:
Looks like indeed cleaning the build environment worked :)

Phew! Congratz! ;)

[simbadMarino]

hey guys! hope you are doing great! I have 2 questions, everything is running fine with the integration I'm making, however I'd like to improve the way the user will interact with the binary I'm using... so:

1. Is there a way to copy my termux-compiled binary into the /bin folder in the bootstrap zip file so the user can just type the command name and the terminal auto detects it? I have tried the dirty way just by copying the binary file but it is not recognized so I'm suspecting there is a link file somewhere I'm missing
2. How can I modify the welcome prompt? I'd like to either remove it or keep it shorter :) (I've tried to modify the motd file but somehow it is still taking the text from somewhere else

Thanks and regards!!

[agnostic-apollo]

Is there a way to copy my termux-compiled binary into the /bin folder in the bootstrap zip file so the user can just type the command name and the terminal auto detects it?

Can you explain this a bit more. How have you compiled this binary? And are you trying to place it in the bootstrap zip file manually, so that during installation it is automatically extracted to $PREFIX/bin?

have tried the dirty way just by copying the binary file but it is not recognized

Copied where and how, and what does not "not recognized" mean. Copied into the bootstrap or into $PREFIX/bin?

Note that extraction of bootstrap.zip only takes place once during initial installation of APK. Even updates will not re-extract it, unless $PREFIX has been deleted. So if you are updating the bootstrap, then updating (not installing) the app and hoping to find the binary, it will not work. User data is not modified unless user manually runs apt, pkg commands etc, regardless of if the current app version has a new bootstrap or not.

How can I modify the welcome prompt? I'd like to either remove it or keep it shorter :) (I've tried to modify the motd file but somehow it is still taking the text from somewhere else

I think you are again trying to update the app and hoping for the updated motd to show. Not going to work. You can manually modify the motd in the current termux installation with nano $PREFIX/etc/motd, after this new motd will show for new sessions.

If termux-tools is updated in termux-packages repository by maintainers, with say a new motd, each existing user will have to run pkg install termux-tools to get the new motd. New users with only get it, if the termux-app release they are installing has been built with the bootstrap.zip release that contains that version of termux-tools.

[agnostic-apollo]

If you don't want to reinstall termux, you can run termux-reset to clear the $PREFIX so that updated bootstrap is installed.

[simbadMarino]

Is there a way to copy my termux-compiled binary into the /bin folder in the bootstrap zip file so the user can just type the command name and the terminal auto detects it?

Can you explain this a bit more. How have you compiled this binary? And are you trying to place it in the bootstrap zip file manually, so that during installation it is automatically extracted to $PREFIX/bin?

Yes, I'm adding the btfs binary file here before a clean install:

have tried the dirty way just by copying the binary file but it is not recognized

Copied where and how, and what does not "not recognized" mean. Copied into the bootstrap or into $PREFIX/bin?

Yes, ples check picture above :)

Note that extraction of bootstrap.zip only takes place once during initial installation of APK. Even updates will not re-extract it, unless $PREFIX has been deleted. So if you are updating the bootstrap, then updating (not installing) the app and hoping to find the binary, it will not work. User data is not modified unless user manually runs apt, pkg commands etc, regardless of if the current app version has a new bootstrap or not.

I'm doing a clean install and still not getting te btfs copied in the $PREFIX/bin folder after installation

How can I modify the welcome prompt? I'd like to either remove it or keep it shorter :) (I've tried to modify the motd file but somehow it is still taking the text from somewhere else

I think you are again trying to update the app and hoping for the updated motd to show. Not going to work. You can manually modify the motd in the current termux installation with nano $PREFIX/etc/motd, after this new motd will show for new sessions.

I've tried this and worked, will do this in the meantime :)

If termux-tools is updated in termux-packages repository by maintainers, with say a new motd, each existing user will have to run pkg install termux-tools to get the new motd. New users with only get it, if the termux-app release they are installing has been built with the bootstrap.zip release that contains that version of termux-tools.

Thanks for the reply!

[simbadMarino]

If you don't want to reinstall termux, you can run termux-reset to clear the $PREFIX so that updated bootstrap is installed.

That's a nice tip! thanks!

[agnostic-apollo]

Multiple Welcomes!

I'm doing a clean install and still not getting te btfs copied in the $PREFIX/bin folder after installation

Well, that's weird. How are you checking this? ls -A $PREFIX/bin | grep btfs should work. Are you sure you are adding it to the bootstrap matching the architecture of your install target? Otherwise, try adding some Log entries to TermuxInstaller here and see if it gets logged.

And 135MB! I hope you are not planning for a playstore release, cause 100MB is the limit, although, with master branch you can't upload anyways. Not sure if F-Droid has limits. Sideloading should be fine.

[ghost]

I'm doing a clean install and still not getting te btfs copied in the $PREFIX/bin folder after installation

Bootstrap archives is either overwritten during app build time or changes were made to wrong arch. If not, suggesting to clean all caches and intermediate artifacts (./gradlew clean) before attempting new build.

And 135MB! I hope you are not planning for a playstore release, cause 100MB is the limit,

Bootstrap archives have own size limit, specifically their current implementation through shared library blobs.

[agnostic-apollo]

Lolz, a much better laptop. Good luck though.

[Cymmer]

Hi @agnostic-apollo , just set up a powerful VM and it's done building in less than an hour 😆. However, the build still failed. I'm on the latest master branch of termux-packages. I also made changes to build-bootstraps.sh and build-package.sh from your replies.

Here are the steps I made:

1. git clone https://github.com/termux/termux-packages
2. cd termux-packages
3. Edited ./build-package.sh (Bounty $$: Termux ReactNative Integration issue support #1983 (comment))
4. Edited ./scripts/build-bootstraps.sh (Bounty $$: Termux ReactNative Integration issue support #1983 (comment))
5. Ran ./build-package.sh vim (Successful)
6. Ran ./scripts/build-bootstraps.sh (Failed)

Attached is the build.log.
build.log

P.S. I'm really grateful for taking the time to help me. Thank you so much!

[agnostic-apollo]

@Grimler91 libevent package is failing to build with our latest docker. May be fixable with CMAKE flags. Could you or other packages "people" take a look? Thanks.

alicevision/geogram#2 (comment)

https://stackoverflow.com/questions/54587052/cmake-on-mac-could-not-find-threads-missing-threads-found

agnostic-apollo/termux-packages@8548865

https://github.com/agnostic-apollo/termux-packages/runs/8076472144

 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
CMake Error at /home/builder/.termux-build/_cache/cmake-3.24.1/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
  /home/builder/.termux-build/_cache/cmake-3.24.1/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /home/builder/.termux-build/_cache/cmake-3.24.1/share/cmake-3.24/Modules/FindThreads.cmake:230 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:862 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/builder/.termux-build/libevent/build/CMakeFiles/CMakeOutput.log".
See also "/home/builder/.termux-build/libevent/build/CMakeFiles/CMakeError.log".

[Grimler91]

Seems cmake's scripts for finding pthread no longer supports android. I suppose it started with this 2 yo commit, so a bit weird that we have not seen it before.

"Fixed" in termux/termux-packages@af3d7b8.

[agnostic-apollo]

Ah, interesting. Thanks a lot for the fix. @Cymmer you can go ahead and pull latest changes in termux-packages now and hopefully no more errors.

[Cymmer]

Thank you, everyone! @agnostic-apollo I just finished trying to build the bootstraps again. Unfortunately, there's another error, something on inetutils package. Attached is the build.log.

build.log

[agnostic-apollo]

You are just a curse, ain't you!

[Cymmer]

You are just a curse, ain't you!

I'm really sorry hahaha 😆

[agnostic-apollo]

Yeah, I am are really sorry to meet you too! :D

Should be fixed in termux/termux-packages@6e75ec41

And thanks for the bug reports. But one more and insta-block!

And @Grimler91 maybe we should consider building bootstrap packages without the -I flag once in a while or packages when NDK/Ubuntu is bumped.

[Grimler91]

And @Grimler91 maybe we should consider building bootstrap packages without the -I flag once in a while or packages when NDK/Ubuntu is bumped.

Sounds like a good idea. I should probably write a checklist on stuff to do on ndk updates, this can be one of the things to check

[agnostic-apollo]

Cool. Will look into writing a github workflow for build-boostrap.sh which also takes user input for arguments when I work on it.

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onworkflow_dispatchinputs

[Cymmer]

@agnostic-apollo Sorry for the late reply. Been out for a while

And thanks for the bug reports. But one more and insta-block!
awww don't be like that hahahaha.

So far, I just finished the build and it failed :( . But this time it's on me because I just allocated a small amount of storage on my cloud VM hahahaha. So far, it's working already. It has built the bootstraps for all architectures except for x86_64 due to my storage hahaha (see attachment). Will try again later.

[agnostic-apollo]

No need for apologies for late replies.

Ha, serves you right!

Adding an option to run clean.sh before building bootstrap for each arch would be helpful for this kinda thing. Will do that later.

[Cymmer]

Hi @agnostic-apollo , I have successfully built the bootstraps! Thank you so much.

However, when I tried to use the bootstraps in the latest master branch of termux-app, it shows this error during build. Something about Java heap size. Probably because of getting the zip files. Do you have any idea how to solve this?
build-error-log.txt

Steps made:

1. Placed the zip files to termux-app/app/src/main/cpp/. Each zip file size is about 272MB.
2. Edited app's build.gradle to return in downloadBootstrap() task right away.
3. Built the app.

[agnostic-apollo]

Searching on the internet for gradle build OutOfMemoryError should tell you what is wrong. Your bootstraps size is too large, causing allowed memory to run out. You "may" also get similar errors on device, specially low ram ones, while extracting bootstrap to $PREFIX.

[stellarweb008]

Hello @agnostic-apollo, I have been following the instructed steps for a while trying to build the termux bootstraps for my own app, using the ./scripts/run-docker.sh ./scripts/build-bootstraps.sh command.
Many packages have been built as of now. But some seem to be creating sort of a problem. Like bzip2 & curl. When build-bootstraps.sh comes to these, it simply exits saying that the package was "not found in any selected repo."
Just to confirm, that this error is different from one you talked about here-

If you get curl: (22) The requested URL returned error: 410 Gone or something like 404 Not Found while running build-bootstraps.sh when its downloading a package source, then you will have to update the TERMUX_PKG_VERSION, TERMUX_PKG_SRCURL, etc in the build.sh file for the respective package, and fix any *.patch files or apply additional ones if compilation fails, or open an issue in termux-packages.

Because I have checked the packages directory and in it there is no bzip2 or curl. Is this the reason of the error?
Or do I need to build these packages separately like from their debs available on the termux main repo (like this one I found for bzip2)? If yes, then how?

Remember I am new to Termux so I might be slow to respond. Anyways thank you in advance.

[agnostic-apollo]

Firstly, best wait couple of days for fixes for build-bootstrap.sh to be pushed, it currently has lot of issues.

Secondly, post the exact error for "not found in any selected repo.", don't see anywhere in master branch where that error is triggered.

curl isn't found because it is a subpackage of libcurl, build-bootstrap.sh doesn't work for subpackages, but has been fixed locally.

https://github.com/termux/termux-packages/tree/master/packages/libcurl

[stellarweb008]

1. @agnostic-apollo, I actually found the exact error, it's at line number 603 of build-package.sh-

if [ -z "${TERMUX_PKG_BUILDER_DIR}" ]; then
	termux_error_exit "No package $TERMUX_PKG_NAME found in any of the enabled repositories. Are you trying to set up a custom repository?"
fi

I wonder what is TERMUX_PKG_BUILDER_DIR and how it became empty string & why does that condition run only in case of curl & bzip2 only when all others could be successfully built.

Also the build-package.sh & probably all the .sh files that used to be at the root of the termux-packages repo seem to be missing at the moment.

2. Well anyways, I shall try replacing curl with libcurl and contact you later.

But what about bzip2, it seems to be like a core termux package as it is listed as a default in the build-bootstraps.sh (check line 431). And today I have come accross another one xz-utils

3. And what's this?

Firstly, best wait couple of days for fixes for build-bootstrap.sh to be pushed, it currently has lot of issues.

I have a low end pc & I am running build-bootstrap.sh for a week now. Please don't say that it's broken. And that I will have to run a new one. Will I really lose all the progress that's already made & have to start over the whole thing again? 😭

In conclusion, Please help with bzip2 & confirm that whether the progress of my whole long waited patience will be lost...

Edit- I just found libbz2 in the packages directory, shall check that & contact you soon.

[agnostic-apollo]

condition run only in case of curl & bzip2

Because they both are subpackages.

I have a low end pc & I am running build-bootstrap.sh for a week now. Please don't say that it's broken. And that I will have to run a new one. Will I really lose all the progress that's already made & have to start over the whole thing again?

It is what it is. Rebuild would be required, especially while compiling packages for different architectures. But currently, due to broken state unnecessary packages are also getting built, like libllvm which take lot of time to build and bootstrap sizes will be very large. With the fixes, time will be shorter and size will be same as that of generate.

[stellarweb008]

Can I continue with the already generated bootstraps? And xz-utils?

[agnostic-apollo]

You might be able to use already built debs, but you will have to do figure it out yourself, as out of scope for me to help out with an older version of the script. And you can also use github actions to build bootstraps if you have such a slow pc, a workflow will be provided.

[stellarweb008]

Thank you. I decided I shall run it using GitHub actions. But please do me a favour by notifying me when the build-bootstrap.sh is fixed along with informing about the new changes. Nice to meet you @agnostic-apollo.

[agnostic-apollo]

Welcome. Sure.

[stellarweb008]

Anyways can you tell how to execute it using GitHub actions. I was building nodejs for a necessity and it's so slow that it will never finish. It's generating CXX objects and it has to generate 3563 of them, in twenty minutes it's done only 50. In that speed the whole thing would take like 23 hours!! So please help me how to run the GitHub actions. I have already forked the repo. Please prescribe me which action to execute to build only one package- nodejs.

[agnostic-apollo]

You can use something like the generate-bootstraps.sh workflow, but there is no point in doing it now since build-bootstraps.sh doesn't work. Wait for the changes.

https://github.com/termux/termux-packages/blob/master/.github/workflows/bootstrap_archives.yml

nodejs takes ~2hrs to build in github workflow but already built dependencies are used.

https://github.com/termux/termux-packages/actions/runs/9872444546

[stellarweb008]

Sorry to disturb you again @agnostic-apollo, I had decided to think about the bootstraps later & try open a minimal Termux window (although not working due to absence of bootstraps) under my app name org.stellarweb.spider. I had actually merged my app into termux as I had terribly failed in doing the reverse, so I find-replaces all com.termux in the directory (ofcourse kept the implementation com.termux:termux-am-library one) but still it always crashes. I seem to be unable to figure out the problem.

Report Info

User Action: plugin execution command
Sender: TermuxAmSocketServer
Report Timestamp: 2024-07-17 11:47:18.110 UTC

TermuxAm Socket Server Error

Error Code: 154
Error Message (LocalSocket Error):

Create "TermuxAm" server socket failed.
Retval: `-1`
Errmsg:
```
Exception in createServerSocketNative():
java.lang.UnsatisfiedLinkError: No implementation found for org.stellarweb.spider.shared.jni.models.JniResult org.stellarweb.spider.shared.net.socket.local.LocalSocketManager.createServerSocketNative(java.lang.String, byte[], int) (tried Java_org_stellarweb_spider_shared_net_socket_local_LocalSocketManager_createServerSocketNative and Java_org_stellarweb_spider_shared_net_socket_local_LocalSocketManager_createServerSocketNative__Ljava_lang_String_2_3BI) - is the library loaded, e.g. System.loadLibrary?
	at org.stellarweb.spider.shared.net.socket.local.LocalSocketManager.createServerSocketNative(Native Method)
	at org.stellarweb.spider.shared.net.socket.local.LocalSocketManager.createServerSocket(LocalSocketManager.java:127)
	at org.stellarweb.spider.shared.net.socket.local.LocalServerSocket.start(LocalServerSocket.java:100)
	at org.stellarweb.spider.shared.net.socket.local.LocalSocketManager.start(LocalSocketManager.java:86)
	at org.stellarweb.spider.shared.shell.am.AmSocketServer.start(AmSocketServer.java:73)
	at org.stellarweb.spider.shared.termux.shell.am.TermuxAmSocketServer.start(TermuxAmSocketServer.java:108)
	at org.stellarweb.spider.shared.termux.shell.am.TermuxAmSocketServer.setupTermuxAmSocketServer(TermuxAmSocketServer.java:83)
	at org.stellarweb.spider.spider.Termux.launch(MainActivity.java:372)
	at android.os.MessageQueue.nativePollOnce(Native Method)
	at android.os.MessageQueue.next(MessageQueue.java:341)
	at android.os.Looper.loopOnce(Looper.java:176)
	at android.os.Looper.loop(Looper.java:314)
	at android.os.HandlerThread.run(HandlerThread.java:67)

```

TermuxAm Socket Server Run Config

Path: /data/data/org.stellarweb.spider/files/apps/org.stellarweb.spider/termux-am/am.sock
AbstractNamespaceSocket: false
LocalSocketManagerClient: org.stellarweb.spider.shared.termux.shell.am.TermuxAmSocketServer$TermuxAmSocketServerClient
FD: -1
ReceiveTimeout: 10000
SendTimeout: 10000
Deadline: 0
Backlog: 50

Am Command

CheckDisplayOverAppsPermission: true

Spider App Info

APP_NAME: Spider
PACKAGE_NAME: org.stellarweb.spider
VERSION_NAME: 0.118.0
VERSION_CODE: 118
UID: 10323
TARGET_SDK: 28
IS_DEBUGGABLE_BUILD: true
SE_PROCESS_CONTEXT: u:r:untrusted_app_27:s0:c67,c257,c512,c768
SE_FILE_CONTEXT: u:object_r:app_data_file:s0:c67,c257,c512,c768
SE_INFO: default:targetSdkVersion=28:complete
TERMUX_APP_PACKAGE_MANAGER: APT
TERMUX_APP_PACKAGE_VARIANT: APT_ANDROID_7
APK_RELEASE: Github
SIGNING_CERTIFICATE_SHA256_DIGEST: B6DA01480EEFD5FBF2CD3771B8D1021EC791304BDD6C4BF41D3FAABAD48EE5E1

Device Info

Software

OS_VERSION: 4.19.191-perf-g79b043236d77
SDK_INT: 34
RELEASE: 14
ID: UP1A.231005.007
DISPLAY: UP1A.231005.007
INCREMENTAL: V816.0.4.0.UCVINXM
SECURITY_PATCH: 2024-04-01
IS_DEBUGGABLE: 0
IS_TREBLE_ENABLED: true
TYPE: user
TAGS: release-keys
MAX_PHANTOM_PROCESSES: - (Requires DUMP and PACKAGE_USAGE_STATS permission)
MONITOR_PHANTOM_PROCS: true
DEVICE_CONFIG_SYNC_DISABLED: -

Hardware

MANUFACTURER: Xiaomi
BRAND: POCO
MODEL: 22127PC95I
PRODUCT: earth_p_in
BOARD: earth
HARDWARE: mt6768
DEVICE: earth
SUPPORTED_ABIS: arm64-v8a, armeabi-v7a, armeabi

That JniResult file doesn't implement createServerSocketNative (and a lot more like setSocketReadTimeout, available, send, read, accept, closeSocket) anywhere (not even in Termux's own source code) but they are being overriden for sure in LocalSocketManager. So what am I missing?

[agnostic-apollo]

Have you modified this?

termux-app/termux-shared/src/main/cpp/local-socket.cpp

Line 166 in 57e4ef4

jclass clazz = env->FindClass("com/termux/shared/jni/models/JniResult");

And have you changed the package of termux-shared library?

[stellarweb008]

I am very happy to say IT WORKED!! I changed that & a few Java_com_termuxes in some .cpp files & then opened Termux.

But I cheated, instead of building the bootstraps, I took Termux's own bootstraps & find replaced com.termux instances there.

But I didn't work, elf headers got corrupted. Because my app name was org.stearweb.spider, which is much bigger than com.termux, so all sections extended past the size set in headers & nothing opened.

But then I did a final try, I changed org.stellarweb.spider to only com.spider, which has as many letters as com.termux.

I ran the code and here is the result-

Termux running under my app name

THANKS A LOT @agnostic-apollo, WITHOUT YOU I WOULD HAVE NEVER ACHIEVED THIS!! THANK YOU

[stellarweb008]

Hello @agnostic-apollo, this is probably the last time I shall bother you. I had only a final question- when is the 0.119-beta going to be pushed to stable? (just curious to use it for my app, but merging my app into termux takes me like half an hour, replacing a variety of files, etc; also just now i got confused & in the name of updating to 0.119-beta, i replaced my app's termux 0.118 back with the same 0.118 & wasted 30 minutes)

[agnostic-apollo]

You are very welcome and congratz on getting it working. Yeah, same length of package name would have been required for patching bootstrap, but not feasible for everyone.

The 0.119.0-beta.1 branch won't get merged into master branch as it has patch commits. My local changes will directly get merged with master branch and most of the source code has been changed so replacement would have to be done in a different way. A lot of this is being so that termux can be provided as a library in future so that others don't need to make such replacements in future and can just import the library instead, unless doing a full fork. As for timeline, I can't give that as my estimates keep getting changed based on changes in work being done, but currently, the next beta is planned to likely be released in a few weeks and main release some time after that.