[OpenGL] Ship OpenGL backend #664

yuanming-hu · 2020-03-27T00:46:04Z

Concisely describe the proposed feature

As we approach v0.6 it's time to think about how to ensure users/developers can easily access the OpenGL backend. I'm setting up testing environments on my GPU matches, yet I find that the OpenGL build process is not portable enough. For example, users may have to apt install libglew-dev libglfw3-dev on Ubuntu.

Describe the solution you'd like (if any)

Fortunately, the two libraries are fast to build ( < 10 seconds) and generate small binaries (< 1MB). So we can simply build from source and statically link them into libtaichi_core.so, just like LLVM.

Added the desired versions of glew and glfw (3.3.2 I guess) as submodules of Taichi.
Integrate their CMakeLists.txt using add_subdirectory. Make sure we use static linking.
Update the buildbots to build OpenGL by default.

Additional comments

Ultimately we might want to remove the dependency on glew and glfw. We only use limited functionalities of these libraries. For example, we only use glfw to create the OpenGL context - not sure if there's an easy way without introducing a dependency. For now, we can stick to the current solution.

The text was updated successfully, but these errors were encountered:

archibate · 2020-03-27T03:55:53Z

And release that in taichi_assets/taichi-glew.zip?

archibate · 2020-03-27T03:57:20Z

...we only use glfw to create the OpenGL context...

How about just make use of glfw to create GUI? It's portable and x11.cpp, win32.cpp, cocoa.cpp... can be abandoned (sadly).

yuanming-hu · 2020-03-27T05:41:09Z

And release that in taichi_assets/taichi-glew.zip?

We are building from source. So we should just include glew and glfw as submodules, like spdlog.

libglew.a will be part of libtaichi_core.so.

archibate · 2020-03-27T05:45:32Z

And release that in taichi_assets/taichi-glew.zip?

We are building from source. So we should just include glew and glfw as submodules, like spdlog.

libglew.a will be part of libtaichi_core.so.

May also include llvm&clang as submodules? Yeah they are big repos, maybe just a dir containing taichi-llvm.zip?

archibate · 2020-03-27T05:46:13Z

I think we just need exactly one submodule: taichi_assets can done all this, and make CI happy :)

yuanming-hu · 2020-03-27T05:46:41Z

May also include llvm&clang as submodules?

Unfortunately, no. LLVM itself takes > 1h to build on the CI machines...

archibate · 2020-03-27T05:51:52Z

May also include llvm&clang as submodules?

Unfortunately, no. LLVM itself takes > 1h to build on the CI machines...

It's ok since we just use their binaries :)

archibate · 2020-03-27T06:22:40Z

I mean, create a repo, contains taichi-llvm-8.0.1-linux-x64.zip, taichi-llvm-8.0.1.zip, etc., then include that repo as a submodule.

yuanming-hu · 2020-03-27T16:30:00Z

I mean, create a repo, contains taichi-llvm-8.0.1-linux-x64.zip, taichi-llvm-8.0.1.zip, etc., then include that repo as a submodule.

Yeah we used to have something similar: https://github.com/yuanming-hu/taichi_runtime

The issues are

Since we have to ship LLVM binaries for all platforms in that repo, the repo can get super large. This is an abuse of git (which is used to version control source instead of binaries)
A walkaround is to use different branches for different platforms. Then we can't make use of submodules, such only sticks to a certain commit.

We can, of course, write a script to download corresponding LLVM binaries adaptively. For GLFW/GLEW, since they are lightweight, we can simply include them as submodules and build from source.

(I'm not sure if there's gonna be any build issues of GL libraries on Travis/AppVeyot where no GPU/OpenGL is available. But let's see...)

yuanming-hu · 2020-03-27T16:31:33Z

...we only use glfw to create the OpenGL context...

How about just make use of glfw to create GUI? It's portable and x11.cpp, win32.cpp, cocoa.cpp... can be abandoned (sadly).

The reason we use our own GUI system is to avoid portability issues of OpenGL: Taichi's GUI systems work on machines without OpenGL.

archibate · 2020-03-27T16:54:36Z

Also found http://www.itgo.me/a/2256112962039115366/request-a-specific-opengl-context-version-in-glfw3:

You cannot create an OpenGL context using "plain OpenGL." You have to use a window system API such as WGL, GLX, CGL, EGL, etc. These APIs were all extended around the time GL 3.0 was released to introduce this functionality. If you don't want to be bothered writing platform-specific code, you should consider a framework like GLFW3.

archibate · 2020-03-27T16:56:17Z

...we only use glfw to create the OpenGL context...

How about just make use of glfw to create GUI? It's portable and x11.cpp, win32.cpp, cocoa.cpp... can be abandoned (sadly).

The reason we use our own GUI system is to avoid portability issues of OpenGL: Taichi's GUI systems work on machines without OpenGL.

How about to distribute taichi-opengl-nightly and taichi-nightly seperately?
May also have taichi-opengl-nvidia-nightly and taichi-opengl-mesa-nightly to treat the problem in #666?

yuanming-hu · 2020-03-27T17:05:00Z

...we only use glfw to create the OpenGL context...

How about just make use of glfw to create GUI? It's portable and x11.cpp, win32.cpp, cocoa.cpp... can be abandoned (sadly).

The reason we use our own GUI system is to avoid portability issues of OpenGL: Taichi's GUI systems work on machines without OpenGL.

How about to distribute taichi-opengl-nightly and taichi-nightly seperately?
May also have taichi-opengl-nvidia-nightly and taichi-opengl-mesa-nightly to treat the problem in #666?

I think that's too many packages to release and will bring troubles to users/developers. Ultimately we should end up with a single taichi-nightly package with all backends instead of many packages each with a subset of the backends. I'll open a separate issue for this.

archibate · 2020-04-10T17:14:48Z

...glfw and glew as submodules...

https://github.com/glfw/glfw
https://github.com/nigels-com/glew

They are both big repos, surely will delay the clone&building process...
Like compiler crazyers will definitely have their own clone of llvm and clang on their disk.
So does GPU crazyers do, I believe almost all users (even those end users) have glew/glfw installed.
I suggest not to include them but to let user install it manually. Is that ok?

archibate · 2020-04-11T09:28:36Z

users may have to apt install libglew-dev libglfw3-dev on Ubuntu.

Can we include it in pip dep files? (Sorry I'm not expert at this...)
Or add a line for this in README.md just like libtinfo5?

yuanming-hu · 2020-04-11T14:57:08Z

Actually, glfw-3.2.2 takes only 7.85 seconds to build on my end with 2 CPU cores. They are C libraries and they compile very fast.

Requiring users to install anything is harder than we think. Taichi used to depend on gcc-7 and 99% of the users don't know how to install that.

archibate · 2020-04-12T06:42:07Z

Actually, glfw-3.2.2 takes only 7.85 seconds to build on my end with 2 CPU cores. They are C libraries and they compile very fast.

Wow, then how about the clone time?
We may tell contributors how to deal with shallow submodules in dev_install.rst?

...and 99% of the users don't know how to install that.

Agree, and I had a hardtime downloading clang-7 since it's not in Archlinux's official packages.
The apt-get install clang-7 didn't tell me what to do if not Ubuntu. We may also add a link to clang's repo in dev_install.rst?

Btw, is OpenGL okay now on win? I didn't have a env to test that...

yuanming-hu · 2020-04-12T13:55:28Z

Wow, then how about the clone time?

I'm in US and cloning takes < 2s. I do understand people in China might suffer from poor network conditions...

We may tell contributors how to deal with shallow submodules in dev_install.rst?

Right, git submodule init and then git submodule update --depth 1 should do a shallow clone for you. This will be helpful for people in mainland China.

Btw, is OpenGL okay now on win? I didn't have a env to test that...

I'll test it on Windows once GLFW and GLEW are included as submodules.

Agree, and I had a hardtime downloading clang-7 since it's not in Archlinux's official packages.
The apt-get install clang-7 didn't tell me what to do if not Ubuntu. We may also add a link to clang's repo in dev_install.rst?

It would be great to concisely tell people how to install clang-7 on Arch there!

archibate · 2020-04-12T15:56:36Z

git submodule init

Do we need run this after git clone taichi? Or this is automatically done with a fresh clone? I want to add it into dev_install.rst build command list.

It would be great to concisely tell people how to install clang-7 on Arch there!

See #747, I added (If on Arch Linux) right after the Ubuntu one.

yuanming-hu · 2020-04-12T16:02:00Z

Do we need run this after git clone taichi? Or this is automatically done with a fresh clone? I want to add it into dev_install.rst build command list.

We do. Thanks!

archibate · 2020-04-12T16:04:39Z

Do we need run this after git clone taichi? Or this is automatically done with a fresh clone? I want to add it into dev_install.rst build command list.

We do. Thanks!

See f19f228 to check if I'm understanding right.

yuanming-hu · 2020-04-12T16:07:28Z

You need git submodule init first.

archibate · 2020-04-12T16:10:00Z

I did, I tried any combination of these, and no work done :(

archibate · 2020-04-12T16:17:15Z

$ git submodule add https://github.com/glfw/glfw external/glfw --depth=1
正克隆到 '/home/bate/Develop/taichi/external/glfw'...
remote: Enumerating objects: 25743, done.
接收对象中:   3% (805/25743), 188.01 KiB | 23.00 KiB/s

Happy cloning now! Thanks to the poor network conditions, it may takes 5m+ to clone all of it.

archibate · 2020-04-12T16:27:04Z

I'm in US and cloning takes < 2s.

Sounds so kidding... In GFW it's common to clone for hours and you don't dear to shutdown the computer until it's done...why no 断点续传 support in git...now I knows: maybe the US users don't have such problem at all...

archibate · 2020-04-12T16:32:10Z

My considerations:

What if user already have glfw&glew installed? Will our glfw overwrite their install?
What if user don't want TI_WITH_OPENGL, they should not be forced cloning it?

Also found there is only libglfw.so in my /lib, so in order to do static linkage, build from scratch is needed.

archibate · 2020-04-12T16:33:43Z

What if user already have glfw&glew installed? Will our glfw overwrite their install?

You can only do make all but no make install in glfw. Then we just use the glfw/build/libglfw.a inside.

What if user don't want TI_WITH_OPENGL, they should not be forced cloning it?

Will the submodule clone all? No, taichi_assets for example, is not clone&updated by default.

archibate · 2020-04-12T16:34:15Z

(gldep) [bate@archit taichi]$ git submodule add https://github.com/glfw/glfw external/glfw --depth=1
正克隆到 '/home/bate/Develop/taichi/external/glfw'...
remote: Enumerating objects: 25743, done.
remote: Total 25743 (delta 0), reused 0 (delta 0), pack-reused 25743
接收对象中: 100% (25743/25743), 12.28 MiB | 12.00 KiB/s, 完成.
fatal: 不能打开 /home/bate/Develop/taichi/.git/modules/external/glfw/objects/pack/tmp_pack_eqmEFX: 没有那个文件或目录
fatal: index-pack 失败
fatal: 无法克隆 'https://github.com/glfw/glfw' 到子模组路径 '/home/bate/Develop/taichi/external/glfw'

Why!!!

archibate · 2020-04-12T16:35:22Z

I give up submodule. It's wise to let user install glfw on their own, like libtinfo does.
Btw, libtinfo is also a FAQ, could we add libtinfo.a later?

yuanming-hu · 2020-04-12T16:47:40Z

What if user already have glfw&glew installed? Will our glfw overwrite their install?

It will not overwrite since we are only using the compiled static libraries. In fact, we should not use the users' glfw and glew, since there's no control on their own glf/ew versions. For example, depending on users' gcc/clang verseion has lead to tons of issues that are hard to fix. E.g. #745

What if user don't want TI_WITH_OPENGL, they should not be forced cloning it?

Once we enable shallow clone the total data to be fetched should be <3MB.

Note that we do not need libtinfo-dev for development, but we do need GLFW and GLEW for compiling your OpenGL backend.

What if user already have glfw&glew installed? Will our glfw overwrite their install?

You can only do make all but no make install in glfw. Then we just use the glfw/build/libglfw.a inside.

Right, we do not need to make install since we only need the static lib.

What if user don't want TI_WITH_OPENGL, they should not be forced cloning it?

Will the submodule clone all? No, taichi_assets for example, is not clone&updated by default.

There might be some extra setup. We should probably actually add the submodules when committing. See https://github.com/taichi-dev/taichi/tree/master/external - there is a spdlog @ cf6f1dd folder.

I understand that the network condition in China is probably very poor. If it's too hard I can try to set it up for you next week.

archibate · 2020-04-12T16:55:12Z

I would suggest not to include them as submodules, but let user manually clone or download them like llvm.

yuanming-hu · 2020-04-12T16:58:45Z

I would suggest not to include them as submodules, but let user manually clone or download them like llvm.

My feeling is that will further complexify the development installation process. We have no choice for LLVM since LLVM is too big. GLFW and GLEW are just tiny libraries.

archibate · 2020-04-12T17:05:02Z

Please download this glew-2.2.0-src.tgz file:
https://sourceforge.net/projects/glew/files/glew/snapshots/glew-20200115.tgz/download
And upload it to taichi_assets, thanks!

archibate · 2020-04-12T17:06:51Z

GLFW and GLEW are just tiny libraries.

Yes, GLFW build succeed. But...the current master branch of GLEW cannot build at all :(the file src/glew.c is missing)
I don't think including a bad-maintained repo to taichi is a good idea :(

archibate · 2020-04-12T17:08:36Z

fyi pls do special care for glew, i.e. let them download the buildable snapshot package: https://sourceforge.net/projects/glew/files/glew/snapshots/glew-20200115.tgz/download.
ttyl have to sleep cu tmr!

yuanming-hu · 2020-04-12T17:10:01Z

Sounds good - thanks for the hard work. I'll take a look at this in the afternoon. Good night!

archibate · 2020-04-12T17:10:45Z

since there's no control on their own glf/ew versions

That's what cmake to slove...

yuanming-hu · 2020-04-12T21:10:08Z

GLFW and GLEW are just tiny libraries.

Yes, GLFW build succeed. But...the current master branch of GLEW cannot build at all :(the file src/glew.c is missing)
I don't think including a bad-maintained repo to taichi is a good idea :(

We should not use the HEAD of the master branch but the commit with version 2.1.0 instead. Master branches are sometimes unstable unless rules are made.

#760 introduces GLFW as a submodule and this will save the developers from manually installing it.

You are right that GLEW seems slightly out-dated. ~~It's sad that they don't even use cmake (I did not realize this, sorry) so integration can be tricky.~~

For now, a temporary fix is to:

In the documentation (Developer Installation), ask developers to install GLEW, and to set up the headers and static libraries correctly.

Modify the buildbot scripts to download GLEW include and static libraries.

Ultimately we want to remove the dependency on OpenGL API loaders completely, using a similar approach as #756. This is more C++ coding work (a few hundred LoC I guess, making use of DynamicLoader, given we only use a limited part of OpenGL API), but will greatly simplify documentation, improve the developer installation experience, and simplify buildbot setups.

I'm not sure which one between temporary fix and ultimate solution you prefer. The decision is on you. (Yes, portability/shipping is hard. It took me weeks to ship the CUDA backend in a satisfactory state...)

archibate · 2020-04-13T01:50:46Z

I found git submodule update --init --recursive --depth=1 in CI, and this is prob what users want to have in dev_install.rst.

archibate · 2020-04-13T02:16:29Z

Ultimately we want to remove the dependency on OpenGL API loaders completely...

I know you're thinking about something like:

   auto glCompileShader = dlsym("libGLEW.so", "glCompileShader");

However, this is actually impossible since GLEW already done the wraping work for us:

(yuanming-hu/glfw) [bate@archit taichi]$ nm /lib/libGLEW.so | grep glCompileShader
(yuanming-hu/glfw) [bate@archit taichi]$ nm /lib/libGLEW.so | grep CompileShader
00000000000d6418 B __glewCompileShader
00000000000d7ae0 B __glewCompileShaderARB
00000000000d7c58 B __glewCompileShaderIncludeARB
(yuanming-hu/glfw) [bate@archit taichi]$ cat /usr/include/GL/glew.h | grep CompileShader
#define glCompileShader GLEW_GET_FUN(__glewCompileShader)
#define glCompileShaderARB GLEW_GET_FUN(__glewCompileShaderARB)
#define glCompileShaderIncludeARB GLEW_GET_FUN(__glewCompileShaderIncludeARB)
GLEW_FUN_EXPORT PFNGLCOMPILESHADERPROC __glewCompileShader;
GLEW_FUN_EXPORT PFNGLCOMPILESHADERARBPROC __glewCompileShaderARB;
GLEW_FUN_EXPORT PFNGLCOMPILESHADERINCLUDEARBPROC __glewCompileShaderIncludeARB;

Now you may tell me not to use GLEW, directly dlsym from GL. But to directly load from libGL is much harder than you think, different versions/APIs/profiles/extensions, only GLEW can care those dedicated works well and unify them.

The best sol is to statically-link-against GLEW, it's already a DynamicLoader itself :)

yuanming-hu · 2020-04-13T02:30:50Z

Now you may tell me not to use GLEW, directly dlsym from GL. But to directly load from libGL is much harder than you think, different versions/APIs/profiles/extensions, only GLEW can care those dedicated works well and unify them.

Right, that's why I estimate the implementation to be a few hundred lines of code instead of tens of lines of code...

Let see if glew works with CMake, if so then we are good.

archibate · 2020-04-18T10:48:08Z

TODO: move to GLAD.

archibate · 2020-04-23T07:46:40Z

I think we've done here, and v0.6.0 is all done except some little PyPI specs?

yuanming-hu · 2020-04-24T20:44:46Z

We are very very close. Just one final thing (as also mentioned in #843 (comment)) can we make with_opengl return false on a system where OpenGL is not detected? To my knowledge, the OpenGL backend might cause a crash on systems without OpenGL (e.g. the AppVeyor machines). We can release v0.5.15 and then v0.6 once this is done.

yuanming-hu added the feature request Suggest an idea on this project label Mar 27, 2020

yuanming-hu assigned archibate Mar 27, 2020

archibate mentioned this issue Mar 27, 2020

[infra] Add Ubuntu build for Travis CI #653

Closed

archibate added welcome contribution dependency and removed welcome contribution labels Mar 27, 2020

archibate mentioned this issue Mar 27, 2020

[OpenGL] Support NVIDIA GLSL compiler #666

Merged

archibate mentioned this issue Apr 6, 2020

[OpenGL] 64-bit data type support #717

Merged

yuanming-hu mentioned this issue Apr 12, 2020

[opengl] GLFW 3.3.2 is now included as a submodule #760

Merged

archibate mentioned this issue Apr 13, 2020

[opengl] add GLEW 2.1.0 as a submodule #768

Closed

archibate mentioned this issue Apr 19, 2020

[OpenGL] use GLAD as API loader (replace GLEW) #819

Merged

archibate mentioned this issue Apr 25, 2020

[opengl] Detect driver existance in with_opengl() #864

Merged

archibate linked a pull request Apr 25, 2020 that will close this issue

[opengl] Detect driver existance in with_opengl() #864

Merged

archibate added the opengl OpenGL backend label Apr 25, 2020

yuanming-hu closed this as completed in #864 Apr 26, 2020

[OpenGL] Ship OpenGL backend #664

[OpenGL] Ship OpenGL backend #664

Comments

yuanming-hu commented Mar 27, 2020

archibate commented Mar 27, 2020

archibate commented Mar 27, 2020

yuanming-hu commented Mar 27, 2020

archibate commented Mar 27, 2020

archibate commented Mar 27, 2020

yuanming-hu commented Mar 27, 2020

archibate commented Mar 27, 2020

archibate commented Mar 27, 2020

yuanming-hu commented Mar 27, 2020

yuanming-hu commented Mar 27, 2020

archibate commented Mar 27, 2020

archibate commented Mar 27, 2020

yuanming-hu commented Mar 27, 2020

archibate commented Apr 10, 2020

archibate commented Apr 11, 2020

yuanming-hu commented Apr 11, 2020

archibate commented Apr 12, 2020 • edited Loading

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020 • edited Loading

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020

archibate commented Apr 12, 2020 • edited Loading

yuanming-hu commented Apr 12, 2020

archibate commented Apr 12, 2020

yuanming-hu commented Apr 12, 2020 • edited Loading

archibate commented Apr 13, 2020

archibate commented Apr 13, 2020 • edited Loading

yuanming-hu commented Apr 13, 2020

archibate commented Apr 18, 2020

archibate commented Apr 23, 2020

yuanming-hu commented Apr 24, 2020

archibate commented Apr 12, 2020 •

edited

Loading

archibate commented Apr 12, 2020 •

edited

Loading

archibate commented Apr 12, 2020 •

edited

Loading

yuanming-hu commented Apr 12, 2020 •

edited

Loading

archibate commented Apr 13, 2020 •

edited

Loading