Modular design for build artifacts

[Spasi]

LWJGL currently has several bindings, with more coming in the future. Many users will only use a subset of those, but they all come in a single, monolithic archive. We should consider updating the build process to create separate archives for each binding, that can be downloaded and deployed independently. This should be in addition to the fat archive that will still be available, for convenience.

We may only do this for large APIs, especially those that come with their own shared libraries (currently jemalloc, GLFW, OpenAL and soon hwloc). The new design should ensure that loading the shared libraries is as easy/simple as before.

[ShadowLordAlpha]

This would go hand in hand with the gradle build script but you could try subprojects. That way it would also handle any dependencies that they have on other parts of the lib though it might need some refactoring of the codebase (I would work on it but i still don't really get how the generator works and don't have time to sit down and mess with it right now).

[httpdigest]

After some discussing, here is what my current perspective of a modular build is with respect to deployment:

Motivation
Just to reiterate what Spasi said above in the first post:

LWJGL currently has several bindings, with more coming in the future. Many users will only use a subset of those, but they all come in a single, monolithic archive. We should consider updating the build process to create separate archives for each binding, that can be downloaded and deployed independently. This should be in addition to the fat archive that will still be available, for convenience.

So, from the deployment perspective, I as a user want to choose from the list of possible libraries/modules the ones I would like to use. The result should be that I only have to download the chosen modules and their transitive dependencies, but nothing else.

Project setups

This now raises the question how to go about users with different project setups and build tools. Maven users will likely want to add the modules as Maven dependencies in their pom.xml. Together with the SharedLibraryLoader which extracts the native shared libraries from the classpath, this will work out of the box. This then necessitates that all modules are Maven artifacts being deployed in a public Maven repository.
For Gradle users it's basically the same.

LWJGL Initializr

Non-Maven/Gradle users will ideally be provided with a distribution zip file that can be downloaded from lwjgl.org, as is the case now with the fat distribution. For this, some kind of web application is desirable that lets the user choose which modules should be included in the distribution and then provide an assembled zip download for that, much like the Spring Initializr for Spring Boot.

So, we have a scenario where dependency management is handled by the build tool (Maven/Gradle) and a scenario where we have to bundle all artifacts into the download zip.

Building

To satisfy the Maven/Gradle users, the existing deployment mechanism with Gradle and generated Maven pom.xml files can still be used. Every module will get a pom.xml that describes that module and its dependencies.

For non-Maven users who download a customized zip file, we need a way to automatically provide a selection of available modules to the user. Preferably, the user will be shown a web UI which asks him which version of LWJGL to use and based on that, which modules with which versions are available.
For this to work, the same Maven project comprehension can be used.

Maven project structure

The basis is a parent Maven project. This project (probably named org.lwjgl:lwjgl) uses "pom" packaging and simply enumerates all modules as optional dependencies.
Each module has a main pom.xml with packaging "jar" that describes its Java classes artifact.

In addition to the Java classes artifact, this artifact has profile-dependent dependencies to its native platform artifacts containing the respective shared libraries for the particular platform (windows/linux/osx).
This way, when a Maven user adds a single dependency to org.lwjgl:lwjgl-opengl (example for the OpenGL bindings) he will also get the correct platform jar as dependency.
Each profile can then be activated via Maven profile OS activation rules so that Maven users only download the platform jar they need.
If a user wants to deploy his application for all platforms, he can just enable all profiles explicitly during a Maven build.

The pom.xml files for each module can probably be generated just like now via the Gradle build script. The only dependency so far is that every module will depend on the lwjgl-core module.

Non-Maven Users

Non-Maven users will be given a web UI to select and download a customized distribution zip file.
For this, we will reuse the already established Maven project structure.
The project Eclipse Aether (see demos) can be used to query a Maven repository for all available versions of a particular artifact and enumerate its sub-modules.
With proper pom.xml files describing the modules, their dependencies and versions, the complete selection UI can be generated (the available modules and their versions need not be maintained manually!), also respecting possible dependencies between modules. (when user selects module A then also module B must be used, because B is a (transitive) dependency of A).

To build the distribution with binaries and sources, either all of the selected modules' artifacts can be downloaded and packaged into a single distribution zip, or the modules' artifacts can be unpacked and repackaged into a single binary and source jar. The native shared libraries can also either be provided with their platform jar files or extracted and repackaged in a "natives" directory.

Technically, the actual download can be triggered via a single GET URL containing as query parameters the selected modules, such as:
http://lwjgl.org/starter/download.zip?version=3.0.1&platform=windows&modules=core,stb,opengl,glfw
Behind that, the server-side of the web application can assemble the correct zip distribution and stream it as download.

Version management

Currently, to ease version management, whenever a new version of LWJGL appears, all modules' versions will be updated accordingly to point to the same new version.

Release management

Just like now, every new release will build all module artifacts and deploy them to Maven Central. Like mentioned above, whenever a non-Maven user wants to obtain a new release distribution, he selects the modules he wants to use and will get a customized distribution.

[Spasi]

The latest nightly (3.0.1 build 6) is the first build with modular artifacts. Please give it a go and provide any feedback you may have here.

This build is currently available as a monolithic .zip and only via lwjgl.org. Each module is contained in a subfolder (named after the corresponding package), with different JARs for classes, source, javadoc and (for some) per-platform natives.

Maven support is coming next, when we're happy with the new structure. The site will also be upgraded to support downloading individual modules and a tool will be added that:

• Generates Maven/Gradle scripts with the modules you select.
• Generates IDE projects with the modules you select.
• Packages the modules you select and initiates a download of a .zip with just those modules.

[j-ibarra]

why not put the native libraries in a single * .jar , to facilitate application portability

for generator explained here you may have the option to generate only the *.jar needed for developers, example: I need only generate module GLFW or GL or GLFW+GL!

[Spasi]

why not put the native libraries in a single * .jar , to facilitate application portability

What do you mean exactly? Having a single lwjgl-<module>-natives.jar per module instead of one per platform?

example: I need only generate module GLFW or GL or GLFW+GL!

That's coming, either via Maven/Gradle or the tool I mentioned above.

[httpdigest]

When we eventually have the personalized/modularized build download available, in my opinion there won't be any need anymore for downloading separate platform native jars and even for separating the Java classes jars from the platform jars.
Everything can be downloaded as a single lwjgl.jar. This currently would be my perferred solution for non-Maven and non-Gradle users.
Maven users on the other hand can/should use the Shade plugin to create an "uber" jar for final distribution/deployment containing everything in a single jar.
Gradle users can use the Shadow plugin (similar to Maven's Shade plugin) to achieve the same.
For all those solutions to work we however need separated native platform jars and separate Java classes jars.
Depending on the build system the users uses, the point of consolidation just changes between either download-time (for IDE users) or build-time for Gradle/Maven users.

[j-ibarra]

why not put the native libraries in a single * .jar , to facilitate application portability

What do you mean exactly? Having a single lwjgl--natives.jar per module instead of one per platform?

i see the archives: lwjgl-glfw-natives-linux.jar, lwjgl-glfw-natives-macos.jar, lwjgl-glfw-natives-windows.jar, why no put all natives in only archive*.jar? (lwjgl-glfw-natives.jar)

Some time ago work with JOGL, and I put libraries in a single *.jar as follows:

• Native
  • Windows
    • x86
    • x64
  • Linux
    • x86
    • x64
  • Mac
    • Univelsal

something other than the subject, have not thought lwjgl3 made ​​compatible with Android

[httpdigest]

why no put all natives in only archive*.jar? (lwjgl-glfw-natives.jar)

The zip download currently provided on the lwjgl.org/download page is just one way of assembling a distribution, which currently chooses to separate the platforms' natives and the Java classes into separate jars for each module.
And you apparently want to have all platforms' natives in a single jar. This is certainly doable.
It will be possible to download such a combined jar from the lwjgl.org/download site soon.
We should not merge all platforms' natives too soon in the build process, though, because depending on the deployment mechanism a user chooses for his/her application, he/she may want to distribute the native jars separately for each platform.

[Spasi]

why no put all natives in only archive*.jar?

A few reasons:

• If LWJGL is added as a Maven dependency and we have separate JARs per platform, the download would be quicker (2-4 times faster, depending on the platform). The first time experience for new users matters. LWJGL is also updated quite regularly and downloading unnecessary stuff every time is not ideal.
• If an LWJGL application is distributed via Java Web Start, having separate JARs per platform means quicker download time for all users.
• If an LWJGL application is packaged in native installers, one per platform, only the corresponding natives for each platform would be added to the installer. If everything came in a single JAR, the developer would have to separate the natives manually.

With that said, the tool on lwjgl.org/download will support merging the natives, so I think you'll be satisfied.

something other than the subject, have not thought lwjgl3 made ​​compatible with Android

ARM support is scheduled for LWJGL 3.0.2. Android support will also be explored.

[ghost]

Are you considering easy integration of LWJGL in SWT applications? I managed this with LWJGL 2, but have had a few issues with LWJGL 3. Know there's a project on Github to accomplish this.

[ghost]

https://github.com/httpdigest/lwjgl3-swt link to project.

[httpdigest]

Certainly. We could have an "external modules" section on the lwjgl.org/download site, where you can select lwjgl3-swt to be included in the download for all project types (Maven, Gradle, IDE).
Please note that SWT already brings support for OpenGL and perfect integration with LWJGL3.
Just create a GLCanvas and afterwards do GL.createCapabilities() and you're set.
However, on OS X you only get a "legacy" 2.1 context and you cannot have a multisampled default framebuffer.

[dustContributor]

What do you mean? I can use a core context with SWT on Linux right now? Or you're saying it works as long as you use a compatibility profile?

Because OSX isnt the only one that doesn't provides compatibility profiles, Intel's Linux drivers don't support them either.

[Spasi]

Modular maven artifacts are now available in the snapshot repository.

I'm looking for feedback on the current structure. We now have the lwjgl artifact and one artifact for each binding (lwjgl-glfw, lwjgl-opengl, lwjgl-stb, etc). Each artifact contains the following:

• The java classes
• The java sources (classifier: sources)
• The API javadoc (classifier: javadoc)
• The native libraries for each platform (optional, some bindings have no native code)
  • Linux (classifier: natives-linux)
  • MacOS (classifier: natives-macos)
  • Windows (classifier: natives-windows)

The above structure has the benefit that a user may download exactly the artifacts they need and nothing else. On the other hand, since Maven has no concept of dependencies between classified artifacts, setting it up is quite verbose. For example a project that needs GLFW, stb, jemalloc and OpenGL, using the shortcut Gradle syntax:

// lwjglVersion = '3.0.1-SNAPSHOT'
// lwjglArch = 'windows'
compile 'org.lwjgl:lwjgl:$lwjglVersion' // NOTE: this is optional, all binding artifacts have a dependency on lwjgl
compile 'org.lwjgl:lwjgl-glfw:$lwjglVersion'
compile 'org.lwjgl:lwjgl-stb:$lwjglVersion'
compile 'org.lwjgl:lwjgl-jemalloc:$lwjglVersion'
compile 'org.lwjgl:lwjgl-opengl:$lwjglVersion'
runtime 'org.lwjgl:lwjgl:$lwjglVersion:natives-$lwjglArch'
runtime 'org.lwjgl:lwjgl-glfw:$lwjglVersion:natives-$lwjglArch'
runtime 'org.lwjgl:lwjgl-stb:$lwjglVersion:natives-$lwjglArch'
runtime 'org.lwjgl:lwjgl-jemalloc:$lwjglVersion:natives-$lwjglArch'

Our options:

Option A

Keep the above structure. We're soon going to have a tool on the LWJGL website that will let users choose the bindings they need and it will generate the corresponding Gradle/Maven declarations for them. Verbose, but auto-generated.

Option B

Don't use classifiers and make the natives separate artifacts with appropriate dependencies. The above example becomes:

// lwjglVersion = '3.0.1-SNAPSHOT'
// lwjglArch = 'windows'
compile 'org.lwjgl:lwjgl-glfw-$lwjglArch:$lwjglVersion'
compile 'org.lwjgl:lwjgl-stb-$lwjglArch:$lwjglVersion'
compile 'org.lwjgl:lwjgl-jemalloc-$lwjglArch:$lwjglVersion'
compile 'org.lwjgl:lwjgl-opengl:$lwjglVersion'

The above would automatically resolve the other dependencies declared in Option A. I personally don't like this option for aesthetic reasons (too many artifacts under org.lwjgl).

Option C

We could assume that most projects will use only a few of the available bindings and download speed is not a concern (Maven dependencies is a convenience for developers only, not end-users). The natives for all platforms could be merged into the unclassified artifact. The above example becomes:

// lwjglVersion = '3.0.1-SNAPSHOT'
compile 'org.lwjgl:lwjgl-glfw:$lwjglVersion'
compile 'org.lwjgl:lwjgl-stb:$lwjglVersion'
compile 'org.lwjgl:lwjgl-jemalloc:$lwjglVersion'
compile 'org.lwjgl:lwjgl-opengl:$lwjglVersion'

As simple as possible to configure, but at the price of downloading bigger artifacts.

[arisona]

The ongoing modularisation ist very welcome! At the same time, for those who'd just like to hack away, it would be useful to have something like "lwjgl-all" which would give you an all-in-one package.

[Spasi]

it would be useful to have something like "lwjgl-all" which would give you an all-in-one package.

The website tool is going to have a "I want everything" option, which will give you dependency declarations for all bindings. IMHO, that sufficiently covers users that simply want to try what LWJGL offers.

Having said that, a particularly annoying aspect of having all bindings in a project is autocompletion of OpenGL APIs in the IDE; LWJGL supports both standard OpenGL and OpenGL ES and they share a lot of identically named functions, constants and extensions. It's very common to invoke autocompletion and get duplicate classes/method/constants in the list. It's annoying and you have to be careful with selecting the element you want from the correct package.

[kappaOne]

Just updated nightly builds, something seems to have broken on my mac system (OSX 10.11) between "LWJGL 3.0.1 build 13" and "LWJGL 3.0.1 build 17" with OpenAL, the following calls now return just '0', whereas previously they returned proper values:

int bytes = AL10.alGetBufferi(buffer, AL10.AL_SIZE);
int bits = AL10.alGetBufferi(buffer, AL10.AL_BITS);
int channels = AL10.alGetBufferi(buffer, AL10.AL_CHANNELS);

Same code with build 17 works fine on linux.

[ShadowLordAlpha]

It should be possible to have the native library depend on the java one and, while technically not the correct use of dependencies, should allow you to do Option A but with only half of the declared dependencies.

Also it seems that LWJGL is unable to find the natives itself currently when using the new system.

[Spasi]

Just updated nightly builds, something seems to have broken on my mac system (OSX 10.11)

I tracked this down to what seems like a bug in LLVM's optimizer. I had recently changed our CI build to use Xcode 8 (was on 6.1 before), which produced a buggy build. I've now reverted it to Xcode 7.2. You can download a fixed OpenAL build here, it will also be included in the next LWJGL nightly.

[Spasi]

It should be possible to have the native library depend on the java one and, while technically not the correct use of dependencies, should allow you to do Option A but with only half of the declared dependencies.

Note that with Option A the native library is resolved using a classifier. The Java classes, sources, javadoc and native libraries (one for each platform) are all part of the same artifact. E.g. for GLFW the complete artifact is:

// Java classes
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'

// Java sources
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'
classifier: 'sources'

// Javadoc
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'
classifier: 'javadoc'

// Linux Natives
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'
classifier: 'natives-linux'

// MacOS Natives
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'
classifier: 'natives-macos'

// Windows Natives
groupId: 'org.lwjgl'
artifactId: 'lwjgl-glfw'
version: '3.0.1-SNAPSHOT'
classifier: 'natives-windows'

Afaik, you cannot specify dependencies between secondary artifacts that belong to the same artifactId.

Also it seems that LWJGL is unable to find the natives itself currently when using the new system.

Have you added the native JARs to the classpath? If yes, could you share more details on how you've set up your project? It would also be useful to see the output when running with -Dorg.lwjgl.util.Debug=true and -Dorg.lwjgl.util.DebugLoader=true.

[ShadowLordAlpha]

Sorry it was me derping with the new system and forgetting to import the proper native file for it to actually make the link though. I didn't have it importing the base dependencies native file and that what was causing my problem.

[octylFractal]

I like Option A, and I could even write a simple Gradle plugin to allow you to do code similar to Option C:

// +natives signals to the plugin that there is natives
// Might be possible to allow something like +natives(osx,linux)
compile 'org.lwjgl:lwjgl-glfw:$lwjglVersion+natives'
compile 'org.lwjgl:lwjgl-stb:$lwjglVersion+natives'
compile 'org.lwjgl:lwjgl-jemalloc:$lwjglVersion+natives'
// opengl has no natives, no +natives
compile 'org.lwjgl:lwjgl-opengl:$lwjglVersion'

Edit: The plugin might even be able to do natives detection, though I fear it might not always be 100% accurate.

[kappaOne]

I've now reverted it to Xcode 7.2. You can download a fixed OpenAL build here, it will also be included in the next LWJGL nightly.

Just updated to LWJGL 3.0.1 build 18 and can confirm it works great on macOS now.

[apostolos]

We just deployed a first version of the new download page and it includes the Maven & Gradle script generators ( only for the nightly build at the moment ).

It is still a work-in-progress but we would appreciate your feedback!

Commit:
LWJGL/lwjgl3-www@86c6471

[sriharshachilakapati]

@apostolos I think you should allow multiple natives too, as one might want a cross platform configuration.

And the version is 3.0.1-SNAPSHOT but is generated as 3.0.0-SNAPSHOT I think, for the nightly.

[kappaOne]

Looks great, would love to be able to download custom zip bundles.

[apostolos]

@sriharshachilakapati Just fixed the nightly version. Thanks

I've discussed multiple natives with @Spasi but it makes no sense for Gradle & Maven. You can just generate a new script for your "other" computer.

For the ZIP bundle it will be possible to select one or more platform and include them all in a single download.

[j-ibarra]

Looks excellent!

[sriharshachilakapati]

I think multiple natives do actually make sense. Consider a scenario where me and another guy working on the same project using LWJGL, and we are on different OS. We do want to have the gradle script on the repository, and changing it on one system will cause the new script to go with commit (of course, we can ignore it, but..) making it a bit of a headache maintaining.

Instead, making the radio buttons into check boxes will suit everyone. In case other platforms are unnecessary, the user can of course un-check them right?

[apostolos]

@kappaOne Started work on the bundles. To tell you the truth I'm kind of worried about server performance if we get simultaneous requests from multiple people.

I might offload everything on Lambda since we are on Amazon and artifacts are stored on S3 already. Unfortunately we cannot really cache generated files on the CDN since potential bundle combinations are thousands.

[kappaOne]

Would merging smaller jars/zips (possibly the maven ones) on the client side (using maybe a javascript zip libraries) be a viable option to create a custom zip?

[apostolos]

@sriharshachilakapati We now use Maven profiles and Gradle's org.gradle.internal.os to generate a single cross-platform configuration.

We hid the Natives radio box for both Maven and Gradle since it is no longer necessary. We'll use it just for the ZIP bundle when the generator is ready.

[apostolos]

The ZIP bundle generator is now live! Initially only enabled for nightly builds but it's feature complete. It will be available for stable and release builds as soon as 3.1.0 is released.

Surprisingly, it works on all modern browsers and on IE10+

What we ended up doing is uploading all files on S3 uncompressed. We then retrieve the folder structure in JSON format which is quite fast and pick all the file paths that were selected by the user. We then download each file and generate the ZIP file, both processes are done 100% client-side.

@kappaOne There was no point in downloading ZIP files per binding since the JARs are already compressed. There is a performance penalty because we do many HTTP requests to download the files, but we've minimized it by doing up to 4 parallel downloads at a time ( actually we do a naive batch of 4 files at the time, will refactor it later to a real parallel download queue ).

@Spasi I think you can close the issue!

[httpdigest]

Does this mean that you cannot wget/curl a specific URL with just using query parameters in order to get a customized bundled zip file?

[Spasi]

Does this mean that you cannot wget/curl a specific URL with just using query parameters in order to get a customized bundled zip file?

Correct. Building a service for this would mean additional (bandwidth/computation) cost. The project has no income (donations/ads/etc) so keeping things simple is a priority. Currently the only costs are S3/CloudFront storage/bandwidth and an EC2 t2.small instance (already overwhelmed by TeamCity + build workers).

Downloads should be faster too. Instead of everything going through our puny US-based server, build files are served from the nearest CloudFront edge.

If more customization is required, it can be implemented without a dedicated service. The file structure generated by ant release is now available under https://build.lwjgl.org/nightly/bin/. A bin folder will also be created for stable and release builds after 3.0.0. A simple script could easily download the necessary files in the preferred structure.

[apostolos]

We've implemented a proper download queue. Downloads now complete at almost half the time in a good connection.

@httpdigest I'll put up a page with the file catalog so people can manually curl whatever they need.

[apostolos]

@httpdigest There is now a file browser on the download page that you can use to either download the files directly or copy the paths and put them on a script.