-
Notifications
You must be signed in to change notification settings - Fork 729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run testing at OpenJ9 with debug symbols #8874
Comments
It would also be useful if the testing at AdoptOpenJDK had the debug symbols included by default. |
@smlambert @llxia Is one of these options easier to do in the context of Test playlists? Some thoughts: To have the tests include them, the workflow would probably be something like:
Would this require passing in additional options to TKG to indicate where the to get the debug image? Including the debug symbols in the OpenJ9 builds by default seems simpler but means we'd need a different approach between OpenJ9 and Adopt. If we want to allow developers to specify the debug image to apply in ie: grinders, then we may need all the work for separate package option anyways. |
We already often have to go pick the native test images that are built, can the debug image be included in that gzip file? In that way, there are less additional steps required to the workflow. Mind you, currently, I do not believe the functional native test images are being built at Adopt, only openjdk native test images (so for 'nice to have' at Adopt, there would be more to changes required. |
Certainly the debug image can be combined with the test image, although if it's specific to OpenJ9 I don't think having it separate is much different. |
Having it separate is not much different, it means an additional URL for the gzip file being passed in the CUSTOMIZED_URL_SDK parameter to the test pipeline. Not a big deal to automation, but ever try telling a developer that something new is expected in a param field. I suspect it would throw some people off, suspect it would further confuse people trying to run Grinders, etc. Would like not to introduce more ways to discourage dev team from running tests and more reasons to require extra documentation that may never get referenced. |
Good points, I'm not against combining the debug image with the test image. Personally I usually forget to include the test image when running a grinder. Putting everything (test and debug) in the JVM package at OpenJ9 would be easiest. |
Discussing with Dan and Lan, we're leaning towards combining the debug image into the test image rather than modifying the JVM package. This keeps the separation for OpenJ9 builds the same as at the Adopt builds, since the debug image won't be included with the JVM at Adopt. Aligning with Adopt reduces the number of special cases and things that people need to remember. @keithc-ca can you please make this change to put the debug-image folder into the test image archive. Unfortunately this means the debug image can't be directly overlayed on top of the JVM, and will need to be copied by the user. We could keep the existing debug image archive as well, as long as we don't run out of artifactory space. After the test image archive is updated, Lan will modify the code to copy or move the debug libraries into the right place. |
If the content currently included in |
I feel I need to point out that using such a unified |
I assumed the approach would be same as suggested in #8874 (comment) |
Except on AIX, I don't expect the debug symbol files are used at runtime unless there is a crash. Even on AIX the risk of a problem resulting from removing debug symbols seem low. |
We can try that, it seems a bit trickier to manage. https://github.com/AdoptOpenJDK/openjdk-tests/blob/6d4670a158607cb0e0fd30e76b5c6a0f03a7a74d/get.sh#L273 is not going to put the files in the right place but will have to copy/move them. |
To make #8874 (comment) more concrete, the resulting test-image archive would contain paths such as these:
A consequence of this is that there will then be two or three top-level directories ( @smlambert Can you explain the intent of that part of |
@llxia fyi the previous comment as well. |
|
It is possible, but while this makes it easier for the code, it makes it harder for development or service to apply the debug image to a JVM. It shouldn't be too arduous to handle two directories in the code.
Looking at the get.sh logic, it seems if we add a second directory into the test image, then get.sh will no longer work as it looks for a dir array of length 1. It may need to be modified first. |
This was in response to #8874 (comment) (before I saw Lan and Pete's comments) Once the space separated param list passed to CUSTOMIZED_SDK_URL are unzipped, https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/get.sh#L272-L289 moves the unzipped contents into various top-level directories. CUSTOMIZED_SDK_URL can be one (or two) of many things depending on the name of the file:
But as @llxia mentioned, the plan was that we would handle updating get.sh |
oh, misread the message, as above mentioned that
Anyway, if we want to change
to multiple top-level dirs (after unzipping)
, then the check becomes a little complicated in
We need to handle |
The current test-image does have a top-level directory called
I can put the debug-image contribution wherever we decide; my suggestion of having top-level We agree, the So what pattern do we want? [1] https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_linux_OMR/517/ |
@keithc-ca - are you referencing the OMR build link because there is a separate debug-image.tar.gz file? If you want to directly pass that, then by all means do (I only suggested merging it with test images for convenience of developers using Grinders, but I am sure developers can adapt). How that would look (from an OpenJ9 build perspective... we of course have to support other build systems too)? at the moment in get.sh, all get unzipped and moved based on checking what the name says, we can change that logic however we want. What other consumers of debug-image.tar.gz are there? Please describe some common use cases / scenarios. |
Describing it as an 'OMR' link is misleading - the link is to an 'OMR Acceptance' build of jdk11. Most (all?) jdk builds at ci.eclipse.org/openj9 have a link to a test-image archive; I only offered that as a representative sample illustrating the path names in such an archive (they should all be the same). The primary use-cases for a debug-image are up in the description of this issue. BTW, those builds should also all have links to a debug-image archive. For a brief time, the debug-image URL was included in |
To me, there are two problems: Regardless of what is the solution for problem 1, problem 2 cannot be avoided. On top of my head, I think the best way to solve Problem 1 is to be able to download all jar files at the folder level. Instead of asking the link for each jar, users provide link at the folder level. In this way, users do not need to provide test-images, debug image or anything else in the future. I am able to do it using
However, I am not able to do it using curl options only. We can do it using script, but it gets ugly. Since only SDK filename changes (test-images.tar.gz and debug-image.tar.gz name does not change), can we ask users to provide one link to SDK? Then we try to download test-images.tar.gz and debug-image.tar.gz in In summary,
In this way, we do not need to put the debug-image folder into the test image, as all three jars are available atm. Only one link is needed from users. What do you think about this solution? Please let me know if I missed any scenarios. |
I like this idea, it fixes the problem that occurs for me, as I usually forget to specify the URL to the test image. Finding all the artifacts automatically and putting them in the correct places seems ideal. It keeps the debug and test images separate, which seems clearer for users. The user still has to specify the files manually if they created a build manually and uploaded it, but I expect this is rarely done and would always require manually specifying a test image. Note the test image at Adopt is named differently, and the name varies per build. |
And, following that pattern, the debug image from adopt will have |
Yes, the fact that the builds/naming across the projects is not the same (is unfortunate and unnecessary since many of the people working at both projects could just talk/align, super annoying, but of course we will accommodate as per usual). Agree, I think this is a 'good plan', will have to ensure that we continue to support pick up of openjdk test image artifacts and jre artifacts. If there are cases where someone wanted to pass two separate URLs from separate servers to CUSTOMIZED_SDK_URL, I am not aware of them. |
I too like that idea, but it assumes the server is willing to list the files in a directory; not all are configured that way, but as long as the ones we care about cooperate, we'll be good. |
The jdk, jre, test image and debug image at Adopt are archived in Jenkins and we use Jenkins API to get them (i.e., we use UPSTREAM_JOB_NAME and UPSTREAM_JOB_NUMBER). We do not use url to download them in nightly builds. IMO, Adopt does not need the ability to auto-download test image and debug image. If Adopt uses Artifactory(or other server) to store artifacts, then auto-download test image and debug image is useful. My plan is to make sure openj9 works first. If there is a need to have auto-download in Adopt, we can update the script to support it. |
Yep, sounds good. Adopt will shortly use Artifactory (or some other off-master storage with plan to move to it after this release is finished, related: adoptium/aqa-tests#1659), but we can strongly influence how that looks so its not onerous from a QA perspective. |
While working on this PR, I noticed that Download mac SDK:
Download mac debug image:
The same thing in xlinux SDK from OpenJ9.
Also checked Mac sdk from AdoptOpenJDK. It does not have any |
I expect the absence of debug symbols in builds from adopt is because they use configure option [1] https://github.com/AdoptOpenJDK/openjdk-build/blob/master/sbin/build.sh#L275 |
Do we expect to see debug symbols to be in OpenJ9 builds? The current behavior is that the debugs symbols are partially in OpenJ9 builds (in lib, but not in bin) |
There should be debug symbols for all openj9 contributions (e.g. if jitserver is enabled, you'll see both jitserver and jitserver.debuginfo in the bin directory). On Windows, everything is in the bin directory (there are no .pdb files in the lib directory). If adopt didn't use the configure option |
Seems that debug in OpenJ9 builds is expected, but it varies between jdk8 and jdk11+ atm. Also we may decide to change the OpenJ9 build defaults in the future. |
ok, I will overlay the debug images on top of sdk regardless if the sdk has them or not. |
Running into this problem now over in #8992. It outputs a backtrace but it is not useful:
|
Ya, I think that's a whole different problem with the RAS code not being able to get the symbols. gdb should work with the debug symbols, but you may need the system libraries as well. This is what jextract collects. See #8772 |
- add auto download test-images and debug-images - to simplify the story, auto download will only be triggered if users provide only one CUSTOMIZED_SDK_URL and URL filename contains OpenJ9-JDK. In this way, if users want to provide multiple separate URLs or multiple separate URLs from separate servers, they can still do so and auto-download will not be triggered. - Once auto-download is triggered, by default, the flag test_images_required and debug_images_required are set to true in get.sh. If the flag is set to true, it means the script will retry if the download fails and the program will fail if all retry attempts fail (the same behavior as downloading SDK). If the flag is set to false, it means the script will not try to auto download. - In JenkinsfileBase, pass in test_images_required and debug_images_required to get.sh if users want to overwrite the flag(s) - During unzipping process, if debug-images jar is found, overlay debug-images on top of sdk j2sdk-image/jre dir if it exists. Otherwise, extracts into j2sdk-image dir Issue: eclipse-openj9/openj9#8874 Signed-off-by: lanxia <[email protected]>
- add auto download test-images and debug-images - to simplify the story, auto download will only be triggered if users provide only one CUSTOMIZED_SDK_URL and URL filename contains OpenJ9-JDK. In this way, if users want to provide multiple separate URLs or multiple separate URLs from separate servers, they can still do so and auto-download will not be triggered. - Once auto-download is triggered, by default, the flag test_images_required and debug_images_required are set to true in get.sh. If the flag is set to true, it means the script will retry if the download fails and the program will fail if all retry attempts fail (the same behavior as downloading SDK). If the flag is set to false, it means the script will not try to auto download. - In JenkinsfileBase, pass in test_images_required and debug_images_required to get.sh if users want to overwrite the flag(s) - During unzipping process, if debug-images jar is found, overlay debug-images on top of sdk j2sdk-image/jre dir if it exists. Otherwise, extracts into j2sdk-image dir Issue: eclipse-openj9/openj9#8874 Signed-off-by: lanxia <[email protected]>
- add auto download test-images and debug-images - to simplify the story, auto download will only be triggered if users provide only one CUSTOMIZED_SDK_URL and URL filename contains OpenJ9-JDK. In this way, if users want to provide multiple separate URLs or multiple separate URLs from separate servers, they can still do so and auto-download will not be triggered. - Once auto-download is triggered, by default, the flag test_images_required and debug_images_required are set to true in get.sh. If the flag is set to true, it means the script will retry if the download fails and the program will fail if all retry attempts fail (the same behavior as downloading SDK). If the flag is set to false, it means the script will not try to auto download. - In JenkinsfileBase, pass in test_images_required and debug_images_required to get.sh if users want to overwrite the flag(s) - During unzipping process, if debug-images jar is found, overlay debug-images on top of sdk j2sdk-image/jre dir if it exists. Otherwise, extracts into j2sdk-image dir Issue: eclipse-openj9/openj9#8874 Signed-off-by: lanxia <[email protected]>
- add auto download test-images and debug-images - to simplify the story, auto download will only be triggered if users provide only one CUSTOMIZED_SDK_URL and URL filename contains OpenJ9-JDK. In this way, if users want to provide multiple separate URLs or multiple separate URLs from separate servers, they can still do so and auto-download will not be triggered. - Once auto-download is triggered, by default, the flag test_images_required and debug_images_required are set to true in get.sh. If the flag is set to true, it means the script will retry if the download fails and the program will fail if all retry attempts fail (the same behavior as downloading SDK). If the flag is set to false, it means the script will not try to auto download. - In JenkinsfileBase, pass in test_images_required and debug_images_required to get.sh if users want to overwrite the flag(s) - During unzipping process, if debug-images jar is found, overlay debug-images on top of sdk j2sdk-image/jre dir if it exists. Otherwise, extracts into j2sdk-image dir Issue: eclipse-openj9/openj9#8874 Signed-off-by: lanxia <[email protected]>
PR adoptium/aqa-tests#1692 is merged. To trigger the auto download, upstream builds (openj9) should be updated to pass in only the CUSTOMIZED_SDK_URL=url_to_sdk. The script will then auto-download test-images and debug-images for OpenJ9 SDK. For sdk that does not have debug-images (i.e., zos), DEBUG_IMAGES_REQUIRED should set to false when trigger the test builds. |
Is there anything left to be done for this? |
The change in the test pipeline is completed. The upstream needs to updated to pass in only the CUSTOMIZED_SDK_URL=url_to_sdk. See #8874 (comment) |
I believe everything is completed. |
When crashes occur running tests at the OpenJ9 project, some platforms include native stack traces in the crash output. However since there are no debug symbols included with the JVM, the native stack traces usually aren't helpful. We should have the debug symbols included by default for tests running at the OpenJ9 project in order to make the crash output more useful, help triage go faster.
Most of the work has been completed to create a debug image containing the debug symbols. This is helpful for AdoptOpenJDK builds where the debug symbols are undesirable in the JVM builds in order to keep the size of the JVMs smaller. Developers and service can use the debug image to aid in diagnosing problems.
It's TBD what the best approach should be for OpenJ9 builds/tests. We can either include the debug symbols into the JVM builds at OpenJ9 by default, or download and expand the debug image before running tests at the project. I've created this Issue to discuss it.
The text was updated successfully, but these errors were encountered: