Oculus VR trace fails to replay with 1.3.1 #2586

rishi-rranjan · 2019-02-02T14:02:45Z

Release 1.3.1 is the first release which enabled capturing logs for Oculus Mobile VR. But replay keeps failing consistently. Where are debug logs saved for GAPID?

@AWoloszyn Please let me know if I can provide some logs to resolve this.

com.oculus.vrcubeworldsv_20190202_0550.gfxtrace.zip

AWoloszyn · 2019-02-02T14:21:16Z

They should be in your %APPDATA%\Temp. Gapis.log and gapic.log.

…

On Sat, Feb 2, 2019, 9:02 AM Rishi Ranjan ***@***.***> wrote: Release 1.3.1 is the first release which enabled capturing logs for Oculus Mobile VR. But replay keeps failing consistently. Where are debug logs saved for GAPID? com.oculus.vrcubeworldsv_20190202_0550.gfxtrace.zip <https://github.com/google/gapid/files/2824465/com.oculus.vrcubeworldsv_20190202_0550.gfxtrace.zip> [image: replay_fails] <https://user-images.githubusercontent.com/5673695/52165035-0fa3bf80-26b0-11e9-87f2-45285abf8981.PNG> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2586>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIn9DOBpvp-cqFocccDRFAsiZpQmv5fAks5vJZqGgaJpZM4afmn3> .

rishi-rranjan · 2019-02-02T14:27:59Z

Thanks! I do see there is an exception.

gapir.log
gapis.log
gapic.log

rishi-rranjan · 2019-02-06T07:00:32Z

I see following assertion failing.

W20190205-223102608[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis: Assertion failed!
W20190205-223102608[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis:
W20190205-223102608[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis: Program: C:\Users\rishi_bazel_rishi\xppgjrnc\execroot\gapid\bazel-out\x64_windows-dbg\bin\pkg\gapis.exe
W20190205-223102609[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis: File: external/spirv_tools/source/opt/def_use_manager.cpp, Line 59
W20190205-223102609[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis:
W20190205-223102609[server.ChildProcess$LoggingStringHandler][server.ChildProcess$LoggingStringHandler.lambda$new$0] gapis: Expression: def && "Definition is not registered."

rishi-rranjan · 2019-02-06T14:48:20Z

@AWoloszyn Do you have any suggestion on WA for this issue. external/spirv_tools/source/opt/def_use_manager.cpp file is not part of GAPID repo. I see that it is from https://github.com/KhronosGroup/SPIRV-Tools/tree/master/source/opt

AWoloszyn · 2019-02-06T15:29:31Z

That is correct. The first thing you can try is updating the version of SPIRV-Tools that we use.

This would be a matter of updating
https://github.com/google/gapid/blob/master/tools/build/workspace.bzl
to the latest hash in upstream.

maybe_repository(
        github_repository,
        name = "spirv_tools",
        locals = locals,
        organization = "KhronosGroup",
        project = "SPIRV-Tools",
        commit = "8d8a71278bf9e83dd0fb30d5474386d30870b74d",
        build_file = "@gapid//tools/build/third_party:spirv-tools.BUILD",
    )

You may have to also update SPIRV-Headers as well to its updated version.

rishi-rranjan · 2019-02-06T19:36:36Z

Tried updating SPIRV-Tools and SPIRV-Headers but still getting compilation error.Will update other repos and see if I can get rid of these compilation error.

rishi-rranjan · 2019-02-07T02:29:31Z

Did following changes to make some progress. Still one compilation error is there which I need to fix.
external/spirv_tools/source/opcode.cpp:38:60: error: 'kOpcodeTableEntries' was not declared in this scope
`
Above is generated by external/spirv_tools/utils/generate_grammar_tables.py but need to see why this is not called.

diff --git a/gapis/shadertools/cc/spv_manager.h b/gapis/shadertools/cc/spv_manager.h index b0268fbd..cd54b96d 100644 --- a/gapis/shadertools/cc/spv_manager.h +++ b/gapis/shadertools/cc/spv_manager.h @@ -26,7 +26,7 @@ #include "third_party/SPIRV-Tools/source/opt/build_module.h" #include "third_party/SPIRV-Tools/source/opt/def_use_manager.h" #include "third_party/SPIRV-Tools/source/opt/ir_context.h" -#include "third_party/SPIRV-Tools/source/opt/make_unique.h" +#include "third_party/SPIRV-Tools/source/util/make_unique.h" #include "third_party/SPIRV-Tools/source/opt/reflect.h" #include "third_party/SPIRV-Tools/source/opt/type_manager.h" #include "third_party/SPIRV-Tools/source/opt/types.h" diff --git a/tools/build/workspace.bzl b/tools/build/workspace.bzl index 2a3677c0..d5e13bee 100644 --- a/tools/build/workspace.bzl +++ b/tools/build/workspace.bzl @@ -147,7 +147,7 @@ def gapid_dependencies(android = True, mingw = True, locals = {}): locals = locals, organization = "KhronosGroup", project = "SPIRV-Headers", - commit = "9f6846f973a1ef53790e75b9190820ab1557434f", + commit = "8bea0a266ac9b718aa0818d9e3a47c0b77c2cb23", build_file = "@gapid//tools/build/third_party:spirv-headers.BUILD", ) @@ -167,7 +167,7 @@ def gapid_dependencies(android = True, mingw = True, locals = {}): locals = locals, organization = "KhronosGroup", project = "SPIRV-Tools", - commit = "8d8a71278bf9e83dd0fb30d5474386d30870b74d", + commit = "9b6ba4d1c57e7dac46406ef96c726348cdea49b9", build_file = "@gapid//tools/build/third_party:spirv-tools.BUILD", )

rishi-rranjan · 2019-02-09T19:22:18Z

@AWoloszyn I am able to compile gapid with latest SPIRV-Tools and generated header. But I am still getting same exception. I have attached all the files I had to change on master branch.

Let me know if you have any other suggestions. Should I change other SPIRV repos ?
files.zip

Updated tools/build/workspace.bzl with latest SPIRV-Tools and SPIRV-Headers
Regenerated SPIRV-Headers. The steps at tools/build/third_party/SPIRV-Tools-Generated/README.md are outdated. I had to checkout git clone https://github.com/google/effcee.git as per https://github.com/KhronosGroup/SPIRV-Tools
Following files had to be modified to compile with latest SPIRV-Tools and Header.
modified: gapis/shadertools/cc/common.cpp modified: gapis/shadertools/cc/common.h modified: gapis/shadertools/cc/name_manager.h modified: gapis/shadertools/cc/spv_manager.cpp modified: gapis/shadertools/cc/spv_manager.h modified: tools/build/third_party/SPIRV-Tools-Generated/DebugInfo.h modified: tools/build/third_party/SPIRV-Tools-Generated/build-version.inc modified: tools/build/third_party/SPIRV-Tools-Generated/core.insts-unified1.inc modified: tools/build/third_party/SPIRV-Tools-Generated/enum_string_mapping.inc modified: tools/build/third_party/SPIRV-Tools-Generated/extension_enum.inc modified: tools/build/third_party/SPIRV-Tools-Generated/generators.inc modified: tools/build/third_party/SPIRV-Tools-Generated/glsl.std.450.insts.inc modified: tools/build/third_party/SPIRV-Tools-Generated/operand.kinds-unified1.inc modified: tools/build/workspace.bzl
Had to change following file to correct for correct header file include.
external\spirv_tools\source\opt\function.cpp external\spirv_tools\source\opt\optimizer.cpp

rishi-rranjan · 2019-02-09T19:33:48Z

@AWoloszyn I changed all four SPIRV repos and tried but it also didn't help. Let me know if you have any other suggestions.

rishi-rranjan · 2019-02-10T14:31:13Z

KhronosGroup/SPIRV-Tools#2203 had similar issue. I did confirm that I have this fix.

I don't understand SPIRV much so will continue looking and decide if I should open a bug on SPIRV-Tools.

AWoloszyn · 2019-02-11T14:45:17Z

Sorry, I was out of the country last week.

I am trying to figure out where spirv-opt is being invoked in our chain. If you are trying to replay on the original device (it does not appear like this is the case in your screenshot) tip of tree should work, as it avoids a lot of this SPIR-V work.

#2600 Fixes one of the issues, in that now your sample replays, at least on my desktop.

I do not think it is replaying correctly, but it gets further.

rishi-rranjan · 2019-02-11T21:24:21Z

@AWoloszyn I had been trying to replay in on original device but it crashes there also.

I will try #2600 and let you know.

rishi-rranjan · 2019-02-11T21:46:11Z

@AWoloszyn I tried #2600 but it didn't help. I will do a clean bazel build and see if it helps.

rishi-rranjan · 2019-02-11T23:27:16Z

With a clean build also, I am getting same issue.

rishi-rranjan · 2019-02-22T01:11:25Z

@AWoloszyn Do you have any other suggestion?

AWoloszyn · 2019-02-25T20:50:56Z

Hey, sorry for the slow reply. We were working on getting 1.4 out the door.
Could you try 1.4 and let me know if there is still an issue?

rishi-rranjan · 2019-02-26T00:06:48Z

@AWoloszyn I tried 1.4 but it has gone worse with Oculus VR. It crashes right away. Attaching logs.
gapic.log
gapis.log

rishi-rranjan · 2019-02-26T14:15:22Z

@AWoloszyn I can help with any logs to fix the issue. I need this working to deliver on a critical project so let me know if I can facilitate anything.

AWoloszyn · 2019-02-26T14:23:03Z

Do you happen to have the adb logcat output from the time of the crash?

rishi-rranjan · 2019-02-26T14:30:51Z

Attaching full adb log and filtered one with GAPID.
log_gapid.txt

AWoloszyn · 2019-02-26T14:35:54Z

I think the link to your full adb log may be incorrect.

rishi-rranjan · 2019-02-26T14:38:38Z

log.zip

Attaching zipped version of full adb log.

AWoloszyn · 2019-02-26T14:50:07Z

Thanks, that was somewhat helpful.

It looks like there is no crash, but the application in question is just shutting itself down (relatively gracefully)

02-26 21:27:40.243 12573 12709 V VrCubeWorld:         eglGetCurrentSurface( EGL_DRAW ) = 0x6fb3e29860
...
...

GAPID functioning normally here

02-26 21:27:40.247 12573 12895 I GAPID   : [gapii/cc/spy.cpp:404] Framebuffer config: R8 G8 B8 A8 D24 S8

02-26 21:27:40.247  4197  4197 D BrightnessMirror: hideMirror cancelled since it's not visible
02-26 21:27:40.250 12573 12573 D SurfaceView: BG show() Surface(name=Background for - SurfaceView - com.oculus.vrcubeworldsv/com.oculus.gles3jni.GLES3JNIActivity@cf13a41@0) android.view.SurfaceView{cf13a41 V.E...... ........ 0,0-2960,1440}

Surface View destroyed for some reason

02-26 21:27:40.251 12573 12573 D SurfaceView: surfaceDestroyed 1 android.view.SurfaceView{cf13a41 V.E...... ........ 0,0-2960,1440}
02-26 21:27:40.251 12573 12573 V VrCubeWorld: GLES3JNIActivity::surfaceDestroyed()
02-26 21:27:40.252 12573 12573 V VrCubeWorld:     GLES3JNILib::onSurfaceDestroyed()
02-26 21:27:40.252 12573 12573 V VrCubeWorld:         ANativeWindow_release( NativeWindow )
02-26 21:27:40.253 12573 12573 V VrCubeWorld: GLES3JNIActivity::onStop()
02-26 21:27:40.253 12573 12573 V VrCubeWorld:     GLES3JNILib::onStop()
02-26 21:27:40.261 12573 12573 D SurfaceView: BG show() Surface(name=Background for - SurfaceView - com.oculus.vrcubeworldsv/com.oculus.gles3jni.GLES3JNIActivity@cf13a41@0) android.view.SurfaceView{cf13a41 V.E...... ........ 0,0-2960,1440}

rishi-rranjan · 2019-02-26T14:52:30Z

Before dumping the adb logs, I had closed both gapid on PC and application on Android so I don't hit the logcat buffer wrap around. That's why you might be seeing the application closing gracefully.

AWoloszyn · 2019-02-26T15:08:08Z

The time difference here was only 6ms though, which seems odd to me.

rishi-rranjan · 2019-02-26T18:01:44Z

With 1.3.1 I was able to capture the logs but was not able to open it. I tried opening logs that I captured with 1.3.1 in 1.4 but it doesn't work.

pmuetschard · 2019-02-26T20:20:41Z

Since you were able to trace on 1.3.1, can you try replaying on device rather than on desktop for 1.3.1? When replaying on device, SPIRV-Cross should not even be in the code path. There was a bug, which was fixed in b81b362, so if you could apply that patch (it's a one line change) on top of 1.3.1 and see if that works for you, it might get you going in the mean time.

rishi-rranjan · 2019-02-27T02:27:56Z

I see this issue with 1.3.1 with this fix also. Will try by going back to 1.3.1. if I see issue there also, I have moved to newer OVR SDK. Will go back to older one and try to isolate the issue.

rishi-rranjan · 2019-02-27T14:29:43Z

Tried all the older SDK also which had worked with 1.3.1 but now it's not working. Not sure if any Android upgrade since then is breaking the capture.

pau-baiget · 2019-02-27T14:46:16Z

Hi! I'm trying to reproduce the issue here. Can you confirm which device are you using? Also, which CubeWorld demo are you tracing? Is it VrCubeWorld_Vulkan?

rishi-rranjan · 2019-02-27T14:53:21Z

@pau-baiget I am using Galaxy S8.
Android Version: 8.0.0
Samsung Experience Version: 9.0

I am using VrCubeWorld_SurfaceView.

We are building our own Cloud XR solution on top of this sample but I can make progress there once I am able to trace basic demo.

pau-baiget · 2019-02-27T15:11:17Z

Great, thanks! Can you please attach here the output of the following command in the root GAPID folder?

$ ./gapit devices

(I want to check whether I have the same Galaxy S8 here with same driver version).

rishi-rranjan · 2019-02-27T15:30:16Z

gapit_log.txt

Are you looking for this? I am on Windows 10 so ran 'gapit devices' from comamnd line and attached the log.

rishi-rranjan · 2019-02-28T15:00:39Z

Hi @pau-baiget , were you able to reproduce this?

pau-baiget · 2019-03-01T13:15:06Z

I'm waiting to get a Gear VR device so I can enable the Gear VR Service on my Galaxy S8, necessary to run the VrCubeWorld_SurfaceView app.

rishi-rranjan · 2019-03-01T16:00:45Z

FYI.. You can launch the application without HMD if you enable Dev mode: https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-device-setup/

Don't remember if it let's you enter this dev mode without setting it up with HMD first time.

rishi-rranjan · 2019-03-01T18:05:27Z

I tried this after resettign my S8. You can start debugging VrCubeWorld_SurfaceView without Gear VR HMD also.

You will need to add OSIG file to your project before the dev mode settings from
https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-device-setup will be visible.

Note: If you do not see the Developer Mode toggle switch after tapping VR Service Version several times, close the Gear VR Service and relaunch and you should see it.

pau-baiget · 2019-03-05T12:35:35Z

Sorry for late response. I got the Gear VR headset and was able to run the app using GAPID. I could obtain a trace and open it with GAPID 1.4 (I can also open the trace you attached in the ticket description, they look similar). The trace has 4 contexts:

Context 0: Lots of 32-command groups ending with a glDrawElementsInstanced, but not inside any specific frame (They're using frontbuffer rendering so they're not using eglSwapBuffers) . Looking at the glDrawElementsInstanced, this is drawing 1500 cubes (which is the view for each eye as it's using GL_OVR_multiview2).
Context 1: 1 frame and 1 incomplete frame. The one frame contains a few glDrawArrays calls with index_count:4 (render a quad somewhere). Looking at the attached shaders looks like the chromatic aberration pass. (again, front buffer rendering misses the frame info)
Context 2: The selected context by default. Same as Context 1: 1 frame and 1 incomplete frame (again, front buffer rendering misses the frame info). The incomplete frame has lots of drawcalls (this is also part of the TimeWarp process
Context 3: No drawcalls there, only shader management (glCreateShader, glAttachShader, glLinkProgram, glGetUniformLocation, etc..)

I did find the reported error while trying to replay on desktop, but replaying on the device didn't cause any error or crash. I'm using OVR_sdk_mobile v. 1.21. and a Samsung Galaxy S9. I will try this afternoon with the 1.22 and my personal Galaxy S8 to check for any different behavior.

rishi-rranjan · 2019-03-05T14:11:32Z

@pau-baiget Thanks for the response. I am also able to replay the trace on phone now with 1.4. Not sure if resetting the phone helped.

I tried replaying the trace on two different PCs but both of them failed. One has Nvidia 1060M and other has GTX 970 desktop GPU. I can start debugging my issue with phone so will I am unblocked now for my tracing and debugging.

rishi-rranjan · 2019-03-05T22:19:54Z

I am able to replay the log that I attached to this issue but I am not able to capture new trace using 1.4.
I tried capturing on two different PCs. I see the same issue where GAPID crashes.

AWoloszyn · 2019-03-13T14:31:22Z

Can you try with 1.5.0?

rishi-rranjan · 2019-03-13T17:27:43Z

It worked once for me with 1.5.0 but once I rebooted my Laptop it is stuck at same issue. I will upload the logs from 1.5.0 in sometime.

rishi-rranjan · 2019-03-13T17:39:53Z

Attaching logs:

When I start gapid and start capturing the logs I see following:

I press Start and then it gets stuck at following:

When I press stop:

gapic.log
gapis.log

AWoloszyn · 2019-03-14T14:19:14Z

Do you have a screenshot of your trace options? I am wondering if something in there that can help.

rishi-rranjan · 2019-03-14T15:53:33Z

Are you looking for this?

rishi-rranjan · 2019-03-18T18:43:03Z

Tried 1.5.0 on a freshly installed PC also and I see exactly the same issue.

rishi-rranjan · 2019-03-20T01:31:31Z

It works if I select all the option. Will close the issue.

rishi-rranjan closed this as completed Mar 20, 2019

Oculus VR trace fails to replay with 1.3.1 #2586

Oculus VR trace fails to replay with 1.3.1 #2586

Comments

rishi-rranjan commented Feb 2, 2019 • edited Loading

AWoloszyn commented Feb 2, 2019 via email

rishi-rranjan commented Feb 2, 2019 • edited Loading

rishi-rranjan commented Feb 6, 2019

rishi-rranjan commented Feb 6, 2019

AWoloszyn commented Feb 6, 2019

rishi-rranjan commented Feb 6, 2019

rishi-rranjan commented Feb 7, 2019 • edited Loading

rishi-rranjan commented Feb 9, 2019

rishi-rranjan commented Feb 9, 2019

rishi-rranjan commented Feb 10, 2019

AWoloszyn commented Feb 11, 2019

rishi-rranjan commented Feb 11, 2019

rishi-rranjan commented Feb 11, 2019

rishi-rranjan commented Feb 11, 2019

rishi-rranjan commented Feb 22, 2019

AWoloszyn commented Feb 25, 2019

rishi-rranjan commented Feb 26, 2019

rishi-rranjan commented Feb 26, 2019

AWoloszyn commented Feb 26, 2019

rishi-rranjan commented Feb 26, 2019 • edited Loading

AWoloszyn commented Feb 26, 2019

rishi-rranjan commented Feb 26, 2019

AWoloszyn commented Feb 26, 2019

rishi-rranjan commented Feb 26, 2019

AWoloszyn commented Feb 26, 2019 • edited Loading

rishi-rranjan commented Feb 26, 2019

pmuetschard commented Feb 26, 2019

rishi-rranjan commented Feb 27, 2019

rishi-rranjan commented Feb 27, 2019

pau-baiget commented Feb 27, 2019

rishi-rranjan commented Feb 27, 2019

pau-baiget commented Feb 27, 2019

rishi-rranjan commented Feb 27, 2019 • edited Loading

rishi-rranjan commented Feb 28, 2019

pau-baiget commented Mar 1, 2019

rishi-rranjan commented Mar 1, 2019

rishi-rranjan commented Mar 1, 2019

pau-baiget commented Mar 5, 2019 • edited Loading

rishi-rranjan commented Mar 5, 2019

rishi-rranjan commented Mar 5, 2019

AWoloszyn commented Mar 13, 2019

rishi-rranjan commented Mar 13, 2019

rishi-rranjan commented Mar 13, 2019

AWoloszyn commented Mar 14, 2019

rishi-rranjan commented Mar 14, 2019

rishi-rranjan commented Mar 18, 2019

rishi-rranjan commented Mar 20, 2019

rishi-rranjan commented Feb 2, 2019 •

edited

Loading

rishi-rranjan commented Feb 2, 2019 •

edited

Loading

rishi-rranjan commented Feb 7, 2019 •

edited

Loading

rishi-rranjan commented Feb 26, 2019 •

edited

Loading

AWoloszyn commented Feb 26, 2019 •

edited

Loading

rishi-rranjan commented Feb 27, 2019 •

edited

Loading

pau-baiget commented Mar 5, 2019 •

edited

Loading