Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[meta] Dota performance issues on Metal #2161

Closed
19 of 22 tasks
kvark opened this issue Jun 20, 2018 · 17 comments
Closed
19 of 22 tasks

[meta] Dota performance issues on Metal #2161

kvark opened this issue Jun 20, 2018 · 17 comments

Comments

@kvark
Copy link
Member

kvark commented Jun 20, 2018

Analyzing the Instruments profiles reveals quite a few things we could do better.

Native API calls

Metal backend issues

  • Too much heap (re)allocation - Various Metal performance optimizations #2185. We should avoid any allocations at run-time by re-using the storage and/or using the iterators more aggressively.
  • Too many command buffers - Command buffer allocation pool in Metal #2180. The instrumented memory profile doesn't show command buffers being re-used. I suppose the queue just creates a new one every time until it reaches the capacity, and then starts trying to re-use the completed ones. This is undesired, and we may attempt to address it (temporarily) by playing with the capacity limit.
  • Dynamic creation of a render pass - [mtl] minimize creation of render passes #2178. We create one every time a render pass is started by copying one from the framebuffer and filling out all the clear values/operations. Apparently, this is super heavy, and we should probably avoid creating any Metal objects at normal run time.
  • Dynamic creation of depth-stencil states - [mtl] cache depth/stencil states #2195.
  • Binding descriptor sets is visibly slow - Heap-less descriptor sets in Metal #2183. We should avoid doing work for repeated bindings, and need to explore ways to share heap-allocated data between sets of the same group.
  • Non-existent semaphore frame synchronization - Frame synchronization in Metal #2143
  • Too much locking going on - [meta] Locking on Metal #2229
  • Too many command buffer callbacks - [mtl] dynamic depth bias, callback coalescence, and more stats #2224
  • Copying the render pass descriptor (which we do at each start of an RP) takes 7.4% of the time in our library
  • Binding descriptor sets still involves a few hot loops that could be faster. In particular, we set resources in batches, and we provide a closure, which checks for the current pre-render status. All of that can be simplified if we are outside of pass.

Portability issues

Application concerns

  • The engine operates within an assumption that command buffer recording is cheap, while submission is expensive. They move the submission onto a separate thread, which doesn't appear to be saturated enough.
    • the chosen threading model (of having 2 threads for job execution and a dedicated submission thread) allows MoltenVK to effectively run on 3 threads, while we are limited to 2. This is something Dota should fix or expose for us to test.
    • passing "-threads 3" technically solves this
  • Submissions are too late - Metal enqueue() advantage #2232, Remote command sink in Metal #2260 (more work is possible)

Things to investigate

  • Actual memory types/heaps used by an application. Maybe we could tweak the queries and/or ask Valve to fix those in order to use our exposed memory more efficiently. This is certainly a difference with Molten.
@grovesNL
Copy link
Contributor

Other ideas:

@kvark kvark mentioned this issue Jun 21, 2018
3 tasks
bors bot added a commit that referenced this issue Jun 22, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

Co-authored-by: Dzmitry Malyshau <[email protected]>
bors bot added a commit that referenced this issue Jun 22, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

Co-authored-by: Dzmitry Malyshau <[email protected]>
bors bot added a commit that referenced this issue Jun 22, 2018
2071: Remaping descriptor sets in the gl backend r=kvark a=ZeGentzy

I'll rebase off master when I'm done.

Uniforms in gl only have a bindings field, not a set one. This means that for shaders that use multiple sets to work, we must change where we are binding them.

See page 14 for what I mean: https://www.khronos.org/assets/uploads/developers/library/2016-vulkan-devday-uk/4-Using-spir-v-with-spirv-cross.pdf

PR checklist:
- [ ] `make` succeeds (on *nix)
- [ ] `make reftests` succeeds
- [ ] tested examples with the following backends:


2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

2166: Update example instructions in README.md r=kvark a=king6cong



Co-authored-by: Hal Gentz <[email protected]>
Co-authored-by: Dzmitry Malyshau <[email protected]>
Co-authored-by: king6cong <[email protected]>
bors bot added a commit that referenced this issue Jun 22, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

2166: Update example instructions in README.md r=kvark a=king6cong



Co-authored-by: Dzmitry Malyshau <[email protected]>
Co-authored-by: king6cong <[email protected]>
bors bot added a commit that referenced this issue Jun 22, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

Co-authored-by: Dzmitry Malyshau <[email protected]>
@jrmuizel
Copy link
Contributor

So getting counts of calls to objc_msgSend turned out to be more difficult than I thought. I started with lldb which was very, very slow. I switched to gdb, had to fix gdb and then it was still too slow. Finally I settled on using dtrace, which ran at a reasonable speed.

My results are:
MoltenVK: 47 thousand objc_msgSend/frame
gfx: 131 thousand objc_msgSend/frame

@grovesNL
Copy link
Contributor

grovesNL commented Jun 22, 2018

@jrmuizel wow, quite a big difference! How did you create this count with dtrace? Is there a way we could break it down by object name/selector/call stack to determine the difference?

@jrmuizel
Copy link
Contributor

Here's a per selector breakdown.

MoltenVK

  _addResource:                                                     0
  addResourceToList:                                                0
  allocatedSize                                                     0
  getCopyKernels                                                    0
  getUniqueID:clampMax:                                             0
  initWithDevice:options:args:argsSize:                             0
  initWithDevice:pointer:length:options:sysMemSize:vidMemSize:args:argsSize:deallocator:                0
  newBufferWithLength:options:                                      0
  sharedRef                                                         0
  copyFromBuffer:sourceOffset:toBuffer:destinationOffset:size:                0
  __new::                                                           0
  addPresentScheduledHandler:                                       0
  addScheduledHandler:                                              0
  containsObject:                                                   0
  didScheduledPresent                                               0
  error                                                             0
  immutablePlaceholder                                              0
  initWithArray:range:copyItems:                                    0
  initWithDrawablePrivate:layer:                                    0
  initWithObjects:count:                                            0
  member:                                                           0
  nextDrawable                                                      0
  present                                                           0
  presentDrawable:                                                  0
  presentScheduledInsertSeed                                        0
  priv                                                              0
  releasePrivateReferences:                                         0
  removeAllObjects                                                  0
  removeObjectsInArray:                                             0
  removeObjectsInRange:inArray:range:                               0
  setPresentScheduledInsertSeed:                                    0
  setWithArray:range:copyItems:                                     0
  vkpresent                                                         1
  blitCommandEncoder                                                1
  __new:::                                                          1
  initWithCapacity:                                                 1
  mutablePlaceholder                                                1
  setVisibilityResultMode:offset:                                   1
  HasDCL3                                                           1
  renderEncoderInitHWState                                          1
  contents                                                          2
  drawableSize                                                      2
  virtualAddress                                                    2
  copyFromBuffer:sourceOffset:sourceBytesPerRow:sourceBytesPerImage:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:                2
  pixelFormat                                                       2
  IABEmulationData                                                  2
  computeCommandEncoder                                             2
  dispatchThreadgroups:threadsPerThreadgroup:                       2
  getComputePipelineState                                           2
  setComputePipelineState:                                          2
  description                                                       2
  initWithFormat:locale:arguments:                                  2
  stringWithFormat:                                                 2
  _dispose                                                          3
  _xref_dispose                                                     3
  _submitAvailableCommandBuffers                                    3
  commandBufferDidComplete:startTime:completionTime:error:                3
  commit                                                            3
  commitCommandBuffer:wake:                                         3
  didCompleteWithStartTime:endTime:error:                           3
  didSchedule:error:                                                3
  didScheduleWithStartTime:endTime:error:                           3
  insertObjects:count:atIndex:                                      3
  releaseAllObjectsAndReset                                         3
  removeObject:                                                     3
  replaceObjectsInRange:withObjects:count:                          3
  replaceObjectsInRange:withObjectsFromArray:range:                 3
  setCommitted:                                                     3
  submitCommandBuffers:count:                                       3
  commandBufferWithUnretainedReferences                             3
  enqueue                                                           3
  enqueueCommandBuffer:                                             3
  getAndIncrementNumCommandBuffers                                  3
  initWithQueue:retainedReferences:                                 3
  initWithQueue:retainedReferences:synchronousDebugMode:                3
  initWithCommandBuffer:                                            4
  isCommitted                                                       4
  self                                                              4
  addCompletedHandler:                                              4
  countByEnumeratingWithState:objects:count:                        5
  setSamplerState:atIndex:                                          5
  _delayedEventMatchingMask:pull:                                   5
  _nextEventMatchingEventMask:untilDate:inMode:dequeue:                5
  distantFuture                                                     5
  drain                                                             5
  isEqualToDate:                                                    5
  isMainThread                                                      5
  nextEventMatchingMask:untilDate:inMode:dequeue:                   5
  timeIntervalSinceNow                                              5
  isNSArray__                                                       6
  setBuffer:offset:atIndex:                                         6
  addObject:                                                        6
  respondsToSelector:                                               6
  drawPrimitives:vertexStart:vertexCount:                           7
  isProfilingEnabled                                                7
  popDebugGroup                                                     7
  pushDebugGroup:                                                   7
  setFragmentBytes:length:atIndex:                                  7
  setStencilReferenceValue:                                         7
  getObjects:range:                                                 8
  _mutate                                                           9
  getUniqueID:                                                      9
  removeObjectAtIndex:                                             10
  distantPast                                                      11
  insertObject:atIndex:                                            12
  setTexture:atIndex:                                              14
  removeObjectsInRange:                                            14
  setVertexBytes:length:atIndex:                                   15
  setClearDepth:                                                   15
  setClearStencil:                                                 15
  timeIntervalSinceReferenceDate                                   17
  setClearColor:                                                   42
  depth                                                            70
  depthAttachment                                                  70
  height                                                           70
  stencilAttachment                                                70
  width                                                            70
  textureBarrier                                                   77
  setViewport:                                                     97
  count                                                           103
  setDepthBias:slopeScale:clamp:                                  107
  setHasDCL3:                                                     108
  initWithCommandBuffer:descriptor:stateDescriptor:               108
  renderCommandEncoderWithDescriptor:                             108
  renderPassDescriptor                                            108
  setVisibilityResultBuffer:                                      108
  commandBufferStorage                                            112
  fenceGroupPool                                                  112
  getScratchPageGPUAddress                                        112
  getStatLocations                                                112
  getStatOptions                                                  112
  globalTraceObjectID                                             112
  setStencilFrontReferenceValue:backReferenceValue:               114
  copyWithZone:                                                   116
  deviceRef                                                       116
  setLabel:                                                       116
  setBlendColorRed:green:blue:alpha:                              117
  colorAttachments                                                119
  objectAtIndexedSubscript:                                       119
  pTelemetry                                                      120
  resourceSize                                                    147
  sampleCount                                                     203
  objectAtIndex:                                                  209
  drawPrimitives:vertexStart:vertexCount:instanceCount:baseInstance:              217
  commandQueue                                                    221
  setCurrentCommandEncoder:                                       225
  device                                                          235
  autorelease                                                     236
  setScissorRect:                                                 238
  setDepthPlane:                                                  261
  setLevel:                                                       261
  setLoadAction:                                                  261
  setSlice:                                                       261
  setStoreAction:                                                 261
  setTexture:                                                     261
  texture                                                         262
  setDepthClipMode:                                               306
  setCullMode:                                                    306
  setFrontFacingWinding:                                          306
  setTriangleFillMode:                                            306
  setDepthStencilState:                                           314
  setRenderPipelineState:                                         314
  isStatEnabled                                                   346
  _descriptorPrivate                                              370
  class                                                           485
  init                                                            486
  allocWithZone:                                                  490
  akPrivateResourceList                                           543
  ioAccelResourceList                                             543
  drawIndexedPrimitives:indexCount:indexType:indexBuffer:indexBufferOffset:instanceCount:baseVertex:baseInstance:              594
  dealloc                                                         600
  alloc                                                           605
  endEncoding                                                     694
  setVertexSamplerState:atIndex:                                  763
  setVertexTexture:atIndex:                                       812
  fragmentIABEmulationData                                        820
  getRenderPipelineState                                          820
  vertexIABEmulationData                                          820
  setFragmentBuffer:offset:atIndex:                               971
  textureType                                                     990
  retain                                                         1059
  _descriptorAtIndex:                                            1086
  bindIndirectArgumentsWithFunctions:                            1419
  commandBufferResourceInfo                                      1512
  setFragmentSamplerState:atIndex:                               1722
  setVertexBuffer:offset:atIndex:                                2470
  release                                                        3026
  setFragmentTexture:atIndex:                                    3772
  objc_msgSend                                                  36863

Gfx:

  allocatePerUnitThreadCount                                        0
  compiler                                                          0
  extractPSCompilerOutput:                                          0
  extractVSCompilerOutput:                                          0
  fragmentFunction                                                  0
  fragmentFunctionKeyWithRenderPipelineDescriptor:options:previousStateVariant:fragmentKeySize:                0
  freeFragmentFunctionKey:fragmentKeySize:                          0
  freeVertexFunctionKey:vertexKeySize:                              0
  generate3DStateConstants:                                         0
  generate3DStateDS:                                                0
  generate3DStateHS:                                                0
  generate3DStatePS:                                                0
  generate3DStateSBE:                                               0
  generate3DStateTE:                                                0
  generate3DStateURB:                                               0
  generate3DStateVF:                                                0
  generate3DStateVS:                                                0
  generate3DStateWM:                                                0
  generateBindingWorkarounds:                                       0
  generateBlendState:                                               0
  generateMultisampleState:                                         0
  generateShaderState:                                              0
  initWithDevice:pipelineStateDescriptor:                           0
  initWithDevice:pipelineStateDescriptor:vertexVariant:fragmentVariant:                0
  inputInfoAndSize:                                                 0
  isPostTessellationVertexFunction:                                 0
  isTessellationFactorScaleEnabled                                  0
  label                                                             0
  newRenderPipelineStateWithDescriptor:error:                       0
  newRenderPipelineStateWithDescriptor:options:reflection:error:completionHandler:                0
  newRenderPipelineWithDescriptor:vertexVariant:fragmentVariant:                0
  newSerializedVertexDataWithFlags:error:                           0
  rasterSampleCount                                                 0
  serialQueue                                                       0
  serializeFragmentData                                             0
  setAlphaBlendOperation:                                           0
  setBlendingEnabled:                                               0
  setDestinationAlphaBlendFactor:                                   0
  setDestinationRGBBlendFactor:                                     0
  setFragmentFunction:                                              0
  setFragmentIABEmulationData:                                      0
  setInputPrimitiveTopology:                                        0
  setRgbBlendOperation:                                             0
  setSourceAlphaBlendFactor:                                        0
  setSourceRGBBlendFactor:                                          0
  setStepFunction:                                                  0
  setStride:                                                        0
  setVertexDescriptor:                                              0
  setVertexFunction:                                                0
  setVertexIABEmulationData:                                        0
  stepFunction                                                      0
  stepRate                                                          0
  stride                                                            0
  tessellationControlPointIndexType                                 0
  tessellationFactorFormat                                          0
  tessellationFactorStepFunction                                    0
  tessellationPartitionMode                                         0
  validateWithVertexFunction:error:                                 0
  vertexAttributes                                                  0
  vertexDescriptor                                                  0
  vertexFunction                                                    0
  vertexFunctionKeyWithRenderPipelineDescriptor:options:nextStageVariant:vertexKeySize:                0
  initNewTextureDataWithDevice:descriptor:sysMemSize:sysMemRowBytes:vidMemSize:vidMemRowBytes:args:                0
  initWithDevice:descriptor:sysMemSize:sysMemRowBytes:vidMemSize:vidMemRowBytes:args:argsSize:                0
  initWithDevice:shmemSize:                                         0
  initializeFastClearMCS:descPriv:texData:                          0
  maxTextureHeight2D                                                0
  maxTextureWidth2D                                                 0
  newTextureWithDescriptor:                                         0
  renderPassDescriptor                                              0
  setDepth:                                                         0
  setHeight:                                                        0
  setMipmapLevelCount:                                              0
  setRenderTargetArrayLength:                                       0
  setShmemSize:                                                     0
  setStorageMode:                                                   0
  setTextureType:                                                   0
  setUsage:                                                         0
  setWidth:                                                         0
  allocPushConstantSpace:                                           0
  debuggingEnabled                                                  0
  denormsEnabled                                                    0
  distributeRemainingURBAllocations:countAvailable:gtVariant:                0
  fastMathEnabled                                                   0
  framebufferReadEnabled                                            0
  functionConstantsDictionary                                       0
  generateSamplerToTextureMap:table:                                0
  getCString:maxLength:encoding:                                    0
  getVariantID                                                      0
  hash                                                              0
  initWithData:                                                     0
  initWithLibraryContainer:device:                                  0
  languageVersion                                                   0
  layouts                                                           0
  lengthOfBytesUsingEncoding:                                       0
  maxColorAttachments                                               0
  newFunctionWithName:                                              0
  newFunctionWithNameInternal:                                      0
  newLibraryWithSource:options:error:                               0
  objectForKey:                                                     0
  preprocessorMacros                                                0
  programObject                                                     0
  releaseCacheEntry:                                                0
  renderTargetArrayIndexType                                        0
  respondsToSelector:                                               0
  setLanguageVersion:                                               0
  setupURBState:gtVariant:                                          0
  supportsSampleCount:                                              0
  supportsTextureSampleCount:                                       0
  setPixelFormat:                                                   0
  initWithDevice:options:args:argsSize:                             0
  getCompilerOutputData                                             0
  requiresIABEmulation                                              0
  format                                                            0
  offset                                                            0
  validateWithDevice:                                               0
  _removeResource:                                                  0
  removeResourceFromList:                                           0
  functionConstants                                                 0
  initWithBytes:length:encoding:                                    0
  needsFunctionConstantValues                                       0
  limits                                                            0
  arrayLength                                                       0
  buffer                                                            0
  bufferBytesPerRow                                                 0
  bufferOffset                                                      0
  descriptorPrivate                                                 0
  initWithResource:                                                 0
  initWithTexture:pixelFormat:textureType:levels:slices:                0
  initWithTextureInternal:pixelFormat:textureType:levels:slices:compressedView:                0
  mipmapLevelCount                                                  0
  newTextureViewWithPixelFormat:textureType:levels:slices:                0
  numFaces                                                          0
  sharedRef                                                         0
  pixelFormat                                                       0
  generateClearRequirements                                         0
  generateSurfaceStateTarget                                        0
  getUniqueID:clampMax:                                             0
  initializeHWSurfaceState                                          0
  bufferIndex                                                       0
  __new::                                                           1
  containsObject:                                                   1
  didScheduledPresent                                               1
  immutablePlaceholder                                              1
  initWithArray:range:copyItems:                                    1
  initWithDrawablePrivate:layer:                                    1
  initWithObjects:count:                                            1
  isCapturing                                                       1
  member:                                                           1
  nextDrawable                                                      1
  present                                                           1
  presentScheduledInsertSeed                                        1
  priv                                                              1
  releasePrivateReferences:                                         1
  removeAllObjects                                                  1
  removeObjectsInArray:                                             1
  removeObjectsInRange:inArray:range:                               1
  setPresentScheduledInsertSeed:                                    1
  setWithArray:range:copyItems:                                     1
  sharedCaptureManager                                              1
  vkpresent                                                         1
  _addResource:                                                     1
  addResourceToList:                                                1
  attributeIndex                                                    1
  attributeType                                                     1
  isActive                                                          1
  setBufferIndex:                                                   1
  setFormat:                                                        1
  setOffset:                                                        1
  attributes                                                        1
  calculateNumURBEntries:stageIndex:countAvailable:                 1
  populateURBState:stageIndex:startAddress:gtVariant:                1
  blitCommandEncoder                                                1
  initWithCommandBuffer:                                            1
  patchType                                                         1
  allocatedSize                                                     1
  __new:::                                                          2
  addCompletedHandler:                                              2
  initWithCapacity:                                                 2
  mutablePlaceholder                                                2
  copyFromBuffer:sourceOffset:sourceBytesPerRow:sourceBytesPerImage:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:                2
  setVertexBytes:length:atIndex:                                    3
  initWithDevice:                                                   4
  self                                                              5
  countByEnumeratingWithState:objects:count:                        5
  _delayedEventMatchingMask:pull:                                   6
  _nextEventMatchingEventMask:untilDate:inMode:dequeue:                6
  distantFuture                                                     6
  drain                                                             6
  isEqualToDate:                                                    6
  isMainThread                                                      6
  nextEventMatchingMask:untilDate:inMode:dequeue:                   6
  timeIntervalSinceNow                                              6
  _dispose                                                          6
  _xref_dispose                                                     6
  insertObjects:count:atIndex:                                      7
  replaceObjectsInRange:withObjects:count:                          7
  replaceObjectsInRange:withObjectsFromArray:range:                 7
  submitCommandBuffers:count:                                       7
  _submitAvailableCommandBuffers                                    7
  isNSArray__                                                      10
  setClearStencil:                                                 10
  distantPast                                                      12
  getObjects:range:                                                15
  setClearDepth:                                                   15
  timeIntervalSinceReferenceDate                                   18
  setTexture:                                                      23
  HasDCL3                                                          39
  renderEncoderInitHWState                                         39
  setClearColor:                                                   52
  commit                                                           60
  commitCommandBuffer:wake:                                        60
  didSchedule:error:                                               60
  didScheduleWithStartTime:endTime:error:                          60
  enqueueCommandBuffer:                                            60
  setCommitted:                                                    60
  isCommitted                                                      60
  commandBufferDidComplete:startTime:completionTime:error:               61
  didCompleteWithStartTime:endTime:error:                          61
  removeObject:                                                    61
  addObject:                                                       62
  commandBufferWithUnretainedReferences                            65
  getAndIncrementNumCommandBuffers                                 65
  initWithQueue:retainedReferences:                                65
  initWithQueue:retainedReferences:synchronousDebugMode:               65
  releaseAllObjectsAndReset                                        65
  removeObjectAtIndex:                                             68
  insertObject:atIndex:                                            68
  depth                                                            68
  depthAttachment                                                  68
  height                                                           68
  stencilAttachment                                                68
  width                                                            68
  _mutate                                                          70
  removeObjectsInRange:                                            75
  isProfilingEnabled                                              120
  setHasDCL3:                                                     129
  initWithCommandBuffer:descriptor:stateDescriptor:               129
  renderCommandEncoderWithDescriptor:                             129
  copy                                                            130
  endEncoding                                                     131
  commandBufferStorage                                            131
  fenceGroupPool                                                  131
  getScratchPageGPUAddress                                        131
  getStatLocations                                                131
  getStatOptions                                                  131
  globalTraceObjectID                                             131
  resourceSize                                                    139
  colorAttachments                                                159
  objectAtIndexedSubscript:                                       163
  initWithDevice:depthStencilDescriptor:                          169
  initializeDepthStencilState:                                    169
  isDepthWriteEnabled                                             169
  newDepthStencilStateWithDescriptor:                             169
  setReadMask:                                                    188
  setWriteMask:                                                   188
  deviceRef                                                       196
  autorelease                                                     204
  sampleCount                                                     210
  setBlendColorRed:green:blue:alpha:                              212
  count                                                           240
  setViewport:                                                    250
  commandQueue                                                    260
  pTelemetry                                                      261
  setCurrentCommandEncoder:                                       262
  setLoadAction:                                                  273
  setStoreAction:                                                 273
  texture                                                         274
  drawPrimitives:vertexStart:vertexCount:instanceCount:baseInstance:              298
  new                                                             324
  backFaceStencil                                                 339
  frontFaceStencil                                                339
  readMask                                                        339
  getUniqueID:                                                    369
  depthCompareFunction                                            383
  copyWithZone:                                                   393
  _descriptorPrivate                                              404
  setDepthClipMode:                                               407
  setRenderPipelineState:                                         410
  setStencilFrontReferenceValue:backReferenceValue:               438
  stencilCompareFunction                                          460
  device                                                          465
  setScissorRect:                                                 486
  depthFailureOperation                                           522
  stencilFailureOperation                                         522
  isStatEnabled                                                   523
  setDepthBias:slopeScale:clamp:                                  550
  setDepthStencilState:                                           597
  depthStencilPassOperation                                       602
  akPrivateResourceList                                           649
  ioAccelResourceList                                             649
  length                                                          679
  writeMask                                                       679
  drawIndexedPrimitives:indexCount:indexType:indexBuffer:indexBufferOffset:instanceCount:              712
  alloc                                                           865
  init                                                            873
  allocWithZone:                                                  875
  dealloc                                                         912
  class                                                           925
  fragmentIABEmulationData                                       1011
  getRenderPipelineState                                         1011
  vertexIABEmulationData                                         1011
  textureType                                                    1243
  isEqual:                                                       1288
  _descriptorAtIndex:                                            1300
  objectAtIndex:                                                 1670
  bindIndirectArgumentsWithFunctions:                            1779
  commandBufferResourceInfo                                      1807
  setFragmentBuffer:offset:atIndex:                              1964
  setFragmentSamplerState:atIndex:                               2874
  setVertexSamplerState:atIndex:                                 2874
  setVertexBuffer:offset:atIndex:                                3843
  setFragmentTexture:atIndex:                                    5615
  setVertexTexture:atIndex:                                      5615
  retain                                                        49918
  release                                                       51151
  objc_msgSend                                                 161044

And the dtrace script to produce it

#pragma D option quiet
BEGIN {
	self->count = 0;
}


pid$target::objc_msgSend:entry
{
	@msgCount[copyinstr(arg1)] = count();
	@msgCount["objc_msgSend"] = count();
}

pid$target::__ZN15portability_gfx5impls18gfxQueuePresentKHR17h17b127701ff0fa81E:entry
{
        self->count++;
	@msgCount["vkpresent"] = count();
        normalize(@msgCount, self->count);

}

bors bot added a commit that referenced this issue Jun 23, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

Co-authored-by: Dzmitry Malyshau <[email protected]>
bors bot added a commit that referenced this issue Jun 23, 2018
2164: [mtl] Borrowed commands r=grovesNL a=kvark

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends:

r? @gfx-rs/metallists 

This PR attempts to have lightweight software commands that don't take any heap space or own ObjC objects. In most cases, where a command list is live-recorded and executed once, this should reduce the amount of work we do per command, which is especially important if those commands are thrown away (e.g. because we are not inside a render pass).

My expectation would be to see an improvement in #2161 due to us doing less work. The actual results are somewhat shocking: with v-sync enabled I'm getting the same 59-60 fps as usual. With v-sync OFF, I'm getting between 25 and 50 fps now (which is lower than the previous 50-70). Not sure what's going on, the instrumental profile doesn't give a clue. Please check out the code.

Co-authored-by: Dzmitry Malyshau <[email protected]>
@jrmuizel
Copy link
Contributor

jrmuizel commented Jun 23, 2018

I took a look at where the majority of the -release calls were and here's the breakdown of the top callers per frame:

  MTLIGSamplTexture                                                56
  MTLIGAccelCommandQueue                                           59
  MTLIGAccelBuffere                                                69
  MTLIGAccelTextur                                                 70
  MTLIGAcceler                                                     87
  MTLRenderPassColorAttachmentDescriptorArrayInternal              89
  MTLRenderPassColorAttachmentDescriptorInternal                   94
  MTLIGAccelRenderCommandEncoder                                  113
  MTLRenderPassDescriptorInternal                                 203
  MTLIOAccelPooledResource                                        227
  NSAutoreleasePool                                               294
  MTLIGAccelCommandBuffer                                         298
  MTLIGAccelDevice                                                303
  MTLIGDepthStencilState                                         1087
  MTLIGRenderPipelineState                                       1103
  MTLIGSampler                                                  10834
  MTLIGAccelBuffer                                              13325
  MTLIGAccelTexture                                             21279

  total                                                         50958

Calling -release on MTLIGAccelTexture 21 thousand times per frame seems too high.

@jrmuizel
Copy link
Contributor

For comparison here's what it looks like with MoltenVK

  MTLIOAccelPooledResource                                         11
  MTLRenderPassStencilAttachmentDescriptorInternal                 75
  MTLRenderPassDepthAttachmentDescriptorInternal                   75
  MTLRenderPassColorAttachmentDescriptorArrayInternal             141
  MTLRenderPassDescriptorInternal                                 142
  __NSCFConstantString                                            147
  MTLRenderPassColorAttachmentDescriptorInternal                  153
  MTLIGAccelCommandBuffer                                         154
  MTLIGAccelDevice                                                292
  MTLIGAccelTexture                                               304
  MTLIGRenderPipelineState                                        445

  total                                                          1972

@grovesNL
Copy link
Contributor

grovesNL commented Jun 23, 2018

I did something similar last night (just logging all objc_msgSend calls):

https://gist.github.com/grovesNL/80453d146327c87432c30191f3b1f579

You can generate this output in Xcode with a breakpoint on objc_msgSend with a debug command of expr -- (void) printf("[%s %s]\n", (char *)object_getClassName($arg1),$arg2) and automatically continue after evaluating.

Most time seems to be retain/release on textures/samplers/buffers. This seems mostly due to how we use retain/release to match Rust ownership semantics, but we could bypass most of that and use raw pointers, or some other way.

@jrmuizel
Copy link
Contributor

Here are stack traces of where the majority of the texture retain/releases are happening from 23 frames:



              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x2e
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857c09f0+0x2f
              libscenesystem.dylib`0x00000000bcba2e90+0x104
              libscenesystem.dylib`0x00000000bcba3500+0x465
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              462

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libclient.dylib`0x00000000c2838c70+0x29f
              libscenesystem.dylib`0x00000000bcae0820+0xb4
              libvstdlib.dylib`0x000000000dd9ebd0+0x275
              libscenesystem.dylib`0x00000000bcc389c0+0x4f
              466

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x1c
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857c09f0+0x2f
              libscenesystem.dylib`0x00000000bcba2e90+0x104
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              537

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x2e
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857c09f0+0x2f
              libscenesystem.dylib`0x00000000bcba2e90+0x104
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              537

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcede190+0xc29
              libparticles.dylib`0x00000000bcec4d40+0xe30
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              552

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcede190+0xc29
              libparticles.dylib`0x00000000bcec4d40+0xe30
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              552

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              libpanorama.dylib`0x00000000c7a89b70+0x1df
              libpanorama.dylib`0x00000000c7a8cc40+0x2aa
              libpanorama.dylib`0x00000000c7a0ee60+0xb5
              572

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              librendersystemvulkan.dylib`0x00000000857bf2c0+0xb8
              libpanorama.dylib`0x00000000c7a82eb0+0x4c1
              libpanorama.dylib`0x00000000c7a89b70+0x83b
              572

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7b410+0x262c
              libpanorama.dylib`0x00000000c7a8a910+0x18e9
              572

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x465
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba5b70+0xe
              598

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x59a
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaf0bd0+0x20cd
              624

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x1c
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857bf3a0+0x16d
              libscenesystem.dylib`0x00000000bcb35790+0xb0
              libscenesystem.dylib`0x00000000bcb99420+0x1143
              libscenesystem.dylib`0x00000000bcb9af50+0x172
              624

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x2e
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857bf3a0+0x16d
              libscenesystem.dylib`0x00000000bcb35790+0xb0
              libscenesystem.dylib`0x00000000bcb99420+0x1143
              libscenesystem.dylib`0x00000000bcb9af50+0x172
              624

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x86e
              libpanorama.dylib`0x00000000c7a850f0+0x4dd
              libpanorama.dylib`0x00000000c7a0ee60+0x91
              libpanorama.dylib`0x00000000c7a0ee60+0x198
              libpanorama.dylib`0x00000000c7a0ec40+0x152
              670

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x86e
              libpanorama.dylib`0x00000000c7a850f0+0x4dd
              libpanorama.dylib`0x00000000c7a0ee60+0x91
              libpanorama.dylib`0x00000000c7a0ee60+0x198
              670

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x1c
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857c0850+0x3a
              libparticles.dylib`0x00000000bcec0d50+0x40a
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              681

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x2e
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857c0850+0x3a
              libparticles.dylib`0x00000000bcec0d50+0x40a
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              681

              libobjc.A.dylib`objc_msgSend
              Metal`-[MTLRenderPassDepthAttachmentDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassColorAttachmentDescriptorArrayInternal dealloc]+0x53
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              AppleIntelHD5000GraphicsMTLDriver`__ZN12IGRenderPassD2Ev+0x1e
              AppleIntelHD5000GraphicsMTLDriver`-[MTLIGAccelRenderCommandEncoder endEncoding]+0x113
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17h121eb14d7842cafcE+0x88
              libMoltenVK.dylib`__ZN8metal_rs7encoder17CommandEncoderRef12end_encoding17hdadf8d3e19e292c3E+0xb3
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command12EncoderState3end17hc5aaa1fd85b31078E+0xdb
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink13stop_encoding17hc25216176861bde2E+0x4a
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$15end_render_pass17h0a0eb310250198aeE+0x69
              libMoltenVK.dylib`__ZN15portability_gfx5impls19gfxCmdEndRenderPass17haadf6d6f70762c46E+0x2d
              libMoltenVK.dylib`vkCmdEndRenderPass+0x25
              librendersystemvulkan.dylib`0x00000000857b9e20+0x75
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              735

              libobjc.A.dylib`objc_msgSend
              Metal`-[MTLRenderPassStencilAttachmentDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassColorAttachmentDescriptorArrayInternal dealloc]+0x64
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              AppleIntelHD5000GraphicsMTLDriver`__ZN12IGRenderPassD2Ev+0x1e
              AppleIntelHD5000GraphicsMTLDriver`-[MTLIGAccelRenderCommandEncoder endEncoding]+0x113
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17h121eb14d7842cafcE+0x88
              libMoltenVK.dylib`__ZN8metal_rs7encoder17CommandEncoderRef12end_encoding17hdadf8d3e19e292c3E+0xb3
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command12EncoderState3end17hc5aaa1fd85b31078E+0xdb
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink13stop_encoding17hc25216176861bde2E+0x4a
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$15end_render_pass17h0a0eb310250198aeE+0x69
              libMoltenVK.dylib`__ZN15portability_gfx5impls19gfxCmdEndRenderPass17haadf6d6f70762c46E+0x2d
              libMoltenVK.dylib`vkCmdEndRenderPass+0x25
              librendersystemvulkan.dylib`0x00000000857b9e20+0x75
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              735

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x4e0
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaf0bd0+0x20cd
              816

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedc1a0+0x520
              libparticles.dylib`0x00000000bcec3670+0x11a7
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              888

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedc1a0+0x520
              libparticles.dylib`0x00000000bcec3670+0x11a7
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              888

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x59a
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
              920

              libobjc.A.dylib`objc_msgSend
              Metal`-[MTLRenderPassColorAttachmentDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassColorAttachmentDescriptorArrayInternal dealloc]+0x39
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              AppleIntelHD5000GraphicsMTLDriver`__ZN12IGRenderPassD2Ev+0x1e
              AppleIntelHD5000GraphicsMTLDriver`-[MTLIGAccelRenderCommandEncoder endEncoding]+0x113
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17h121eb14d7842cafcE+0x88
              libMoltenVK.dylib`__ZN8metal_rs7encoder17CommandEncoderRef12end_encoding17hdadf8d3e19e292c3E+0xb3
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command12EncoderState3end17hc5aaa1fd85b31078E+0xdb
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink13stop_encoding17hc25216176861bde2E+0x4a
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$15end_render_pass17h0a0eb310250198aeE+0x69
              libMoltenVK.dylib`__ZN15portability_gfx5impls19gfxCmdEndRenderPass17haadf6d6f70762c46E+0x2d
              libMoltenVK.dylib`vkCmdEndRenderPass+0x25
              librendersystemvulkan.dylib`0x00000000857ba350+0x1b
              librendersystemvulkan.dylib`0x0000000085796670+0x19f
              librendersystemvulkan.dylib`0x00000000857959d0+0x9c6
             1012

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x465
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
             1058

              libobjc.A.dylib`objc_msgSend
              Metal`-[MTLRenderPassColorAttachmentDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassColorAttachmentDescriptorArrayInternal dealloc]+0x39
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              Metal`-[MTLRenderPassDescriptorInternal dealloc]+0x31
              libobjc.A.dylib`__ZN11objc_object17sidetable_releaseEb+0x10a
              AppleIntelHD5000GraphicsMTLDriver`__ZN12IGRenderPassD2Ev+0x1e
              AppleIntelHD5000GraphicsMTLDriver`-[MTLIGAccelRenderCommandEncoder endEncoding]+0x113
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17h121eb14d7842cafcE+0x88
              libMoltenVK.dylib`__ZN8metal_rs7encoder17CommandEncoderRef12end_encoding17hdadf8d3e19e292c3E+0xb3
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command12EncoderState3end17hc5aaa1fd85b31078E+0xdb
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink13stop_encoding17hc25216176861bde2E+0x4a
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$15end_render_pass17h0a0eb310250198aeE+0x69
              libMoltenVK.dylib`__ZN15portability_gfx5impls19gfxCmdEndRenderPass17haadf6d6f70762c46E+0x2d
              libMoltenVK.dylib`vkCmdEndRenderPass+0x25
              librendersystemvulkan.dylib`0x00000000857b9e20+0x75
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
             1060

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedef20+0xbe9
              libparticles.dylib`0x00000000bcec4d40+0xf82
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
             1104

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedfc60+0xbf2
              libparticles.dylib`0x00000000bcec4d40+0xeea
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
             1104

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
             1104

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedef20+0xbe9
              libparticles.dylib`0x00000000bcec4d40+0xf82
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
             1104

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcedfc60+0xbf2
              libparticles.dylib`0x00000000bcec4d40+0xeea
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
             1104

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              librendersystemvulkan.dylib`0x00000000857bf2c0+0xb8
              libpanorama.dylib`0x00000000c7a82eb0+0x4c1
              libpanorama.dylib`0x00000000c7a87600+0xade
             1144

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              librendersystemvulkan.dylib`0x00000000857bf2c0+0xb8
              libpanorama.dylib`0x00000000c7a87600+0x396
              libpanorama.dylib`0x00000000c7a0ee60+0xdc
             1144

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba5b70+0xe
             1150

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libclient.dylib`0x00000000c2838c70+0x29f
              libscenesystem.dylib`0x00000000bcae0820+0xb4
              libvstdlib.dylib`0x000000000dda4740+0x38e
              libtier0.dylib`__ZN7CThread10ThreadProcEPv+0x132
             1166

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x4e0
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
             1230

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a8cc40+0x2935
              libpanorama.dylib`0x00000000c7a0ee60+0xb5
              libpanorama.dylib`0x00000000c7a0ee60+0x198
              libpanorama.dylib`0x00000000c7a0ec40+0x152
             1234

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a8cc40+0x2935
              libpanorama.dylib`0x00000000c7a0ee60+0xb5
              libpanorama.dylib`0x00000000c7a0ee60+0x198
             1234

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              libclient.dylib`0x00000000c305fc20+0x8b8
              libpanorama.dylib`0x00000000c7a84a30+0x535
              libpanorama.dylib`0x00000000c7a0ee60+0x157
             1288

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a86d60+0x84d
             1368

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a850f0+0x4dd
              libpanorama.dylib`0x00000000c7a0ee60+0x91
              libpanorama.dylib`0x00000000c7a0ee60+0x198
              libpanorama.dylib`0x00000000c7a0ec40+0x152
             1506

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a850f0+0x4dd
              libpanorama.dylib`0x00000000c7a0ee60+0x91
              libpanorama.dylib`0x00000000c7a0ee60+0x198
             1554

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x86e
              libpanorama.dylib`0x00000000c7a82eb0+0x1156
             1716

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x86e
              libpanorama.dylib`0x00000000c7a82eb0+0x812
             1716

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a850f0+0x4dd
             1794

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x1c
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857bea00+0x3b
              libscenesystem.dylib`0x00000000bcb4b200+0x1c5
              libscenesystem.dylib`0x00000000bcb4b980+0x81
              libscenesystem.dylib`0x00000000bcba5530+0x154
             1818

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$8truncate17h570045b9f662146aE+0x73
              libMoltenVK.dylib`__ZN33_$LT$alloc..vec..Vec$LT$T$GT$$GT$5clear17hcd17863a9742153eE+0x19
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources5clear17hcbdf13b6bd848169E+0x25
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command5State15reset_resources17hf599ad6b84ccbeaeE+0x2e
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$5reset17h9a6731bada0d23a4E+0x21
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxResetCommandBuffer17hb738faef8c2b6f45E+0x62
              libMoltenVK.dylib`vkResetCommandBuffer+0x2b
              librendersystemvulkan.dylib`0x00000000857b8720+0x5e9
              librendersystemvulkan.dylib`0x00000000857bea00+0x3b
              libscenesystem.dylib`0x00000000bcb4b200+0x1c5
              libscenesystem.dylib`0x00000000bcb4b980+0x81
              libscenesystem.dylib`0x00000000bcba5530+0x154
             1818

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7b410+0x262c
              libpanorama.dylib`0x00000000c7a8cc40+0x101e
              libpanorama.dylib`0x00000000c7a0ee60+0xb5
              libpanorama.dylib`0x00000000c7a0ee60+0x198
              libpanorama.dylib`0x00000000c7a0ec40+0x152
             2112

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7b410+0x262c
              libpanorama.dylib`0x00000000c7a8cc40+0x101e
              libpanorama.dylib`0x00000000c7a0ee60+0xb5
              libpanorama.dylib`0x00000000c7a0ee60+0x198
             2160

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcec0d50+0x2474
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libclient.dylib`0x00000000c305fc20+0x1fb2
              libpanorama.dylib`0x00000000c7a84a30+0x535
             2944

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcec0d50+0x2474
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libclient.dylib`0x00000000c305fc20+0x1fb2
             3450

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libclient.dylib`0x00000000c1f5ca20+0x7b2
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
              libscenesystem.dylib`0x00000000bcacd720+0x1dc
              libvstdlib.dylib`0x000000000dd9ebd0+0x275
             6226

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libclient.dylib`0x00000000c1f5ca20+0x7b2
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
              libscenesystem.dylib`0x00000000bcacd720+0x1dc
              libvstdlib.dylib`0x000000000dda4740+0x38e
             6860

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7e380+0x9f2
              libpanorama.dylib`0x00000000c7a8cc40+0x2935
             8244

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857c0b80+0x4a
              librendersystemvulkan.dylib`0x00000000857c0b20+0x18
              libpanorama.dylib`0x00000000c7a7b410+0x262c
              libpanorama.dylib`0x00000000c7a8cc40+0x101e
            10940

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x59a
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba57d0+0xcc
            11134

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba57d0+0xcc
            11692

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x59a
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
            13302

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcec0d50+0x2474
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba57d0+0xcc
            13388

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libclient.dylib`0x00000000c1f5ca20+0x7b2
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcaee350+0x222d
              libscenesystem.dylib`0x00000000bcacd720+0x1dc
            13592

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x7d5
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
            16854

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17haf634771414e09deE+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h8e5e25f5b38cc195E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h7aca90669c880c8bE+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h89700e24773e9425E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hb0293488909678bbE+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink17begin_render_pass17h3c147fc7c5772908E+0x418
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$17begin_render_pass17hc1b820e845badc73E+0xdb0
              libMoltenVK.dylib`__ZN15portability_gfx5impls21gfxCmdBeginRenderPass17hab0255577ed7e386E+0x137
              libMoltenVK.dylib`vkCmdBeginRenderPass+0x33
              librendersystemvulkan.dylib`0x00000000857ba0d0+0x26c
              librendersystemvulkan.dylib`0x00000000857b9e20+0x95
              librendersystemvulkan.dylib`0x00000000857bbb90+0x1bc
              librendersystemvulkan.dylib`0x00000000857bea00+0x309
              libpanorama.dylib`0x00000000c7a89a00+0x8d
              libpanorama.dylib`0x00000000c7a88c90+0xce4
              libpanorama.dylib`0x00000000c7a0ee60+0xa3
            18692

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcec0d50+0x2474
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba5b70+0xe
            18800

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x4e0
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba57d0+0xcc
            24738

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bca7bd20+0x4d7
              libscenesystem.dylib`0x00000000bca7cd60+0x4e0
              libscenesystem.dylib`0x00000000bca7d330+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
            28428

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libparticles.dylib`0x00000000bcec0d50+0x2474
              libparticles.dylib`0x00000000bcd7fb40+0x404
              libparticles.dylib`0x00000000bcd6ec60+0x2bc
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
            34488

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command14StageResources12add_textures17h8a63d87b7e14ad12E+0x24d
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0xca7
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x465
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
              libscenesystem.dylib`0x00000000bcba57d0+0xcc
            44174

              libobjc.A.dylib`objc_msgSend
              libMoltenVK.dylib`__ZN60_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$6invoke17ha1d98ea9d22e5853E+0x3a
              libMoltenVK.dylib`__ZN4objc7message15send_unverified17hc01f333448f177d1E+0x88
              libMoltenVK.dylib`__ZN8metal_rs8obj_drop17h3df9aa87c251f74dE+0xb3
              libMoltenVK.dylib`__ZN68_$LT$metal_rs..texture..Texture$u20$as$u20$core..ops..drop..Drop$GT$4drop17hdd5d029014dc3f92E+0x18
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h4fe5183fe8c4e801E+0x15
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h25cc3189a46ed336E+0x2e
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17h2711cd568f7862b8E+0x3b
              libMoltenVK.dylib`__ZN4core3ptr13drop_in_place17hdcfc0fbeb5c21ff2E+0x95
              libMoltenVK.dylib`__ZN17gfx_backend_metal7command11CommandSink19pre_render_commands17h41295a2120bcd21dE+0x2f6
              libMoltenVK.dylib`__ZN135_$LT$gfx_backend_metal..command..CommandBuffer$u20$as$u20$gfx_hal..command..raw..RawCommandBuffer$LT$gfx_backend_metal..Backend$GT$$GT$29bind_graphics_descriptor_sets17hfa3900a41ff45d8fE+0x1c70
              libMoltenVK.dylib`__ZN15portability_gfx5impls24gfxCmdBindDescriptorSets17hc58f0b2a55217a63E+0x1bb
              libMoltenVK.dylib`vkCmdBindDescriptorSets+0x86
              librendersystemvulkan.dylib`0x00000000857b4290+0x57b
              librendersystemvulkan.dylib`0x00000000857c0b80+0x195
              librendersystemvulkan.dylib`0x00000000857c0dd0+0x1c
              libscenesystem.dylib`0x00000000bcba2e90+0x428
              libscenesystem.dylib`0x00000000bcba3500+0x465
              libscenesystem.dylib`0x00000000bcba34d0+0x25
              libscenesystem.dylib`0x00000000bcba5a20+0xf1
            48978
  vkpresent                                                        23
  vkBeginRenderPass                                              3086
  MTLIGAccelTexture                                            403754

A big chunk of them are from pre_render_commands() and add_textures(). It feels like we're cloning them way more than we need to.

@grovesNL
Copy link
Contributor

grovesNL commented Jun 23, 2018

Thanks @jrmuizel! I think we'll try to avoid these clones entirely by working with the pointer of the native object directly and upgrading them (probably transmute) to *Ref types whenever we need to do something with them. We shouldn't need this reference counting on the Objective-C side at all for these objects.

@grovesNL
Copy link
Contributor

We should also consider disabling all interactions with the capture manager in release, or hide behind a feature.

bors bot added a commit that referenced this issue Jun 25, 2018
2175: mtl: Use pointers for temporary state r=kvark a=grovesNL

Partially fix to #2161
PR checklist:
- [ ] `make` succeeds (on *nix)
- [ ] `make reftests` succeeds
- [ ] tested examples with the following backends:

This replaces buffer/texture/sampler object usage with pointers instead. This allows us to bypass Objective-C's reference counting (i.e. retain/release). The idea is to create the same objects as before, but purposely leak them and acquire a pointer to them instead. When we're finished with the objects, we take back ownership and release as usual. We should be able to do this either through `drop` on the owning struct, or through relevant `destroy_*` functions.

Still heavily WIP (but works with quad/Dota 2) because there are a few places that will cause leaks or use-after-free in its current state (need to fix cases where the same image is returned for an image view, verify `drop` exists on all the correct owners, etc.)

Co-authored-by: Joshua Groves <[email protected]>
@grovesNL
Copy link
Contributor

Dota also attempts to use VK_FORMAT_BC1_RGB_UNORM and VkPipelineCache, so we should ensure we support them.

@kvark
Copy link
Member Author

kvark commented Jun 25, 2018

Too many command buffers

I tried to reduce the capacity and didn't see any visible impact. Created #2180 to possibly address this properly in the future.

@kvark
Copy link
Member Author

kvark commented Jun 27, 2018

With all the optimization coming (e.g. #2183) the heaviest call stack that Instruments show is about creating a render pass. We do this on Vulkan render passes of course, but also quite a few times for the VkCmdClearImage - once for each (slice, level) pair. There is a faster path (see CLEAR_IMAGE_ARRAY) currently blocked by OSX driver issues (partly filed as 40769756). What I do wonder about though is how our render pass creation count is different from MoltenVK at the moment. The target message is [MTLDebugCommandBuffer remderCommandEncoderWithDescriptor].

bors bot added a commit that referenced this issue Jun 28, 2018
2183: Heap-less descriptor sets in Metal r=grovesNL a=kvark

Fixes the heap allocations in descriptor sets, to help #2161.
Edit: the improvement is there, but it's not the one we were looking for.

Adds real value semantics for the `DescriptorPool`, which now owns the allocation and has the actual descriptor sets pointing to it.
PR checklist:
- [x] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends: metal

~~Note: it doesn't work correctly yet, but I'd be happy to get the review comments and notes. Perhaps, you can spot the mistake? ;)
Sadly, my CTS is non-operational atm, so it's not easy to figure out a test case.
Note2: the individual commits are not build-able. Will squash once finished.~~

Co-authored-by: Dzmitry Malyshau <[email protected]>
bors bot added a commit that referenced this issue Jun 28, 2018
2185: Various Metal performance optimizations r=grovesNL a=kvark

Helps #2161 . I'm now getting 80-85 fps on the test run.
Aside from Metal, also changes HAL to avoid heap allocation for vertex buffer binding.
PR checklist:
- [x] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends: metal


Co-authored-by: Dzmitry Malyshau <[email protected]>
@kvark
Copy link
Member Author

kvark commented Jun 28, 2018

Just updated a bunch of items in the issue body.

What I do wonder about though is how our render pass creation count is different from MoltenVK at the moment.

Those counts are already provided by @jrmuizel . We call it 129 times versus 108 in Molten. Still a difference to investigate, but not one to be too worried about.

The engine operates within an assumption that command buffer recording is cheap, while submission is expensive.

I clarified the threading model now, and there appears to be no good reason for us to be CPU bound, on a main thread where we only run 1/3 of time... Dota just needs to use more threads for the job execution.

Too much heap (re)allocation

Some of it is addressed in #2185. Iterators are fine where they are applicable. In other places we have to resort to SmallVec, which is never optimal and often causes too much data moves... Also we really need dynamic on-stack allocation (alloca) to help here - rust-lang/rust#48055.

Too many ObjectiveC messages

With the actual threading model clarified, I now see ~5% time spent by MoltenVK in objc_msgSend. Thus, I don't think there is any difference in how effective we are at message sending, but rather in a sheer number of the sends we do.

@kvark
Copy link
Member Author

kvark commented Jul 20, 2018

Here is what our frame looks like when in immediate recording mode:
trace-dota-gfx-immediate

Some observations:

  • a frame consists of one large submission (fenced), then present call, then one dummy submission (also fenced)
  • all the work is done in that large submission, but it's actually spread over 2 different frames if we believe the colors. So a frame submission has some command buffers from the next frame?..
  • the driver doesn't bother doing any work until we actually commit() - Metal enqueue() advantage #2232
  • a lot of command buffers sit idle after being recorded
  • there isn't that much overlap between GPU and CPU work happening, unfortunately, so there is a big room for optimization, presumably on the application side

@fkaa
Copy link
Contributor

fkaa commented Jul 21, 2018

Ran Vulkan Dota 2 on Windows using RGP (trace file) to compare:
rgp trace

bors bot added a commit that referenced this issue Jul 24, 2018
2264: Render pass descriptor cache for Metal r=grovesNL a=kvark

~~Includes #2260~~
Fixes two of the performance issues in  #2161 (RP desc locking and copying costs).

Immediate recording FPS doesn't seem to change, maybe slightly lower (touching 90 from below more than from above, as I recall it doing - need to confirm. Edit - confirmed to not be caused by the PR).
Deferred recording FPS seem to go from lower 100s to higher, or even from 100 to 110, roughly speaking. There are barely any bottlenecks left for it, outside of the general architecture. Main thread now spends about 14.5% in our code, which is mostly covered by driver interaction.

PR checklist:
- [ ] `make` succeeds (on *nix)
- [x] `make reftests` succeeds
- [x] tested examples with the following backends: Metal
- [ ] `rustfmt` run on changed code


Co-authored-by: Dzmitry Malyshau <[email protected]>
@kvark
Copy link
Member Author

kvark commented Dec 28, 2018

I believe this can be closed now, our results were published on the blog

@kvark kvark closed this as completed Dec 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants