GLTFLoader: record association between glTF elements and Three.js objects #19257
Comments
|
If this request is broadly acceptable, I would like to suggest that approach (1) suggested by @donmccurdy seems preferable because it will be easier to retain associations when making clones of the scene graph. |
|
@cdata approach (1) sounds good to me 👌 |
|
One hitch with approach (1) is that some of the objects created by Three.js
do not have a `userData` property. For example, `AnimationClip` and
`BufferAttribute`.
Would it make sense to add `userData` to these? Or perhaps this is a good
reason to go with option (2) instead?
…On Fri, May 1, 2020, 5:45 PM Don McCurdy ***@***.***> wrote:
@drcmda <https://github.com/drcmda> and @fms-cat
<https://github.com/FMS-Cat>: I think you've both asked about variations
of this in the past. Would either of these approaches be helpful for you?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB2TU6CCO6AO5CNLCSCEC3RPNUKDANCNFSM4MUBEYTA>
.
|
|
i would most definitively prefer a lookup table. react-three-fibers useLoader adds "nodes" and "materials" to the gltf data. with these two the entire graph can be represented. this enables full automation, for instance this is how a file from sketchfab looks like: https://codesandbox.io/s/romantic-tharp-0buje?file=/src/Model.js in plain threejs it could look like this: new GLTFLoader().load(url, ({ nodes, materials }) => {
nodes.cube.receiveShadow = true
materials.metal.roughness = 0.5
...
})users now could go ahead and build their own graphs: const mesh = new THREE.Mesh(nodes.cube.geometry, materials.metal)
scene.add(mesh)as for userData, i think object mutation isn't the best choice imo. i don't like mutation in general, and it still needs traversal in the end, which is the very thing that makes handling gltf so hard in threejs. if you add a lookup table with gltf indices that would be nice. official support for nodes/materials would be welcome, too - then i can remove it from r3f. if you go for a index table alone, pretty please dont name it "nodes" 👀 |
|
My original intention of the idea 1 was to load the custom material from the glTF and that still should be achieved using plugins. End developer have to delete the userData field when they try to clone a gltf derived materials in this case, and that sounds weird a bit 🤔 |
I'm not inclined to include every possible object type, until we have concrete reasons to do so. If the solution happens to handle everything easily, neat, but it can probably wait.
I'd be OK with exposing some kind of lookup table, either on GLTFLoader's result object or on the GLTFParser it already returns. It probably should not rely on unique names, which are not guaranteed, but indices or a Map would work.
I don't understand — the name conflicts with something you need? |
the names though is what makes it usable - and typesafe - , otherwise to find out where "robotLeg" is you traverse again, as opposed to: const { nodes: { robotLeg } } = await new GLTFLoader<Robot>().loadAsync(url)
robotLeg.receiveShadows = truesomeone could hack into a gltf with a text editor and use one name multiple times, it isn't possible in modeling software like blender and unnamed objects are named by three with an additive index. all it would do is overwrite a named slot, no harm done. but we're forcing all users to traverse in order to find back their assets because of it.
if you do decide to add something that's working differently than a name:value collection, please consider naming it something else than "nodes" and "materials" if it's in any way possible. i know that's a lot to ask for, but it would throw us into major breaking changes. |
|
@drcmda @donmccurdy I have taken to referring to the glTF descriptors as glTF "elements." Not sure if that helps the discussion, but worth considering in case you want another word to refer to those things collectively 💁 It sounds like a look-up table is the preferred approach. I took a quick stab at this last week. Perhaps I could put up a PR for considering?
Is there a reason we should keep in mind for why you might be disinclined to include some object types? Subtle complexity or anticipated changes in the implementation, perhaps? |
We do see objects with duplicate names periodically; there have been multiple bug reports against three.js as a result. That's solvable: I think GLTFLoader should automatically rename duplicates. But more importantly, mapping from a unique name (possibly generated by GLTFLoader) to a three.js object does not provide the mapping back to the glTF definition that @cdata presumably needs. You would need to also return an annotated version of the original glTF JSON, with new names. 😕
Mostly I'm nervous of modifying three.js' API just to match glTF, if that change doesn't also bring some real benefit to the API itself. And, there are various objects (samplers, textures vs images) that exist in glTF but don't obviously map to a three.js object in the final scene. Mapping nodes, meshes, and materials is a much more tractable problem, but still has plenty of edge cases. |
|
I'm definitely in agreement about not using names to map these things. The glTF spec is really explicit about not being able to rely on name uniqueness:
Source: https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#indices-and-names
FWIW all that matters to me at the end of the day is that it is possible to cross-associate between the two hierarchies. The relative stability of this association over time is not nearly as important. That said, I appreciate that any nature of association will be considered an implicit contract in the the API. To look at this somewhat differently, if it were possible for a three.js/examples/jsm/loaders/GLTFLoader.js Lines 1591 to 1665 in 51c70b6 It might be sufficient to support a callback that is invoked at the tail-end of loading each dependency. For example: GLTFParser.prototype.getDependency = function ( type, index ) {
// ...
if ( ! dependency ) {
switch ( type ) {
case 'scene':
dependency = this.loadScene( index );
break;
// ...
}
if ( this.dependencyCallback ) {
this.dependencyCallback( type, index, dependency );
}
}
// ...
};The nice thing about this approach is that it is simple to implement and the structure of glTF is the primary driver of the API (rather than the ways in which the glTF is interpreted in terms of Three.js). This perhaps conceptually lightens the burden on Three.js implementation. WDYT @donmccurdy ? |
|
Although I should note that the use case is orthogonal to the current feature request, it is worth considering that the This isn't to suggest we should support this, but it is an interesting capability that is potentially unlocked with the right design. The idea came up in discussion with some folks from 3D Commerce today while we were speculating about how we might experiment with |
|
Big 👍 for this. I built a glTF compression tool in three.js (lets you optimize the mesh + textures) and it was a pain having to maintain all those references manually. |
|
That use case is meant to be addressed by #19144. Feedback and testing on the PR would be welcome, although we'd like to keep it fairly simple to start. I think providing a lookup table would probably be independent of that extension API, but open to suggestions. |
|
@pushmatrix this is still in progress, but for use cases like optimizing a model you may find https://github.com/donmccurdy/glTF-Transform helpful. |
|
@donmccurdy in the fullness of time, we will need to participate in parsing of the entire scene graph. I want to make sure that if we begin to build some cross-association between the glTF scene graph and the Three.js one, that it is coherent for all parts of the glTF (eventually). I'm happy to implement that in whatever form makes sense to you, while being sensitive to your feedback in #19257 (comment) The dependency callback approach seems like a nice middle-ground, as it doesn't explicitly require some specific Three.js output, but does give us enough information to know what part of a glTF a With regards to the plug-in system, that seems relatively complex and doesn't seem to cover most of the glTF scene graph yet. With your feedback in mind, would you prefer that we focus on giving that PR feedback rather then implement cross-associations in some other way? |
|
cc @takahirox I may be overestimating the utility of having a single dependency callback; maybe you can advise me 🙏 |
|
A per-dependency callback doesn't provide enough flexibility to implement existing extensions, which the targeted hooks in #19144 would allow. We can certainly add more hooks, especially following the If the addition of |
|
Just to be clear, I'm asking for a way to know the relationship between two
scene graphs. The hypothetical ability to implement extensions was a
possible bonus, but ultimately orthogonal to the feature request.
I'm not seeking to build up alternative interpretations of the glTF being
parsed; I seek to reflect changes made to the live Three.js scene graph
(regardless of shape) back to the glTF that it represents.
My problem boils down to the fact that information is lost when GLTFLoader
parses the glTF and creates the Three.js scene graph. A dependency hook
seems like a good place where the lost information could be observed and
preserved. But, there are other ways to preserve this information as
discussed.
…On Sat, May 9, 2020, 2:38 PM Don McCurdy ***@***.***> wrote:
A per-dependency callback doesn't provide enough flexibility to implement
existing extensions, which the targeted hooks in #19144
<#19144> would allow. We can
certainly add more hooks, especially following the loadFoo pattern there.
But complete flexibility to reinterpret any part of the glTF schema is not
a goal of GLTFLoader; I think that will always be the domain of custom
code. Our goal for now is to enable most known use cases, but not to get
bogged down with hypothetical ones.
If the addition of loadNode (or similar) to #19144
<#19144> would provide what you
need, let's use that? If you expect to also need hooks for accessors,
samplers, buffer views, and other minutiae of the glTF spec, I'm not sure
GLTFLoader can provide it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB2TU3JY5ITXWBLS2W5YS3RQXEPBANCNFSM4MUBEYTA>
.
|
|
Typically the term "scene graph" refers only to objects with positions in the scene. Materials, accessors, animations, samplers, etc. are not typically considered part of the scene graph. Mapping the relationship between the three.js scene graph and the glTF scene graph, we certainly can do, and perhaps a bit more (e.g. materials). I'm not sure if that's what you're asking here, or if you need a mapping for everything that comes out of the glTF file? |
|
I'm perhaps using a pedantic definition of graph, but to me nodes are just
elements of the graph that can have edges leading to more nodes, while
everything else is by structure limited to being a leaf in the graph (e.g.,
materials may not refer to nodes). If I were referring to an ECS, I might
include components as elements of the scene graph.
I don't think the definition weighs heavily on the nature of the request,
but I apologize for making it more confusing than it needs to be.
There is a glTF, and there is the in-memory realization of that glTF
expressed in terms of Three.js classes. I want to be able to make a change
to the Three.js classes and (assuming the change is in the scope of things that
can be expressed with glTF) reflect that change back to the original glTF
without also making unrelated changes in the process.
In time, the scope of this use case could encompass all top-level
constructs defined by the glTF spec, although in the immediate future I
only care about materials, and only because it is not possible to
disambiguate them in some circumstances. It may not be necessary to do much
additional work beyond support for materials because GLTFLoader already
yields most of the information needed to correlate the glTF with the
Three.js in-memory representation of that glTF.
…On Sat, May 9, 2020, 3:00 PM Don McCurdy ***@***.***> wrote:
Typically the term "scene graph" refers only to objects with positions in
the scene. Materials, accessors, animations, samplers, etc. are not
typically considered part of the scene graph.
Mapping the relationship between the three.js scene graph and the glTF
scene graph, we certainly can do, and perhaps a bit more (e.g. materials).
I'm not sure if that's what you're asking here, or if you need a mapping
for everything that comes out of the glTF file?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#19257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB2TU5GIF5WI4FV2UJHKZ3RQXG6PANCNFSM4MUBEYTA>
.
|
|
Sorry for the wording nit; I'm trying to understand the scope of what you're asking for, and in that sense the scene graph and the glTF dependency graph are very different. I do not think we should provide mappings for all top-level constructs in the glTF spec from GLTFLoader. Mappings for nodes and materials would be fine, sure. But your longer-term goal here is likely to require a custom loader. |
|
GLTFParser's internal cache was never intended as a public API. It may change over time, and I would not expect those changes to appear in the changelog. The goal here is to arrive at a public API that is not at risk of changing as a result of internal refactoring. I'm not sure how to answer a question about supporting discrete edits, the full implications of that question are unclear to me. But as for mappings/associations: I would strongly prefer that GLTFLoader not be burdened with maintaining mappings to low-level (but still top-level, as far as schema is concerned) glTF elements like buffers, buffer views, accessors, and samplers. The other elements are open for discussion, but nodes and materials seem like the place to start. |
|
I have been operating on the assumption that being able to map glTF dependencies to their associated Three.js constructs is a prerequisite to enabling discrete edits to the glTF. This might not be correct, and I would probably benefit from the wisdom of others if that is the case. It sounds like there is some agreement to disambiguate materials and nodes, and that is a reasonable starting place to me. I reviewed react-three-fiber's implementation. Although it uses names (which seems perilous), we could probably substitute indexes and get similar complexity when correlating the two trees. Thank you @donmccurdy for the clarifying discussion. At this point I would like to proceed with a proposal that implements something along the lines of (2) as described in the original issue text, specifically: a |
|
I have opened a PR for your consideration: #19359 |
and others
Note: I don't want to bury the lede for this request, so I moved a brief summary to the top of the issue. Please refer to the second and third headings for a more detailed background and rationale!
Proposal
When loading models with
GLTFLoader, it would be nice if we could retain associations between glTF scene graph elements and the Three.js objects that correspond to them. I briefly discussed this problem with @donmccurdy and he suggested that one of the following approaches might be acceptable:userDatae.g.,material.userData.gltfMeta.index = 3parser.getDefinition( material: THREE.Material ): GLTF.MaterialWhat do others think about this problem? Would you be willing to consider a change that makes either of the suggested enhancements?
Background
Three.js'
GLTFLoadercurrently produces an artifact that contains a Three.js scene graph as well as aGLTFParserinstance that holds the original, deserialized glTF.The
GLTFParserholds associations between the elements in the original glTF and the Three.js scene graph. These associations are expressed both byGLTFParser.prototype.getDependenciesand thecacheobject on theGLTFParserinstance.In some cases, the relationship between the original glTF element and the associated Three.js objects is obscured and cannot be trivially re-established. For example, materials associated with a skinned mesh are cloned, and the resulting clones are cached against a key that cannot be reconstructed easily:
three.js/examples/jsm/loaders/GLTFLoader.js
Lines 2118 to 2127 in 648c4bb
Description of the problem
Consider a model editor that allows a user to:
In an ideal world, the editor should faithfully re-export the model with its original hierarchy and shape. The only meaningful change in the exported file should be the one made interactively by the user.
In the pursuit of this goal, it is very useful (and probably required) that an association between the source glTF scene graph and the Three.js scene graph be made, so that for any change we make to a glTF element, we can reflect that change in the rendered Three.js scene in real-time without completely re-loading the glTF.
Some of the information required to correctly correlate the glTF scene graph with the Three.js scene graph is lost or obscured, and this makes the above described editor scenario very difficult (perhaps impossible) to achieve.
Three.js version
Browser
OS
Hardware Requirements (graphics card, VR Device, ...)
The text was updated successfully, but these errors were encountered: