several orbitcontrols: one per canvas in several views?

[MainzerKaiser]

I want to render several models and each shall have its own canvas and orbitcontrols. I made that work by the following code, but I make two observation:

1. clicking first time on a model doesnt initiate the rotation and looking into your code you somehow try to detect the first touch and only regard everything afterwards as rotation control. is there a way to customize that?
2. when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

import React, { Suspense, useRef, useMemo, useContext, useCallback, useEffect, useState, lazy } from 'react';
import Chair from './src/components/chair';
import { SafeAreaView, StyleSheet, View } from 'react-native';
import { Canvas } from '@react-three/fiber/native';
import useControls from 'r3f-native-orbitcontrols';
import * as THREE from "three";
import Trigger from './src/components/Trigger';
import Loader from './src/components/Loader';
import { useVisibleStore } from './src/context/MyZustand';
import { Dimensions } from 'react-native';


const Game: React.FC = () => {
  const { width, height } = Dimensions.get('window');
  const isPortrait = height > width;
  const nCubes= useVisibleStore((state) => state.nCubes)
  const [loading, setLoading] = useState<boolean>(false);

  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    return { OrbitControls, events, index };
  });

return (
      <SafeAreaView style={styles.container}>
        <View style={styles.modelsContainer}>
        {loading && <Loader />}
          <View style={isPortrait ? styles.columnContainer : styles.rowContainer}>
            {controlsAndEvents.map(({ OrbitControls, events, index }) => (
              <View key={index} style={isPortrait ? styles.modelColumn : styles.modelRow}{...events}>
                  <Canvas key={index}>
                    <OrbitControls
                      key={index} 
                      enabled={true}
                      enableZoom={false}
                      enablePan={false}
                      dampingFactor={0.5}
                      enableRotate={true}
                      rotateSpeed={0.5} 
                      // onChange={(event) => handleCameraChange(event, index, letters)} 
                      {...OrbitControls}
                    />
                    <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                    <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                    <Suspense fallback={<Trigger setLoading={setLoading} />}>
                      <Chair />
                    </Suspense>
                  </Canvas>
              </View> 
             ))}
          </View>
         </View>
      </SafeAreaView>
  );
};

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: 'grey', // Add this line to set the background color to grey
  },
  modelsContainer: {
    flex: 0.9,
  },
   columnContainer: {
    // flexDirection: 'column',
    flex: 1,
  },
  rowContainer: {
    // flexDirection: 'row',
    flex: 1,
  },
  modelColumn: {
    flex: 1,
    justifyContent: 'center',
  },
  modelRow: {
    flex: 1,
    justifyContent: 'center',
  },
});

export default Game;

[TiagoCavalcante]

I think (1) should be solved in 1.0.10 — can you check it? —. I still have to look at (2).

[MainzerKaiser]

Hi Tiago,
thanks a lot for responding. your fix in 1.0.10 works very well on web and android!

Regarding the second topic, I uploaded this animation, which is a recording from my android phone. (android 12 and expo on dev server)
What you don't see is the touch location of my finger, but I am moving the middle chair and hover around the canvas border to the upper chair. In the web version, I can keep touching even over other canvases/chairs, and I still move the chair where I started touching the screen. On android there is always a reset of the camera location as soon as you leave the canvas where you started the rotation.

chair_leaves_canvas.mp4

[TiagoCavalcante]

@MainzerKaiser I couldn't reproduce this (but still didn't test in an Android though).

Maybe you can share the whole code, or try to reduce the code where the bug happens?

Also, does this bug happen on web too?

[MainzerKaiser]

@TiagoCavalcante, the bug does not happen on web, it only appears on anroid. I could not test ios, though.
I created this repo with the chair, although if loaded via expo it should show 6 chairs, aligned vertically.
It should reflect the code in the first comment. And it is independent of the model loaded.

https://github.com/MainzerKaiser/testing_r3f_orbit_in_reactnative.git

[TiagoCavalcante]

The bug isn't happening on iOS, I'll test it on Android soon.

[MainzerKaiser]

How would you go about resetting the camera settings on button click?

[TiagoCavalcante]

If you are using an "external" camera like in the second example from README.md, you can set its target with camera.lookAt(new Vector3(x, y, z)) and its position (which is only useful if pan is enabled) with camera.position.set(x, y, z).

If you are using the default Canvas camera you can obtain the camera object with useThree (similarly to what is made below) and you can update it like above.

If you are using frameloop="demand" on the Canvas you may need to call Fiber's invalidate:

function ComponentWhichMustBeInsideTheCanvas() {
  const invalidate = useThree((state) => state.invalidate)

  useEffect(() => {
    // Click stuff
    invalidate()
  }, [dependencies])

  return null as unknown as JSX.Element // This must be a JSX element
}

[MainzerKaiser]

Unfortunately I could not make it work. I cannot use the external camera as it screws up the rest of my code (placement of all objects). How would i go about resetting each of my canvas cameras, if they are introduced by the mapping of controlsAndEvents.
I try to use a useEffect triggered by the button click and reloading boolean starting the resetCameraTarget() function, but my code does not reset the individual cameras in the nCubes=6 canvases.
I get the error: OrbitControls instance not available.

  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    const orbitControlsRef = useRef<typeof OrbitControls>();
    const canvasRef = useRef<any>(null);

    return { OrbitControls, events, orbitControlsRef, canvasRef, index };
  });

  interface ModelCallProps {
    index: number;
    canvasRef: React.RefObject<any>; // Adjust the type as needed
    reloading: boolean;
  }


  function ModelCall({ index, canvasRef, reloading }: ModelCallProps) {
    const { camera } = useThree();
  
    const resetCameraTarget = () => {
      if (camera && camera.userData && camera.userData.controls) {
        const orbitControls = camera.userData.controls;
        orbitControls.target.set(0, 0, 0); // Set the target position, adjust as needed
        orbitControls.update(); // Make sure to call update after modifying properties
      } else {
        console.error("OrbitControls instance not available.");
      }
    };
  
  
    // Reset camera target when reloading is true
    useEffect(() => {
      if (reloading) {
        resetCameraTarget();
        setQALoading(false); // Assuming setQALoading is a function to update qaloading state
      }
    }, [reloading]);
    return (
      <Model
        letters={letters[index]}
        cubeindex={index}
      />
    );
  }
  
  return (

      <SafeAreaView style={styles.container}>
        <View style={styles.modelsContainer}>
        {loading && <Loader />}
          <View style={isPortrait ? styles.columnContainer : styles.rowContainer}>
            {controlsAndEvents.map(({ OrbitControls, events, orbitControlsRef, canvasRef, index }) => {
                return (
                  <View
                  key={index}
                  style={isPortrait ? styles.modelColumn : styles.modelRow}
                  {...events}
                >
                    <Canvas 
                      key={index} 
                      ref={canvasRef}
                      
                    >
                      <OrbitControls
                        key={index} 
                        enableZoom={false}
                        enablePan={false}
                        dampingFactor={0.75}
                        enableRotate={true}
                        rotateSpeed={0.45}
                        onChange={(event) => handleCameraChange(event, index, letters)} 
                        {...OrbitControls}
                      />
                      <directionalLight position={[-1, 1, 1]} intensity={4} args={["white", 10]}  />
                      <directionalLight position={[1, -1, 1]} intensity={5} args={["blue", 10]}  />
                      <directionalLight position={[-1, -1, 1]} intensity={3} args={["orange", 10]}  />
                      <directionalLight position={[1, 1, -1]} intensity={5} args={["white", 10]}  />
                      <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                      <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                      <Suspense fallback={<Trigger setLoading={setLoading} />}>
                          <ModelCall index={index} canvasRef={canvasRef} reloading={qaloading}/>
                        {/* <Chair /> */}
                      </Suspense>
                    </Canvas>
                    {/* </View> 
                    </Pressable>*/}
                </View> 
                )
              })}
          </View>
         </View>
</SafeAreaView>
    )
  // );
};

[TiagoCavalcante]

@MainzerKaiser can you check if the error is still happening with 1.0.12?

[MainzerKaiser]

Hi Tiago,
i get this error when hitting the web version in expo:

ERROR in ./node_modules/r3f-native-orbitcontrols/lib/index.esm.js:3
Module not found: Can't resolve '@react-three/fiber/native'
Did you mean 'react-three-fiber-native.esm.js'?
BREAKING CHANGE: The request '@react-three/fiber/native' failed to resolve only because it was resolved as fully specified
(probably because the origin is strict EcmaScript Module, e. g. a module with javascript mimetype, a '*.mjs' file, or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
  1 | import React, { useMemo, useEffect } from 'react';
  2 | import { Vector3, Vector2, Spherical, Quaternion } from 'three';
> 3 | import { invalidate, useThree, useFrame } from '@react-three/fiber/native';

This is my package.json with comparable packages as in your package.json. Do you see what the problem is?

{
  "name": "cubewordle",
  "version": "1.0.0",
  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web"
  },
  "dependencies": {
    "@expo/webpack-config": "~19.0.1",
    "@mediapipe/tasks-vision": "^0.10.8",
    "@react-three/drei": "^9.89.0",
    "@react-three/fiber": "^8.15.11",
    "base64-arraybuffer": "^1.0.2",
    "expo": "^50.0.14",
    "expo-crypto": "^12.8.1",
    "expo-gl": "~13.6.0",
    "expo-status-bar": "~1.11.1",
    "jotai": "^2.6.1",
    "nanoid": "^5.0.4",
    "r3f-native-orbitcontrols": "^1.0.12",
    "react": "18.2.0",
    "react-native": "0.73.6",
    "react-native-3d-model-view": "^1.2.0",
    "react-native-gesture-handler": "~2.14.0",
    "react-native-safe-area-context": "4.8.2",
    "react-native-screens": "^3.29.0",
    "react-native-web": "~0.19.6",
    "react-promise-suspense": "^0.3.4",
    "react-use-refs": "^1.0.1",
    "rn-fetch-blob": "^0.12.0",
    "three": "^0.159.0",
    "three-bmfont-text": "^2.3.0",
    "three-stdlib": "^2.29.4",
    "xlsx": "^0.18.5",
    "zustand": "^4.4.7"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@types/expo": "^33.0.1",
    "@types/expo__vector-icons": "^10.0.0",
    "@types/react-dom": "~18.0.10",
    "@types/uuid": "^9.0.7",
    "typescript": "^5.3.3",
    "vite": "^5.0.10",
    "@types/react": "^18.2.56",
    "@types/react-native": "^0.72.8",
    "@types/three": "^0.161.2"
  },
  "lint-staged": {
    "*.{ts,tsx,md}": "prettier --write --no-semi"
  },
  "private": true
}

and the tsconfig.json:

{
  "compilerOptions": {
    "target": "es2020",
    "module": "commonjs",
    "jsx": "react-native",
    "strict": true,
    "esModuleInterop": true
  },
  "include": [
    "src/**/*.ts",
    "src/**/*.tsx",
    "App.tsx"
  ],
  "exclude": [
    "node_modules"
  ],
  "extends": "expo/tsconfig.base"
}

[MainzerKaiser]

Hi Tiago,
when starting the web version, i still get the error above, that ./node_modules/r3f-native-orbitcontrols/lib/index.esm.js:3 cannot resolve '@react-three/fiber/native'. Don't know why that is. Could it be, that the types are not working in the new version?

But i could test the android version with your new 1.0.12.. And my original second issue:

when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

still persists, unfortunately.

Best regards,
MK

[markhuang19994]

@MainzerKaiser

when keeping the finger on the display of my android (i run via expo) and leaving the current canvas the target camera position resets somehow to a weird position that i can make visible within the handleCameraChange function via console.logs. This does not happen when testing rotation outside of the canvas in a browser (firefox or chrome). Is that behavior on android intentional?

I encountered the same issue as you and have resolved it. Hopefully, the same approach works for you.

I calculate the Euclidean distance between two positions using a formula. If the distance is greater than 2, I assume the position update is incorrect.

const cameraRef = useRef<Camera>();
const lastCameraPositionRef = useRef<Vector3>();

const handleOrbitPositionUpdate = (position?: Vector3) => {
  if (!position) return;

  const lastCameraPosition = lastCameraPositionRef.current;
  if (lastCameraPosition) {
    const { x, y, z } = position;
    const [x0, y0, z0] = lastCameraPosition;

    const camera = cameraRef.current;
    if (camera) {
      // if the distance exceeds 2, assume the new position is incorrect.
      const distance = Math.sqrt((x0 - x) ** 2 + (y0 - y) ** 2 + (z0 - z) ** 2);
      if (distance >= 2) {
        camera.position.set(x0, y0, z0);
        invalidate();
        return;
      }
    }

    if(lastCameraPositionRef.current) {
      lastCameraPositionRef.current = position;    
    }
  }
};


<OrbitControls
  onChange={event =>
    handleOrbitPositionUpdate(event.target.camera?.position)
  }
/>

[MainzerKaiser]

Thanks for that idea. I adopted it to my problem and in principle it works fine. What does your invalidate() function do?
The important part of my code is now:

  const handleCameraChange = useCallback((event: any, index: number, letters:string[][], lastCameraPositionRef:any) => {
    const { x, y, z } = event.target.camera.position;
    const { x: x0, y: y0, z: z0 } = lastCameraPositionRef.current;

    const distance = Math.sqrt((x0 - x) ** 2 + (y0 - y) ** 2 + (z0 - z) ** 2);
    // console.log(distance, y)
    if (distance >= 2) {
      event.target.camera.position.set(x0,y0,z0);
      return;
    } else {
      lastCameraPositionRef.current = new THREE.Vector3(x, y, z);
  [...]
  
  
  const controlsAndEvents = Array.from({ length: nCubes }, (_, index) => {
    const [OrbitControls, events] = useControls();
    const orbitControlsRef = useRef<typeof OrbitControls>();
    const canvasRef = useRef<any>(null);
    const lastCameraPositionRef = useRef<THREE.Vector3>(new THREE.Vector3(0,0,5));

    return { OrbitControls, events, orbitControlsRef, canvasRef, index, lastCameraPositionRef};
  });
  
  [...]
{controlsAndEvents.map(({ OrbitControls, events, orbitControlsRef, canvasRef, index, lastCameraPositionRef }) => {
             return (
                  <View
                  key={index*3}
                  style={isPortrait ? styles.modelColumn : styles.modelRow}
                  {...events}
                >
                    <Canvas 
                      key={index*5} 
                      ref={canvasRef}
                      
                    >
                      <OrbitControls
                        key={index} 
                        enableZoom={false}
                        enablePan={false}
                        dampingFactor={0.75}
                        enableRotate={true}
                        rotateSpeed={0.45}
                        onChange={(event) => handleCameraChange(event, index, letters, lastCameraPositionRef)} 
                        {...OrbitControls}
                      />
                      <directionalLight position={[-1, 1, 1]} intensity={4} args={["white", 10]}  />
                      <directionalLight position={[1, -1, 1]} intensity={5} args={["blue", 10]}  />
                      <directionalLight position={[-1, -1, 1]} intensity={3} args={["orange", 10]}  />
                      <directionalLight position={[1, 1, -1]} intensity={5} args={["white", 10]}  />
                      <directionalLight position={[1, -1, -1]} intensity={5} args={["yellow", 10]}  />
                      <directionalLight position={[-1, -1, -1]} intensity={7} args={["white", 10]}  />
                      <Suspense fallback={<Trigger setLoading={setLoading} />}>
                          <ModelCall index={index} canvasRef={canvasRef} reloading={qaloading}/>
                      </Suspense>
                    </Canvas>

                </View> 
                )
              })}

  
  

Setting the event.target.camera.position.set(x0,y0,z0); to the lastCameraPosition in case the distance>2 is visibile as a small stutter.
But apart from that it works fine..

I'd rather have an option to invalidate every touch event outside of the canvas.

WhatsApp.Video.2024-07-14.um.12.07.27_96d1e932.mp4

[markhuang19994]

Thanks for that idea. I adopted it to my problem and in principle it works fine. What does your invalidate() function do?

I use the invalidate function to quickly update the corrected position on the canvas. This is especially useful when your canvas frameLoop is set to demand (On-demand rendering).

[ashkalor]

Hey @MainzerKaiser and @markhuang19994

Can you confirm if #47 fixes the problem you were facing, If this your only requirement then this could fix #33 as well.
Let me know.

[MainzerKaiser]

Thx ahkalor!
That sounds like great news. It was the first time for me to try to install a committed pull request code via gh and then link it to my project. I could not make it, even after trying for 1hr. I'll have to come back to you with testing this after Tiago merged it into the main branch.
Issue #33 is not really present anymore, I don't have a use case to test it.
However, thank you for improving this package!

[ashkalor]

Hey @MainzerKaiser

All you have to do is git clone my forked repo and change it to the map controls branch inside your project directory.
Then change your imports to this folder instead of node-modules. That should get the library working locally.
If you want to use map controls pass CONTROLMODES.MAP to useControls hook as a parameter.
Let me know if you face any issues or need any additional help.

[MainzerKaiser]

Ok, ill give it a try when I have time again. I read your code changes a bit and saw warnings about the camera being used. Does it work with my appraoch from above, where i only use the camera of the canvas or do i have to explicetly set a perpective camera like in the example code by Tiago?

function Canvases() {
  // You also can use the same OrbitControls for multiple canvases
  // creating an effect like the game
  // The Medium (https://store.steampowered.com/app/1293160)
  const [OrbitControls, events] = useControls()

  // In this case the same camera must be used in all the canvases
  const camera = new PerspectiveCamera()
  // You need to configure the camera too. Follow three.js' documentation.
  // Default configuration:
  //   camera.position.set(10, 10, 10)
  //   camera.lookAt(0, 0, 0)

  return (
    <View {...events}>
      <Canvas camera={camera}>
        <OrbitControls />
      </Canvas>
      <Canvas camera={camera}>
        <OrbitControls />
      </Canvas>
    </View>
  )
}

[ashkalor]

Hey @MainzerKaiser

That's only if you're using multiple canvases. There is no configuration change. Try and use it just like before and see if you're facing any issues.