diff --git a/README.md b/README.md
index f1b3abde1..f4c5b228e 100644
--- a/README.md
+++ b/README.md
@@ -110,6 +110,7 @@ Conference on Computer Vision and Pattern Recognition (CVPR) 2024
- ["Hello World": Generate your first Infinigen-Nature scene](docs/HelloWorld.md)
- ["Hello Room": Generate your first Infinigen-Indoors scene](docs/HelloRoom.md)
- [Configuring Infinigen](docs/ConfiguringInfinigen.md)
+- [Configuring Cameras](docs/ConfiguringCameras.md)
- [Downloading pre-generated data](docs/PreGeneratedData.md)
- [Generating individual assets](docs/GeneratingIndividualAssets.md)
- [Exporting to external fileformats (OBJ, OpenUSD, etc)](docs/ExportingToExternalFileFormats.md)
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index e03659896..a7c9b7384 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -129,4 +129,11 @@ v1.9.1
- Fix gin configs not correctly passed to slurm jobs in generate_individual_assets
- Fix integration test image titles
- Fix integration test asset image alignment
-- Make multistory houses disabled by default
\ No newline at end of file
+- Make multistory houses disabled by default
+
+v1.10.0
+- Add Configuring Cameras documentation
+- Add config for multiview cameras surrounding a point of interest
+- Add MaterialSegmentation output pass
+- Add passthrough mode to direct manage_jobs stdout directly to terminal
+- Add "copyfile:destination" upload mode
\ No newline at end of file
diff --git a/docs/ConfiguringCameras.md b/docs/ConfiguringCameras.md
new file mode 100644
index 000000000..7a353f786
--- /dev/null
+++ b/docs/ConfiguringCameras.md
@@ -0,0 +1,102 @@
+# Configuring Cameras
+
+This document gives examples of how to configure cameras in Infinigen for various computer vision tasks.
+
+### Example Commands
+
+##### Stereo Matching
+
+Generate many nature, scenes each with 1 stereo camera:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/stereo_nature --num_scenes 30 \
+--pipeline_config stereo.gin local_256GB.gin cuda_terrain.gin blender_gt.gin --configs high_quality_terrain
+```
+
+Generate many indoor rooms, each with 20 stereo cameras:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/stereo_indoors --num_scenes 30 \
+--pipeline_configs local_256GB.gin stereo.gin blender_gt.gin indoor_background_configs.gin --configs singleroom \
+--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' \
+--overrides camera.spawn_camera_rigs.n_camera_rigs=20 compute_base_views.min_candidates_ratio=2 compose_indoors.terrain_enabled=False compose_indoors.restrict_single_supported_roomtype=True
+```
+
+We recommend 20+ cameras per indoor room since room generation is not view-dependent and can be rendered from many angles. This helps overall GPU utilization since many frames are rendered per scene generated. In nature scenes, the current camera code would place cameras very far apart, meaning visible content does not overlap and there is minimal benefit to simply increasing `n_camera_rigs` in nature scenes without also customizing their arrangement. Thus, if you wish to extract more stereo frames per nature scene, we recommend instead rendering a low fps video using the "Random Walk Videos" commands below.
+
+##### Random Walk Videos
+
+Nature video, slow & smooth random walk camera motion:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/video_smooth_nature --num_scenes 30 \
+--pipeline_config monocular_video.gin local_256GB.gin cuda_terrain.gin blender_gt.gin --configs high_quality_terrain \
+--pipeline_overrides iterate_scene_tasks.cam_block_size=24
+```
+
+Nature video, fast & noisy random walk camera motion:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/video_smooth_nature --num_scenes 30 \
+--pipeline_config monocular_video.gin local_256GB.gin cuda_terrain.gin blender_gt.gin --configs high_quality_terrain noisy_video \
+--pipeline_overrides iterate_scene_tasks.cam_block_size=24 --overrides configure_render_cycles.adaptive_threshold=0.05
+```
+
+Indoor video, slow moving camera motion:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/video_slow_indoor --num_scenes 30 \
+--pipeline_configs local_256GB.gin monocular_video.gin blender_gt.gin indoor_background_configs.gin --configs singleroom \
+--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' \
+--overrides compose_indoors.terrain_enabled=False compose_indoors.restrict_single_supported_roomtype=True AnimPolicyRandomWalkLookaround.speed=0.5 AnimPolicyRandomWalkLookaround.step_range=0.5 compose_indoors.animate_cameras_enabled=True
+```
+
+:warning: Random walk camera generation is very unlikely to find paths between indoor rooms, and therefore will fail to generate long or fast moving videos for indoor scenes. We will followup soon with a pathfinding-based camera trajectory generator to handle these cases.
+
+##### Multi-view Camera Arrangement (for Multiview Stereo, NeRF, etc.)
+
+Many tasks require cameras placed in a roughly circular arrangement. Below with some noise added to their angle, roll, pitch, and yaw with respect to the object.
+
+
+ 
+ 
+ 
+ 
+
+
+Generate a quick test scene (indoor room with no furniture etc) with 5 multiview cameras:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/mvs_test --num_scenes 1 --configs multiview_stereo.gin fast_solve.gin no_objects.gin --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin cuda_terrain.gin indoor_background_configs.gin --overrides camera.spawn_camera_rigs.n_camera_rigs=5 compose_nature.animate_cameras_enabled=False compose_indoors.restrict_single_supported_roomtype=True --pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' iterate_scene_tasks.n_camera_rigs=5
+```
+
+Generate a dataset of indoor rooms with 30 multiview cameras:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/mvs_indoors --num_scenes 30 --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin --configs singleroom.gin multiview_stereo.gin --pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' iterate_scene_tasks.n_camera_rigs=30 --overrides compose_indoors.restrict_single_supported_roomtype=True camera.spawn_camera_rigs.n_camera_rigs=30
+```
+
+Generate a dataset of nature scenes with 30 multiview cameras:
+```bash
+python -m infinigen.datagen.manage_jobs --output_folder outputs/mvs_nature --num_scenes 30 --configs multiview_stereo.gin --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin cuda_terrain.gin --overrides camera.spawn_camera_rigs.n_camera_rigs=30 compose_nature.animate_cameras_enabled=False --pipeline_overrides iterate_scene_tasks.n_camera_rigs=30
+```
+
+##### Custom camera arrangement
+
+Camera poses can be easily manipulated using the Blender API to create any camera arrangement you wish
+
+For example, you could replace our `pose_cameras` step in `generature_nature.py` or `generate_indoors.py` with code as follows:
+
+```python
+for i, rig in enumerate(camera_rigs):
+ rig.location = (i, 0, np.random.uniform(0, 10))
+ rig.rotation_euler = np.deg2rad(np.array([90, 0, 180 * i / len(camera_rigs)]))
+```
+
+If you wish to animate the camera rigs to move over the course of a video, you would use code similar to the following:
+
+```python
+
+for i, rig in enumerate(camera_rigs):
+
+ for t in range(bpy.context.scene.frame_start, bpy.context.scene.frame_end + 1):
+
+ rig.location = (t, i, 0)
+ rig.keyframe_insert(data_path="location", frame=t)
+
+ rig.rotation_euler = np.deg2rad(np.array((90, 0, np.random.uniform(-10, 10))))
+ rig.keyframe_insert(data_path="rotation_euler", frame=t)
+```
\ No newline at end of file
diff --git a/docs/ConfiguringInfinigen.md b/docs/ConfiguringInfinigen.md
index 4636dffea..6c46618f9 100644
--- a/docs/ConfiguringInfinigen.md
+++ b/docs/ConfiguringInfinigen.md
@@ -101,12 +101,12 @@ If you have more than one GPU and are using a `local_*.gin` compute config, each
### Rendering Video, Stereo and other data formats
-Generating a video, stereo or other dataset typically requires more render jobs, so we must instruct `manage_jobs.py` to run those jobs. `datagen/configs/data_schema/` provides many options for you to use in your `--pipeline_configs`, including `monocular_video.gin` and `stereo.gin`.
These configs are typically mutually exclusive, and you must include at least one
+Generating a video, stereo or other dataset typically requires more render jobs, so we must instruct `manage_jobs.py` to run those jobs. `datagen/configs/data_schema/` provides many options for you to use in your `--pipeline_configs`, including `monocular_video.gin`, `stereo.gin` and `multiview_stereo.gin`.
These configs are typically mutually exclusive, and you must include at least one
To create longer videos, modify `iterate_scene_tasks.frame_range` in `monocular_video.gin` (note: we use 24fps video by default). `iterate_scene_tasks.view_block_size` controls how many frames will be grouped into each `fine_terrain` and render / ground-truth task.
-If you need more than two cameras, or want to customize their placement, see `infinigen_examples/configs_nature/base.gin`'s `camera.spawn_camera_rigs.camera_rig_config` for advice on existing options, or write your own code to instantiate a custom camera setup.
+If you need more than two cameras, or want to customize their placement, see `infinigen_examples/configs_nature/base.gin`'s `camera.spawn_camera_rigs.camera_rig_config` for advice on existing options, or write your own code to instantiate a custom camera setup. For multiview stereo data, you may include `multiview_stereo.gin` in `--configs`, which creates 30 cameras by default.
### Config Overrides to Customize Scene Content
diff --git a/docs/images/multiview_stereo/mvs_indoors.png b/docs/images/multiview_stereo/mvs_indoors.png
new file mode 100644
index 000000000..2c670a452
Binary files /dev/null and b/docs/images/multiview_stereo/mvs_indoors.png differ
diff --git a/docs/images/multiview_stereo/mvs_indoors_2.png b/docs/images/multiview_stereo/mvs_indoors_2.png
new file mode 100644
index 000000000..5ff678b0c
Binary files /dev/null and b/docs/images/multiview_stereo/mvs_indoors_2.png differ
diff --git a/docs/images/multiview_stereo/mvs_nature.png b/docs/images/multiview_stereo/mvs_nature.png
new file mode 100644
index 000000000..71f3b4aac
Binary files /dev/null and b/docs/images/multiview_stereo/mvs_nature.png differ
diff --git a/docs/images/multiview_stereo/mvs_ocean.png b/docs/images/multiview_stereo/mvs_ocean.png
new file mode 100644
index 000000000..e2186bd73
Binary files /dev/null and b/docs/images/multiview_stereo/mvs_ocean.png differ
diff --git a/infinigen/__init__.py b/infinigen/__init__.py
index 9f7272bb5..944db1c65 100644
--- a/infinigen/__init__.py
+++ b/infinigen/__init__.py
@@ -6,7 +6,7 @@
import logging
from pathlib import Path
-__version__ = "1.9.2"
+__version__ = "1.10.0"
def repo_root():
diff --git a/infinigen/assets/objects/rocks/boulder.py b/infinigen/assets/objects/rocks/boulder.py
index 02bbd2658..d67233e50 100644
--- a/infinigen/assets/objects/rocks/boulder.py
+++ b/infinigen/assets/objects/rocks/boulder.py
@@ -35,7 +35,7 @@ class BoulderFactory(AssetFactory):
def __init__(
self,
factory_seed,
- meshing_camera=None,
+ meshing_cameras=None,
adapt_mesh_method="remesh",
cam_meshing_max_dist=1e7,
coarse=False,
@@ -43,7 +43,7 @@ def __init__(
):
super(BoulderFactory, self).__init__(factory_seed, coarse)
- self.camera = meshing_camera
+ self.cameras = meshing_cameras
self.cam_meshing_max_dist = cam_meshing_max_dist
self.adapt_mesh_method = adapt_mesh_method
@@ -166,10 +166,10 @@ def geo_extrusion(nw: NodeWrangler, extrude_scale=1):
nw.new_node(Nodes.GroupOutput, input_kwargs={"Geometry": geometry})
def create_asset(self, i, placeholder, face_size=0.01, distance=0, **params):
- if self.camera is not None and distance < self.cam_meshing_max_dist:
+ if self.cameras is not None and distance < self.cam_meshing_max_dist:
assert self.adapt_mesh_method != "remesh"
skin_obj, outofview, vert_dists, _ = split_inview(
- placeholder, cam=self.camera, vis_margin=0.15
+ placeholder, cameras=self.cameras, vis_margin=0.15
)
butil.parent_to(outofview, skin_obj, no_inverse=True, no_transform=True)
face_size = detail.target_face_size(vert_dists.min())
diff --git a/infinigen/assets/objects/trees/generate.py b/infinigen/assets/objects/trees/generate.py
index 32ca44f7a..0019ac90d 100644
--- a/infinigen/assets/objects/trees/generate.py
+++ b/infinigen/assets/objects/trees/generate.py
@@ -58,7 +58,7 @@ def __init__(
child_col,
trunk_surface,
realize=False,
- meshing_camera=None,
+ meshing_cameras=None,
cam_meshing_max_dist=1e7,
coarse_mesh_placeholder=False,
adapt_mesh_method="remesh",
@@ -73,7 +73,7 @@ def __init__(
self.trunk_surface = trunk_surface
self.realize = realize
- self.camera = meshing_camera
+ self.cameras = meshing_cameras
self.cam_meshing_max_dist = cam_meshing_max_dist
self.adapt_mesh_method = adapt_mesh_method
self.decimate_placeholder_levels = decimate_placeholder_levels
@@ -168,12 +168,12 @@ def create_asset(
),
)
- if self.camera is not None and distance < self.cam_meshing_max_dist:
+ if self.cameras is not None and distance < self.cam_meshing_max_dist:
assert self.adapt_mesh_method != "remesh"
skin_obj_cleanup = skin_obj
skin_obj, outofview, vert_dists, _ = split_inview(
- skin_obj, cam=self.camera, vis_margin=0.15
+ skin_obj, cameras=self.cameras, vis_margin=0.15
)
butil.parent_to(outofview, skin_obj, no_inverse=True, no_transform=True)
diff --git a/infinigen/core/execute_tasks.py b/infinigen/core/execute_tasks.py
index e741d017e..1436b5b13 100644
--- a/infinigen/core/execute_tasks.py
+++ b/infinigen/core/execute_tasks.py
@@ -47,7 +47,7 @@ def get_scene_tag(name):
def render(
scene_seed,
output_folder,
- camera_id,
+ camera,
render_image_func=render_image,
resample_idx=None,
hide_water=False,
@@ -59,7 +59,7 @@ def render(
if resample_idx is not None and resample_idx != 0:
resample_scene(int_hash((scene_seed, resample_idx)))
with Timer("Render Frames"):
- render_image_func(frames_folder=Path(output_folder), camera_id=camera_id)
+ render_image_func(frames_folder=Path(output_folder), camera=camera)
def is_static(obj):
@@ -87,8 +87,9 @@ def is_static(obj):
@gin.configurable
def save_meshes(
- scene_seed,
- output_folder,
+ scene_seed: int,
+ output_folder: Path,
+ cameras: list[bpy.types.Object],
frame_range,
resample_idx=False,
point_trajectory_src_frame=1,
@@ -108,8 +109,9 @@ def save_meshes(
for obj in bpy.data.objects:
obj.hide_viewport = not (not obj.hide_render and is_static(obj))
frame_idx = point_trajectory_src_frame
- frame_info_folder = Path(output_folder) / f"frame_{frame_idx:04d}"
+ frame_info_folder = output_folder / f"frame_{frame_idx:04d}"
frame_info_folder.mkdir(parents=True, exist_ok=True)
+
logger.info("Working on static objects")
exporting.save_obj_and_instances(
frame_info_folder / "static_mesh",
@@ -128,9 +130,9 @@ def save_meshes(
):
bpy.context.scene.frame_set(frame_idx)
bpy.context.view_layer.update()
- frame_info_folder = Path(output_folder) / f"frame_{frame_idx:04d}"
+ frame_info_folder = output_folder / f"frame_{frame_idx:04d}"
frame_info_folder.mkdir(parents=True, exist_ok=True)
- logger.info(f"Working on frame {frame_idx}")
+ logger.info(f"save_meshes processing {frame_idx=}")
exporting.save_obj_and_instances(
frame_info_folder / "mesh",
@@ -138,7 +140,7 @@ def save_meshes(
current_frame_mesh_id_mapping,
)
cam_util.save_camera_parameters(
- camera_ids=cam_util.get_cameras_ids(),
+ camera_ids=cameras,
output_folder=frame_info_folder / "cameras",
frame=frame_idx,
)
@@ -246,12 +248,15 @@ def execute_tasks(
with open(outpath / "info.pickle", "wb") as f:
pickle.dump(info, f, protocol=pickle.HIGHEST_PROTOCOL)
- cam_util.set_active_camera(*camera_id)
+ camera_rigs = cam_util.get_camera_rigs()
+ camrig_id, subcam_id = camera_id
+ active_camera = camera_rigs[camrig_id].children[subcam_id]
+ cam_util.set_active_camera(active_camera)
group_collections()
if Task.Populate in task and populate_scene_func is not None:
- populate_scene_func(output_folder, scene_seed)
+ populate_scene_func(output_folder, scene_seed, camera_rigs)
need_terrain_processing = "atmosphere" in bpy.data.objects
@@ -267,10 +272,9 @@ def execute_tasks(
whole_bbox=info["whole_bbox"],
)
- cameras = [cam_util.get_camera(i, j) for i, j in cam_util.get_cameras_ids()]
terrain.fine_terrain(
output_folder,
- cameras=cameras,
+ cameras=[c for rig in camera_rigs for c in rig.children],
optimize_terrain_diskusage=optimize_terrain_diskusage,
)
@@ -326,7 +330,7 @@ def execute_tasks(
render(
scene_seed,
output_folder=output_folder,
- camera_id=camera_id,
+ camera=active_camera,
resample_idx=resample_idx,
)
diff --git a/infinigen/core/placement/camera.py b/infinigen/core/placement/camera.py
index 23157efbe..b4856e7ff 100644
--- a/infinigen/core/placement/camera.py
+++ b/infinigen/core/placement/camera.py
@@ -40,8 +40,6 @@
logger = logging.getLogger(__name__)
-CAMERA_RIGS_DIRNAME = "CameraRigs"
-
@gin.configurable
def get_sensor_coords(cam, H, W, sparse=False):
@@ -119,55 +117,54 @@ def spawn_camera():
return cam
-def camera_name(rig_id, cam_id):
- return f"{CAMERA_RIGS_DIRNAME}/{rig_id}/{cam_id}"
+def cam_name(cam_rig, subcam):
+ return f"camera_{cam_rig}_{subcam}"
+
+
+def get_id(camera: bpy.types.Object):
+ _, rig, subcam = camera.name.split("_")
+ return int(rig), int(subcam)
@gin.configurable
def spawn_camera_rigs(
camera_rig_config,
n_camera_rigs,
-):
+) -> list[bpy.types.Object]:
+ rigs_col = butil.get_collection("camera_rigs")
+ cams_col = butil.get_collection("cameras")
+
def spawn_rig(i):
- rig_parent = butil.spawn_empty(f"{CAMERA_RIGS_DIRNAME}/{i}")
+ rig_parent = butil.spawn_empty(f"camrig.{i}")
+ butil.put_in_collection(rig_parent, rigs_col)
+
for j, config in enumerate(camera_rig_config):
cam = spawn_camera()
- cam.name = camera_name(i, j)
+ cam.name = cam_name(i, j)
cam.parent = rig_parent
-
cam.location = config["loc"]
cam.rotation_euler = config["rot_euler"]
- return rig_parent
+ butil.put_in_collection(cam, cams_col)
- camera_rigs = [spawn_rig(i) for i in range(n_camera_rigs)]
- butil.group_in_collection(camera_rigs, CAMERA_RIGS_DIRNAME)
+ return rig_parent
- return camera_rigs
+ return [spawn_rig(i) for i in range(n_camera_rigs)]
-def get_cameras_ids() -> list[tuple]:
- res = []
- col = bpy.data.collections[CAMERA_RIGS_DIRNAME]
- rigs = [o for o in col.objects if o.name.count("/") == 1]
- for i, root in enumerate(rigs):
- for j, subcam in enumerate(root.children):
- assert subcam.name == camera_name(i, j)
- res.append((i, j))
+def get_camera_rigs() -> list[bpy.types.Object]:
+ if "camera_rigs" not in bpy.data.collections:
+ raise ValueError("No camera rigs found")
- return res
+ result = list(bpy.data.collections["camera_rigs"].objects)
+ for i, rig in enumerate(result):
+ for j, child in enumerate(rig.children):
+ expected = cam_name(i, j)
+ if child.name != expected:
+ raise ValueError(f"child {i=} {j} was {child.name=}, {expected=}")
-def get_camera(rig_id, subcam_id, checkonly=False):
- col = bpy.data.collections[CAMERA_RIGS_DIRNAME]
- name = camera_name(rig_id, subcam_id)
- if name in col.objects.keys():
- return col.objects[name]
- if checkonly:
- return None
- raise ValueError(
- f"Could not get_camera({rig_id=}, {subcam_id=}). {list(col.objects.keys())=}"
- )
+ return result
@node_utils.to_nodegroup(
@@ -181,8 +178,7 @@ def nodegroup_active_cam_info(nw: NodeWrangler):
)
-def set_active_camera(rig_id, subcam_id):
- camera = get_camera(rig_id, subcam_id)
+def set_active_camera(camera: bpy.types.Object):
bpy.context.scene.camera = camera
ng = (
@@ -193,34 +189,6 @@ def set_active_camera(rig_id, subcam_id):
return bpy.context.scene.camera
-def positive_gaussian(mean, std):
- while True:
- val = np.random.normal(mean, std)
- if val > 0:
- return val
-
-
-def set_camera(
- camera,
- location,
- rotation,
- focus_dist,
- frame,
-):
- camera.location = location
- camera.rotation_euler = rotation
- if focus_dist is not None:
- camera.data.dof.focus_distance = (
- focus_dist # this should come before view_layer.update()
- )
- bpy.context.view_layer.update()
-
- camera.keyframe_insert(data_path="location", frame=frame)
- camera.keyframe_insert(data_path="rotation_euler", frame=frame)
- if focus_dist is not None:
- camera.data.dof.keyframe_insert(data_path="focus_distance", frame=frame)
-
-
def terrain_camera_query(
cam, scene_bvh, terrain_tags_queries, vertexwise_min_dist, min_dist=0
):
@@ -264,6 +232,9 @@ def apply(self, cam):
def camera_pose_proposal(
scene_bvh,
location_sample: typing.Callable | tuple,
+ center_coordinate=None,
+ radius=None,
+ bbox=None,
altitude=("uniform", 1.5, 2.5),
roll=0,
yaw=("uniform", -180, 180),
@@ -279,6 +250,32 @@ def location_sample():
if override_loc is not None:
loc = Vector(random_general(override_loc))
+ elif center_coordinate:
+ while True:
+ # Define the radius of the circle
+ random_angle = np.random.uniform(2 * np.math.pi)
+ xoff = np.random.uniform(-radius / 10, radius / 10)
+ yoff = np.random.uniform(-radius / 10, radius / 10)
+ zoff = random_general(altitude)
+ loc = Vector([0, 0, 0])
+ loc[0] = center_coordinate[0] + radius * np.math.cos(random_angle) + xoff
+ loc[1] = center_coordinate[1] + radius * np.math.sin(random_angle) + yoff
+ loc[2] = center_coordinate[2] + zoff
+ if bbox is not None:
+ out_of_bbox = False
+ for i in range(3):
+ if loc[i] < bbox[0][i] or loc[i] > bbox[1][i]:
+ out_of_bbox = True
+ break
+ if out_of_bbox:
+ continue
+ hit, *_ = scene_bvh.ray_cast(
+ loc,
+ Vector(center_coordinate) - loc,
+ (Vector(center_coordinate) - loc).length,
+ )
+ if hit is None:
+ break
elif altitude is None:
loc = location_sample()
else:
@@ -291,7 +288,22 @@ def location_sample():
desired_alt = random_general(altitude)
loc[2] = loc[2] + desired_alt - curr_alt
- rot = np.deg2rad([random_general(pitch), random_general(roll), random_general(yaw)])
+ if center_coordinate:
+ direction = loc - Vector(center_coordinate)
+ direction = Vector(direction)
+ rotation_matrix = direction.to_track_quat("Z", "Y").to_matrix()
+ rotation_euler = rotation_matrix.to_euler("XYZ")
+ roll, pitch, yaw = rotation_euler
+ noise_range = np.deg2rad(5.0) # 5 degrees of noise in radians
+ # Add random noise to roll, pitch, and yaw
+ roll += np.random.uniform(-noise_range, noise_range)
+ pitch += np.random.uniform(-noise_range, noise_range)
+ yaw += np.random.uniform(-noise_range, noise_range)
+ rot = np.array([roll, pitch, yaw])
+ else:
+ rot = np.deg2rad(
+ [random_general(pitch), random_general(roll), random_general(yaw)]
+ )
focal_length = random_general(focal_length)
return CameraProposal(loc, rot, focal_length)
@@ -385,7 +397,7 @@ def __call__(self, camera_rig, frame_curr, retry_pct, bvh):
bbox = (camera_rig.location - margin, camera_rig.location + margin)
for _ in range(self.retries):
- res = camera_pose_proposal(bvh, bbox)
+ res = camera_pose_proposal(bvh, bbox) # !
if res is None:
continue
dist = np.linalg.norm(np.array(res.loc) - np.array(camera_rig.location))
@@ -408,6 +420,9 @@ def compute_base_views(
terrain,
scene_bvh,
location_sample: typing.Callable,
+ center_coordinate=None,
+ radius=None,
+ bbox=None,
placeholders_kd=None,
camera_selection_answers={},
vertexwise_min_dist=None,
@@ -418,11 +433,21 @@ def compute_base_views(
):
potential_views = []
n_min_candidates = int(min_candidates_ratio * n_views)
+
with tqdm(total=n_min_candidates, desc="Searching for camera viewpoints") as pbar:
for it in range(1, max_tries):
- props = camera_pose_proposal(
- scene_bvh=scene_bvh, location_sample=location_sample
- )
+ if center_coordinate:
+ props = camera_pose_proposal(
+ scene_bvh=scene_bvh,
+ location_sample=location_sample,
+ center_coordinate=center_coordinate,
+ radius=random_general(radius),
+ bbox=bbox,
+ )
+ else:
+ props = camera_pose_proposal(
+ scene_bvh=scene_bvh, location_sample=location_sample
+ )
if props is None:
logger.debug(
@@ -611,6 +636,10 @@ def configure_cameras(
scene_preprocessed: dict,
init_bounding_box: tuple[np.array, np.array] = None,
init_surfaces: list[bpy.types.Object] = None,
+ terrain_mesh=None,
+ nonroom_objs=None,
+ mvs_setting=False,
+ mvs_radius=("uniform", 12, 18),
):
bpy.context.view_layer.update()
dummy_camera = spawn_camera()
@@ -629,10 +658,43 @@ def location_sample():
else:
raise ValueError("Either init_bounding_box or init_surfaces must be provided")
+ if mvs_setting:
+ if terrain_mesh:
+ vertices = np.array([np.array(v.co) for v in terrain_mesh.data.vertices])
+ sdfs = scene_preprocessed["terrain"].compute_camera_space_sdf(vertices)
+ vertices = vertices[sdfs >= -1e-5]
+ center_coordinate = list(
+ vertices[np.random.choice(list(range(len(vertices))))]
+ )
+ elif nonroom_objs:
+
+ def contain_keywords(name, keywords):
+ for keyword in keywords:
+ if name == keyword or name.startswith(f"{keyword}."):
+ return True
+ return False
+
+ inside_objs = [
+ x
+ for x in nonroom_objs
+ if not contain_keywords(x.name, ["window", "door", "entrance"])
+ ]
+ assert inside_objs != []
+ obj = np.random.choice(inside_objs)
+ vertices = [v.co for v in obj.data.vertices]
+ center_coordinate = vertices[np.random.choice(list(range(len(vertices))))]
+ center_coordinate = obj.matrix_world @ center_coordinate
+ center_coordinate = list(np.array(center_coordinate))
+ else:
+ center_coordinate = None
+
base_views = compute_base_views(
dummy_camera,
n_views=len(cam_rigs),
location_sample=location_sample,
+ center_coordinate=center_coordinate,
+ radius=mvs_radius,
+ bbox=init_bounding_box,
**scene_preprocessed,
)
@@ -704,32 +766,37 @@ def animate_cameras(
@gin.configurable
-def save_camera_parameters(camera_ids, output_folder, frame, use_dof=False):
+def save_camera_parameters(
+ camera_obj: bpy.types.Object, output_folder, frame, use_dof=False
+):
output_folder = Path(output_folder)
output_folder.mkdir(exist_ok=True, parents=True)
+
if frame is not None:
bpy.context.scene.frame_set(frame)
- for camera_pair_id, camera_id in camera_ids:
- camera_obj = get_camera(camera_pair_id, camera_id)
- if use_dof is not None:
- camera_obj.data.dof.use_dof = use_dof
- # Saving camera parameters
- K = camera.get_calibration_matrix_K_from_blender(camera_obj.data)
- suffix = get_suffix(
- dict(cam_rig=camera_pair_id, resample=0, frame=frame, subcam=camera_id)
- )
- output_file = output_folder / f"camview{suffix}.npz"
- height_width = np.array(
- (
- bpy.context.scene.render.resolution_y,
- bpy.context.scene.render.resolution_x,
- )
+ camrig_id, subcam_id = get_id(camera_obj)
+
+ if use_dof is not None:
+ camera_obj.data.dof.use_dof = use_dof
+
+ # Saving camera parameters
+ K = camera.get_calibration_matrix_K_from_blender(camera_obj.data)
+ suffix = get_suffix(
+ dict(cam_rig=camrig_id, resample=0, frame=frame, subcam=subcam_id)
+ )
+ output_file = output_folder / f"camview{suffix}.npz"
+
+ height_width = np.array(
+ (
+ bpy.context.scene.render.resolution_y,
+ bpy.context.scene.render.resolution_x,
)
- T = np.asarray(camera_obj.matrix_world, dtype=np.float64) @ np.diag(
- (1.0, -1.0, -1.0, 1.0)
- ) # Y down Z forward (aka opencv)
- np.savez(output_file, K=np.asarray(K, dtype=np.float64), T=T, HW=height_width)
+ )
+ T = np.asarray(camera_obj.matrix_world, dtype=np.float64) @ np.diag(
+ (1.0, -1.0, -1.0, 1.0)
+ ) # Y down Z forward (aka opencv)
+ np.savez(output_file, K=np.asarray(K, dtype=np.float64), T=T, HW=height_width)
if __name__ == "__main__":
diff --git a/infinigen/core/placement/placement.py b/infinigen/core/placement/placement.py
index ed6718ba7..5e579177c 100644
--- a/infinigen/core/placement/placement.py
+++ b/infinigen/core/placement/placement.py
@@ -18,9 +18,8 @@
NodeWrangler,
geometry_node_group_empty_new,
)
-from infinigen.core.placement import detail
+from infinigen.core.placement import detail, split_in_view
from infinigen.core.util import blender as butil
-from infinigen.core.util import camera as camera_util
from .factory import AssetFactory
@@ -151,7 +150,7 @@ def parse_asset_name(name):
def populate_collection(
factory: AssetFactory,
- placeholder_col,
+ placeholder_col: bpy.types.Collection,
asset_col_target=None,
cameras=None,
dist_cull=None,
@@ -178,33 +177,18 @@ def populate_collection(
continue
if cameras is not None:
- populate = False
- dist_list = []
- vis_dist_list = []
- for i, camera in enumerate(cameras):
- points = get_placeholder_points(p)
- dists, vis_dists = camera_util.min_dists_from_cam_trajectory(
- points, camera
+ mask, min_dists, min_vis_dists = split_in_view.compute_inview_distances(
+ get_placeholder_points(p), cameras, verbose=verbose
+ )
+
+ dist = min_dists.min()
+ vis_dist = min_vis_dists.min()
+
+ if not mask.any():
+ logger.debug(
+ f"{p.name=} culled, not in view of any camera. {dist=} {vis_dist=}"
)
- dist, vis_dist = dists.min(), vis_dists.min()
- if dist_cull is not None and dist > dist_cull:
- logger.debug(
- f"{p.name=} temporarily culled in camera {i} due to {dist=:.2f} > {dist_cull=}"
- )
- continue
- if vis_cull is not None and vis_dist > vis_cull:
- logger.debug(
- f"{p.name=} temporarily culled in camera {i} due to {vis_dist=:.2f} > {vis_cull=}"
- )
- continue
- populate = True
- dist_list.append(dist)
- vis_dist_list.append(vis_dist)
- if not populate:
- p.hide_render = True
continue
- p["dist"] = min(dist_list)
- p["vis_dist"] = min(vis_dist_list)
else:
dist = detail.scatter_res_distance()
@@ -251,7 +235,12 @@ def populate_collection(
@gin.configurable
def populate_all(
- factory_class, camera, dist_cull=200, vis_cull=0, cache_system=None, **kwargs
+ factory_class: type,
+ cameras: list[bpy.types.Object],
+ dist_cull=200,
+ vis_cull=0,
+ cache_system=None,
+ **kwargs,
):
"""
Find all collections that may have been produced by factory_class, and update them
@@ -283,7 +272,7 @@ def populate_all(
factory_class(int(fac_seed), **kwargs),
col,
asset_target_col,
- camera,
+ camera=cameras,
dist_cull=dist_cull,
vis_cull=vis_cull,
cache_system=cache_system,
diff --git a/infinigen/core/placement/split_in_view.py b/infinigen/core/placement/split_in_view.py
index aaae50615..56790b849 100644
--- a/infinigen/core/placement/split_in_view.py
+++ b/infinigen/core/placement/split_in_view.py
@@ -8,6 +8,7 @@
import bpy
import numpy as np
+from mathutils import Matrix
from mathutils.bvhtree import BVHTree
from tqdm import trange
@@ -15,9 +16,18 @@
from infinigen.core.util import blender as butil
from infinigen.core.util import camera as cam_util
from infinigen.core.util.logging import Suppress
+from infinigen.core.util.math import dehomogenize, homogenize
+logger = logging.getLogger(__name__)
-def raycast_visiblity_mask(obj, cam, start=None, end=None, verbose=True):
+
+def raycast_visiblity_mask(
+ obj: bpy.types.Object,
+ cameras: list[bpy.types.Object],
+ start=None,
+ end=None,
+ verbose=True,
+):
bvh = BVHTree.FromObject(obj, bpy.context.evaluated_depsgraph_get())
if start is None:
@@ -30,19 +40,20 @@ def raycast_visiblity_mask(obj, cam, start=None, end=None, verbose=True):
for i in rangeiter(start, end + 1):
bpy.context.scene.frame_set(i)
invworld = obj.matrix_world.inverted()
- sensor_coords, pix_it = get_sensor_coords(cam)
- for x, y in pix_it:
- direction = (
- sensor_coords[y, x] - cam.matrix_world.translation
- ).normalized()
- origin = cam.matrix_world.translation
- _, _, index, dist = bvh.ray_cast(
- invworld @ origin, invworld.to_3x3() @ direction
- )
- if dist is None:
- continue
- for vi in obj.data.polygons[index].vertices:
- mask[vi] = True
+ for cam in cameras:
+ sensor_coords, pix_it = get_sensor_coords(cam)
+ for x, y in pix_it:
+ direction = (
+ sensor_coords[y, x] - cam.matrix_world.translation
+ ).normalized()
+ origin = cam.matrix_world.translation
+ _, _, index, dist = bvh.ray_cast(
+ invworld @ origin, invworld.to_3x3() @ direction
+ )
+ if dist is None:
+ continue
+ for vi in obj.data.polygons[index].vertices:
+ mask[vi] = True
return mask
@@ -74,42 +85,113 @@ def duplicate_mask(obj, mask, dilate=0, invert=False):
return butil.spawn_point_cloud("duplicate_mask", [], [])
+def compute_vis_dists(points: np.array, cam: bpy.types.Object):
+ projmat, K, RT = map(np.array, cam_util.get_3x4_P_matrix_from_blender(cam))
+ proj = points @ projmat.T
+ uv, d = dehomogenize(proj), proj[:, -1]
+
+ clamped_uv = np.clip(uv, [0, 0], butil.get_camera_res())
+ clamped_d = np.maximum(d, 0)
+
+ RT_4x4_inv = np.array(Matrix(RT).to_4x4().inverted())
+ clipped_pos = (
+ homogenize((homogenize(clamped_uv) * clamped_d[:, None]) @ np.linalg.inv(K).T)
+ @ RT_4x4_inv.T
+ )
+
+ vis_dist = np.linalg.norm(points[:, :-1] - clipped_pos[:, :-1], axis=-1)
+
+ return d, vis_dist
+
+
+def compute_inview_distances(
+ points: np.array,
+ cameras: list[bpy.types.Object],
+ dist_max,
+ vis_margin,
+ frame_start=None,
+ frame_end=None,
+ verbose=False,
+):
+ """
+ Compute the minimum distance of each point to any of the cameras in the scene.
+
+ Parameters:
+ - points: an array of 3D points, in world space
+ - cameras: a list of cameras in the scene
+ - dist_max: the maximum distance to consider a point "in view"
+ - vis_margin: how far outside the view frustum to consider a point "in view"
+
+ Returns:
+ - mask: boolean array of whether each point is within within vis_margin and dist_max of any frame of any camera
+ - min_dists: the distance of each point the closest camera
+ - min_vis_dists: the distance of each point to the nearest point in any camera's view frustum
+ """
+
+ assert len(points.shape) == 2 and points.shape[-1] == 3
+
+ if frame_start is None:
+ frame_start = bpy.context.scene.frame_start
+ if frame_end is None:
+ frame_end = bpy.context.scene.frame_end
+
+ points = homogenize(points)
+
+ mask = np.zeros(len(points), dtype=bool)
+ min_dists = np.full(len(points), 1e7)
+ min_vis_dists = np.full(len(points), 1e7)
+
+ rangeiter = trange if verbose else range
+
+ assert frame_start < frame_end + 1, (frame_start, frame_end)
+
+ for frame in rangeiter(frame_start, frame_end + 1):
+ bpy.context.scene.frame_set(frame)
+ for cam in cameras:
+ dists, vis_dists = compute_vis_dists(points, cam)
+ mask |= (dists < dist_max) & (vis_dists < vis_margin)
+ if mask.any():
+ min_vis_dists[mask] = np.minimum(vis_dists[mask], min_vis_dists[mask])
+ min_dists[mask] = np.minimum(dists[mask], min_dists[mask])
+
+ logger.debug(f"Computed dists for {frame=} {cam.name} {mask.mean()=:.2f}")
+
+ return mask, min_dists, min_vis_dists
+
+
def split_inview(
obj: bpy.types.Object,
- cam,
- vis_margin,
- raycast=False,
- dilate=0,
- dist_max=1e7,
+ cameras: list[bpy.types.Object],
+ dist_max: float = 1e7,
+ vis_margin: float = 0,
+ raycast: bool = False,
+ dilate: float = 0,
outofview=True,
verbose=False,
- print_areas=False,
hide_render=None,
suffix=None,
**kwargs,
):
assert obj.type == "MESH"
- assert cam.type == "CAMERA"
bpy.context.view_layer.update()
verts = np.zeros((len(obj.data.vertices), 3))
obj.data.vertices.foreach_get("co", verts.reshape(-1))
verts = butil.apply_matrix_world(obj, verts)
- dists, vis_dists = cam_util.min_dists_from_cam_trajectory(
- verts, cam, verbose=verbose, **kwargs
+ mask, dists, vis_dists = compute_inview_distances(
+ verts,
+ cameras,
+ dist_max=dist_max,
+ vis_margin=vis_margin,
+ verbose=verbose,
+ **kwargs,
)
- vis_mask = vis_dists < vis_margin
- dist_mask = dists < dist_max
- mask = vis_mask * dist_mask
-
- logging.debug(
- f"split_inview {vis_mask.mean()=:.2f} {dist_mask.mean()=:.2f} {mask.mean()=:.2f}"
- )
+ logger.debug(f"split_inview {suffix=} {dist_max=} {vis_margin=} {mask.mean()=:.2f}")
if raycast:
- mask *= raycast_visiblity_mask(obj, cam)
+ mask *= raycast_visiblity_mask(obj, cameras)
inview = duplicate_mask(obj, mask, dilate=dilate)
@@ -118,13 +200,6 @@ def split_inview(
else:
outview = butil.spawn_point_cloud("duplicate_mask", [], [])
- if print_areas:
- sa_in = butil.surface_area(inview)
- sa_out = butil.surface_area(outview)
- print(
- f"split {obj.name=} into inview area {sa_in:.2f} and outofview area {sa_out:.2f}"
- )
-
inview.name = obj.name + ".inview"
outview.name = obj.name + ".outofview"
diff --git a/infinigen/core/rendering/render.py b/infinigen/core/rendering/render.py
index 2adce25a6..75bfa9da9 100644
--- a/infinigen/core/rendering/render.py
+++ b/infinigen/core/rendering/render.py
@@ -90,6 +90,17 @@ def set_pass_indices():
return tree_output
+def set_material_pass_indices():
+ output_material_properties = {}
+ mat_index = 1
+ for mat in bpy.data.materials:
+ if mat.pass_index == 0:
+ mat.pass_index = mat_index
+ mat_index += 1
+ output_material_properties[mat.name] = {"pass_index": mat.pass_index}
+ return output_material_properties
+
+
# Can be pasted directly into the blender console
def make_clay():
clay_material = bpy.data.materials.new(name="clay")
@@ -145,14 +156,25 @@ def configure_compositor_output(
passes_to_save,
saving_ground_truth,
):
- file_output_node = nw.new_node(
+ file_output_node_png = nw.new_node(
+ Nodes.OutputFile,
+ attrs={
+ "base_path": str(frames_folder),
+ "format.file_format": "PNG",
+ "format.color_mode": "RGB",
+ },
+ )
+ file_output_node_exr = nw.new_node(
Nodes.OutputFile,
attrs={
"base_path": str(frames_folder),
- "format.file_format": "OPEN_EXR" if saving_ground_truth else "PNG",
+ "format.file_format": "OPEN_EXR",
"format.color_mode": "RGB",
},
)
+ default_file_output_node = (
+ file_output_node_exr if saving_ground_truth else file_output_node_png
+ )
file_slot_list = []
viewlayer = bpy.context.scene.view_layers["ViewLayer"]
render_layers = nw.new_node(Nodes.RenderLayers)
@@ -161,6 +183,13 @@ def configure_compositor_output(
setattr(viewlayer, f"use_pass_{viewlayer_pass}", True)
else:
setattr(viewlayer.cycles, f"use_pass_{viewlayer_pass}", True)
+ # must save the material pass index as EXR
+ file_output_node = (
+ default_file_output_node
+ if viewlayer_pass != "material_index"
+ else file_output_node_exr
+ )
+
slot_input = file_output_node.file_slots.new(socket_name)
render_socket = render_layers.outputs[socket_name]
if viewlayer_pass == "vector":
@@ -173,24 +202,15 @@ def configure_compositor_output(
nw.links.new(render_socket, slot_input)
file_slot_list.append(file_output_node.file_slots[slot_input.name])
- slot_input = file_output_node.file_slots["Image"]
+ slot_input = default_file_output_node.file_slots["Image"]
image = image_denoised if image_denoised is not None else image_noisy
- nw.links.new(image, file_output_node.inputs["Image"])
+ nw.links.new(image, default_file_output_node.inputs["Image"])
if saving_ground_truth:
slot_input.path = "UniqueInstances"
else:
- image_exr_output_node = nw.new_node(
- Nodes.OutputFile,
- attrs={
- "base_path": str(frames_folder),
- "format.file_format": "OPEN_EXR",
- "format.color_mode": "RGB",
- },
- )
- rgb_exr_slot_input = file_output_node.file_slots["Image"]
- nw.links.new(image, image_exr_output_node.inputs["Image"])
- file_slot_list.append(image_exr_output_node.file_slots[rgb_exr_slot_input.path])
- file_slot_list.append(file_output_node.file_slots[slot_input.path])
+ nw.links.new(image, file_output_node_exr.inputs["Image"])
+ file_slot_list.append(file_output_node_exr.file_slots[slot_input.path])
+ file_slot_list.append(default_file_output_node.file_slots[slot_input.path])
return file_slot_list
@@ -308,6 +328,22 @@ def postprocess_blendergt_outputs(frames_folder, output_stem):
uniq_inst_path.unlink()
+def postprocess_materialgt_output(frames_folder, output_stem):
+ # Save material segmentation visualization if present
+ ma_seg_dst_path = frames_folder / f"IndexMA{output_stem}.exr"
+ if ma_seg_dst_path.is_file():
+ ma_seg_mask_array = load_seg_mask(ma_seg_dst_path)
+ np.save(
+ ma_seg_dst_path.with_name(f"MaterialSegmentation{output_stem}.npy"),
+ ma_seg_mask_array,
+ )
+ imwrite(
+ ma_seg_dst_path.with_name(f"MaterialSegmentation{output_stem}.png"),
+ colorize_int_array(ma_seg_mask_array),
+ )
+ ma_seg_dst_path.unlink()
+
+
def configure_compositor(
frames_folder: Path,
passes_to_save: list,
@@ -341,7 +377,7 @@ def configure_compositor(
@gin.configurable
def render_image(
- camera_id,
+ camera: bpy.types.Object,
frames_folder,
passes_to_save,
flat_shading=False,
@@ -352,8 +388,6 @@ def render_image(
):
tic = time.time()
- camera_rig_id, subcam_id = camera_id
-
for exclude in excludes:
bpy.data.objects[exclude].hide_render = True
@@ -363,6 +397,8 @@ def render_image(
tmp_dir.mkdir(exist_ok=True)
bpy.context.scene.render.filepath = f"{tmp_dir}{os.sep}"
+ camrig_id, subcam_id = cam_util.get_id(camera)
+
if flat_shading:
with Timer("Set object indices"):
object_data = set_pass_indices()
@@ -370,7 +406,7 @@ def render_image(
first_frame = bpy.context.scene.frame_start
suffix = get_suffix(
dict(
- cam_rig=camera_rig_id,
+ cam_rig=camrig_id,
resample=0,
frame=first_frame,
subcam=subcam_id,
@@ -380,22 +416,37 @@ def render_image(
with Timer("Flat Shading"):
global_flat_shading()
+ else:
+ segment_materials = "material_index" in (x[0] for x in passes_to_save)
+ if segment_materials:
+ with Timer("Set material indices"):
+ material_data = set_material_pass_indices()
+ json_object = json.dumps(material_data, indent=4)
+ first_frame = bpy.context.scene.frame_start
+ suffix = get_suffix(
+ dict(
+ cam_rig=camrig_id,
+ resample=0,
+ frame=first_frame,
+ subcam=subcam_id,
+ )
+ )
+ (frames_folder / f"Materials{suffix}.json").write_text(json_object)
if not bpy.context.scene.use_nodes:
bpy.context.scene.use_nodes = True
file_slot_nodes = configure_compositor(frames_folder, passes_to_save, flat_shading)
- indices = dict(cam_rig=camera_rig_id, resample=0, subcam=subcam_id)
+ indices = dict(cam_rig=camrig_id, resample=0, subcam=subcam_id)
## Update output names
fileslot_suffix = get_suffix({"frame": "####", **indices})
for file_slot in file_slot_nodes:
file_slot.path = f"{file_slot.path}{fileslot_suffix}"
- camera = cam_util.get_camera(camera_rig_id, subcam_id)
if use_dof == "IF_TARGET_SET":
use_dof = camera.data.dof.focus_object is not None
- if use_dof is not None:
+ elif use_dof is not None:
camera.data.dof.use_dof = use_dof
camera.data.dof.aperture_fstop = dof_aperture_fstop
@@ -418,10 +469,13 @@ def render_image(
postprocess_blendergt_outputs(frames_folder, suffix)
else:
cam_util.save_camera_parameters(
- camera_ids=cam_util.get_cameras_ids(),
+ camera,
output_folder=frames_folder,
frame=frame,
)
+ bpy.context.scene.frame_set(frame)
+ suffix = get_suffix(dict(frame=frame, **indices))
+ postprocess_materialgt_output(frames_folder, suffix)
for file in tmp_dir.glob("*.png"):
file.unlink()
diff --git a/infinigen/core/util/camera.py b/infinigen/core/util/camera.py
index beb04dd74..fe58f8da7 100644
--- a/infinigen/core/util/camera.py
+++ b/infinigen/core/util/camera.py
@@ -3,15 +3,14 @@
# Authors: Lahav Lipson, Lingjie Mei
+import logging
import bpy
import bpy_extras
import numpy as np
from mathutils import Matrix, Vector
-from tqdm import trange
-from infinigen.core.util import blender as butil
-from infinigen.core.util.math import dehomogenize, homogenize
+logger = logging.getLogger(__name__)
# ---------------------------------------------------------------
# 3x4 P matrix from Blender camera
@@ -133,48 +132,6 @@ def project_by_object_utils(cam, point):
return Vector((co_2d.x * render_size[0], render_size[1] - co_2d.y * render_size[1]))
-def compute_vis_dists(points, cam):
- projmat, K, RT = map(np.array, get_3x4_P_matrix_from_blender(cam))
- proj = points @ projmat.T
- uv, d = dehomogenize(proj), proj[:, -1]
-
- clamped_uv = np.clip(uv, [0, 0], butil.get_camera_res())
- clamped_d = np.maximum(d, 0)
-
- RT_4x4_inv = np.array(Matrix(RT).to_4x4().inverted())
- clipped_pos = (
- homogenize((homogenize(clamped_uv) * clamped_d[:, None]) @ np.linalg.inv(K).T)
- @ RT_4x4_inv.T
- )
-
- vis_dist = np.linalg.norm(points[:, :-1] - clipped_pos[:, :-1], axis=-1)
-
- return d, vis_dist
-
-
-def min_dists_from_cam_trajectory(points, cam, start=None, end=None, verbose=False):
- assert len(points.shape) == 2 and points.shape[-1] == 3
- assert cam.type == "CAMERA"
-
- if start is None:
- start = bpy.context.scene.frame_start
- if end is None:
- end = bpy.context.scene.frame_end
-
- points = homogenize(points)
- min_dists = np.full(len(points), 1e7)
- min_vis_dists = np.full(len(points), 1e7)
-
- rangeiter = trange if verbose else range
- for i in rangeiter(start, end + 1):
- bpy.context.scene.frame_set(i)
- dists, vis_dists = compute_vis_dists(points, cam)
- min_dists = np.minimum(dists, min_dists)
- min_vis_dists = np.minimum(vis_dists, min_vis_dists)
-
- return min_dists, min_vis_dists
-
-
def points_inview(bbox, camera):
proj = np.array(get_3x4_P_matrix_from_blender(camera)[0])
x, y, z = proj @ np.concatenate([bbox, np.ones((len(bbox), 1))], -1).T
diff --git a/infinigen/datagen/configs/data_schema/monocular.gin b/infinigen/datagen/configs/data_schema/monocular.gin
index d74e74718..80655c26a 100644
--- a/infinigen/datagen/configs/data_schema/monocular.gin
+++ b/infinigen/datagen/configs/data_schema/monocular.gin
@@ -1,6 +1,7 @@
iterate_scene_tasks.frame_range=[1,48]
iterate_scene_tasks.render_frame_range=[48,48]
-iterate_scene_tasks.cam_id_ranges = [1,1]
+iterate_scene_tasks.n_camera_rigs = 1
+iterate_scene_tasks.n_subcams = 1
iterate_scene_tasks.global_tasks = [
{'name': 'coarse', 'func': @queue_coarse},
diff --git a/infinigen/datagen/configs/data_schema/monocular_video.gin b/infinigen/datagen/configs/data_schema/monocular_video.gin
index e99d8b9a6..b9d20b0e9 100644
--- a/infinigen/datagen/configs/data_schema/monocular_video.gin
+++ b/infinigen/datagen/configs/data_schema/monocular_video.gin
@@ -1,7 +1,8 @@
iterate_scene_tasks.frame_range = [1, 48]
iterate_scene_tasks.view_block_size = 192
iterate_scene_tasks.cam_block_size = 8
-iterate_scene_tasks.cam_id_ranges = [1, 1]
+iterate_scene_tasks.n_camera_rigs = 1
+iterate_scene_tasks.n_subcams = 1
iterate_scene_tasks.global_tasks = [
{'name': 'coarse', 'func': @queue_coarse},
diff --git a/infinigen/datagen/configs/data_schema/stereo.gin b/infinigen/datagen/configs/data_schema/stereo.gin
index ee28abf51..43bd56ff0 100644
--- a/infinigen/datagen/configs/data_schema/stereo.gin
+++ b/infinigen/datagen/configs/data_schema/stereo.gin
@@ -1,5 +1,6 @@
-iterate_scene_tasks.frame_range=(1, 1)
-iterate_scene_tasks.cam_id_ranges =(1, 2)
+include 'infinigen/datagen/configs/data_schema/monocular.gin'
+iterate_scene_tasks.n_camera_rigs = 1
+iterate_scene_tasks.n_subcams = 2
iterate_scene_tasks.global_tasks = [
{'name': "coarse", 'func': @queue_coarse},
diff --git a/infinigen/datagen/configs/data_schema/stereo_1h_jobs.gin b/infinigen/datagen/configs/data_schema/stereo_1h_jobs.gin
index ca353727f..d160058f2 100644
--- a/infinigen/datagen/configs/data_schema/stereo_1h_jobs.gin
+++ b/infinigen/datagen/configs/data_schema/stereo_1h_jobs.gin
@@ -1,6 +1,7 @@
iterate_scene_tasks.frame_range=(1, 1)
-iterate_scene_tasks.cam_id_ranges =(1, 2)
+iterate_scene_tasks.n_camera_rigs = 1
+iterate_scene_tasks.n_subcams = 2
iterate_scene_tasks.global_tasks = [
{'name': "coarse", 'func': @queue_coarse},
diff --git a/infinigen/datagen/configs/data_schema/stereo_video.gin b/infinigen/datagen/configs/data_schema/stereo_video.gin
index 48caec7f5..5793dabe8 100644
--- a/infinigen/datagen/configs/data_schema/stereo_video.gin
+++ b/infinigen/datagen/configs/data_schema/stereo_video.gin
@@ -1,2 +1,3 @@
include 'infinigen/datagen/configs/data_schema/monocular_video.gin'
-iterate_scene_tasks.cam_id_ranges = [1, 2]
+iterate_scene_tasks.n_camera_rigs = 1
+iterate_scene_tasks.n_subcams = 2
diff --git a/infinigen/datagen/job_funcs.py b/infinigen/datagen/job_funcs.py
index 096ea5ce8..0603171ad 100644
--- a/infinigen/datagen/job_funcs.py
+++ b/infinigen/datagen/job_funcs.py
@@ -217,6 +217,7 @@ def queue_populate(
configs,
taskname=None,
input_prefix="fine",
+ exclude_gpus=[],
overrides=[],
input_indices=None,
output_indices=None,
@@ -250,7 +251,14 @@ def queue_populate(
with (folder / "run_pipeline.sh").open("a") as f:
f.write(f"{' '.join(' '.join(cmd).split())}\n\n")
- res = submit_cmd(cmd, folder=folder, name=name, gpus=0, **kwargs)
+ res = submit_cmd(
+ cmd,
+ folder=folder,
+ name=name,
+ gpus=0,
+ slurm_exclude=nodes_with_gpus(*exclude_gpus),
+ **kwargs,
+ )
return res, output_folder
diff --git a/infinigen/datagen/manage_jobs.py b/infinigen/datagen/manage_jobs.py
index fcfc4e780..b6dd4a4c4 100644
--- a/infinigen/datagen/manage_jobs.py
+++ b/infinigen/datagen/manage_jobs.py
@@ -171,9 +171,12 @@ def slurm_submit_cmd(
@gin.configurable
-def local_submit_cmd(cmd, folder, name, use_scheduler=False, **kwargs):
+def local_submit_cmd(
+ cmd, folder, name, use_scheduler=False, passthrough=False, **kwargs
+):
ExecutorClass = ScheduledLocalExecutor if use_scheduler else ImmediateLocalExecutor
- executor = ExecutorClass(folder=(folder / "logs"))
+ log_folder = (folder / "logs") if not passthrough else None
+ executor = ExecutorClass(folder=log_folder)
executor.update_parameters(name=name, **kwargs)
if callable(cmd[0]):
func, *arg = cmd
@@ -584,7 +587,7 @@ def inflight(s):
)
if max_stuck_at_task is not None and stuck_at_next >= max_stuck_at_task:
- logging.info(
+ logging.debug(
f"{seed} - Not launching due to {stuck_at_next=} >"
f" {max_stuck_at_task} for {started_if_launch=}"
)
@@ -599,12 +602,12 @@ def inflight(s):
queued_key = (JobState.Queued, taskname.split("_")[0])
queued = state_counts.get(queued_key, 0)
if max_queued_task is not None and queued >= max_queued_task:
- logging.info(
+ logging.debug(
f"{seed} - Not launching due to {queued=} > {max_queued_task} for {taskname}"
)
continue
if max_queued_total is not None and total_queued >= max_queued_total:
- logging.info(
+ logging.debug(
f"{seed} - Not launching due to {total_queued=} > {max_queued_total} for {taskname}"
)
continue
@@ -640,21 +643,13 @@ def compute_control_state(args, totals, elapsed, num_concurrent):
return control_state
-def record_states(stats, totals, control_state):
- pretty_stats = copy(stats)
- pretty_stats.update({f"control_state/{k}": v for k, v in control_state.items()})
- pretty_stats.update({f"{k}/total": v for k, v in totals.items()})
-
- if wandb is not None:
- wandb.log(pretty_stats)
- print("=" * 60)
- for k, v in sorted(pretty_stats.items()):
- print(f"{k.ljust(30)} : {v}")
- print("-" * 60)
-
-
@gin.configurable
-def manage_datagen_jobs(all_scenes, elapsed, num_concurrent, disk_sleep_threshold=0.95):
+def manage_datagen_jobs(
+ all_scenes: list[dict],
+ elapsed: float,
+ num_concurrent: int,
+ disk_sleep_threshold=0.95,
+):
if LocalScheduleHandler._inst is not None:
sys.path = ORIG_SYS_PATH # hacky workaround because bpy module breaks with multiprocessing
LocalScheduleHandler.instance().poll()
@@ -671,7 +666,6 @@ def manage_datagen_jobs(all_scenes, elapsed, num_concurrent, disk_sleep_threshol
) # may be less due to jobs_to_launch optional kwargs, or running out of num_jobs
pd.DataFrame.from_records(all_scenes).to_csv(args.output_folder / "scenes_db.csv")
- record_states(stats, totals, control_state)
# Dont launch new scenes if disk is getting full
if control_state["disk_usage"] > disk_sleep_threshold:
@@ -684,12 +678,18 @@ def manage_datagen_jobs(all_scenes, elapsed, num_concurrent, disk_sleep_threshol
wait_duration=3 * 60 * 60,
)
time.sleep(60)
- return
+ return {}
for scene, taskname, queue_func in new_jobs:
logger.info(f"{scene['seed']} - running {taskname}")
run_task(queue_func, args.output_folder / str(scene["seed"]), scene, taskname)
+ log_stats = copy(stats)
+ log_stats.update({f"control_state/{k}": v for k, v in control_state.items()})
+ log_stats.update({f"{k}/total": v for k, v in totals.items()})
+
+ return log_stats
+
@gin.configurable
def main(args, shuffle=True, wandb_project="render", upload_commandfile_method=None):
@@ -718,10 +718,16 @@ def main(args, shuffle=True, wandb_project="render", upload_commandfile_method=N
mode=args.wandb_mode,
)
+ filehandler = logging.FileHandler(str(args.output_folder / "jobs.log"))
+ filehandler.setLevel(logging.INFO)
+
+ streamhandler = logging.StreamHandler()
+ streamhandler.setLevel(args.loglevel)
+
logging.basicConfig(
- filename=str(args.output_folder / "jobs.log"),
level=args.loglevel,
format="[%(asctime)s]: %(message)s",
+ handlers=[filehandler, streamhandler],
)
print(f"Using {get_slurm_banned_nodes()=}")
@@ -734,10 +740,25 @@ def main(args, shuffle=True, wandb_project="render", upload_commandfile_method=N
start_time = datetime.now()
while any(j["all_done"] == SceneState.NotDone for j in all_scenes):
now = datetime.now()
- print(
- f'{args.output_folder} {start_time.strftime("%m/%d %I:%M%p")} -> {now.strftime("%m/%d %I:%M%p")}'
+
+ if args.print_stats:
+ print(
+ f'{args.output_folder} {start_time.strftime("%m/%d %I:%M%p")} -> {now.strftime("%m/%d %I:%M%p")}'
+ )
+
+ log_stats = manage_datagen_jobs(
+ all_scenes, elapsed=(now - start_time).total_seconds()
)
- manage_datagen_jobs(all_scenes, elapsed=(now - start_time).total_seconds())
+
+ if wandb is not None:
+ wandb.log(log_stats)
+
+ if args.print_stats:
+ print("=" * 60)
+ for k, v in sorted(log_stats.items()):
+ print(f"{k.ljust(30)} : {v}")
+ print("-" * 60)
+
time.sleep(2)
any_crashed = any(j.get("any_fatal_crash", False) for j in all_scenes)
@@ -839,11 +860,12 @@ def main(args, shuffle=True, wandb_project="render", upload_commandfile_method=N
action="store_const",
dest="loglevel",
const=logging.DEBUG,
- default=logging.INFO,
+ default=logging.WARNING,
)
parser.add_argument(
"-v", "--verbose", action="store_const", dest="loglevel", const=logging.INFO
)
+ parser.add_argument("--print_stats", type=int, default=1)
args = parser.parse_args()
using_upload = any("upload" in x for x in args.pipeline_configs)
diff --git a/infinigen/datagen/monitor_tasks.py b/infinigen/datagen/monitor_tasks.py
index 8053130b5..cc707790a 100644
--- a/infinigen/datagen/monitor_tasks.py
+++ b/infinigen/datagen/monitor_tasks.py
@@ -154,7 +154,8 @@ def iterate_scene_tasks(
view_dependent_tasks,
camera_dependent_tasks,
frame_range,
- cam_id_ranges,
+ n_camera_rigs,
+ n_subcams,
point_trajectory_src_frame=1,
num_resamples=1,
render_frame_range=None,
@@ -181,11 +182,8 @@ def iterate_scene_tasks(
if cam_block_size is None:
cam_block_size = view_block_size
- if cam_id_ranges[0] <= 0 or cam_id_ranges[1] <= 0:
- raise ValueError(
- f"{cam_id_ranges=} is invalid, both num. rigs and "
- "num subcams must be >= 1 or no work is done"
- )
+ assert n_camera_rigs >= 1
+ assert n_subcams >= 1
assert view_block_size >= 1
assert cam_block_size >= 1
if cam_block_size > view_block_size:
@@ -218,11 +216,13 @@ def iterate_scene_tasks(
view_range = render_frame_range if render_frame_range is not None else frame_range
view_frames = range(view_range[0], view_range[1] + 1, view_block_size)
resamples = range(num_resamples)
- cam_rigs = range(cam_id_ranges[0])
- subcams = range(cam_id_ranges[1])
+ cam_rigs = range(n_camera_rigs)
+ subcams = range(n_subcams)
running_views = 0
for cam_rig, view_frame in itertools.product(cam_rigs, view_frames):
+ logger.debug(f"Checking {seed=} {cam_rig=} {view_frame=}")
+
view_frame_range = [
view_frame,
min(frame_range[1], view_frame + view_block_size - 1),
@@ -240,8 +240,11 @@ def iterate_scene_tasks(
configs=global_configs,
output_indices=view_idxs,
)
+
+ state = JobState.Succeeded
for state, *rest in view_tasks_iter:
yield state, *rest
+
if state not in CONCLUDED_JOBSTATES:
if viewdep_paralell:
running_views += 1
@@ -287,7 +290,7 @@ def iterate_scene_tasks(
input_indices=view_idxs if len(view_dependent_tasks) else None,
output_indices={**camdep_indices, **extra_indices},
)
-
+ state = JobState.Succeeded
for state, *rest in camera_dep_iter:
yield state, *rest
if state not in CONCLUDED_JOBSTATES:
diff --git a/infinigen/datagen/util/submitit_emulator.py b/infinigen/datagen/util/submitit_emulator.py
index b79c1f4b4..13fa95451 100644
--- a/infinigen/datagen/util/submitit_emulator.py
+++ b/infinigen/datagen/util/submitit_emulator.py
@@ -14,6 +14,7 @@
import re
import subprocess
import sys
+from contextlib import nullcontext
from dataclasses import dataclass
from multiprocessing import Process
from pathlib import Path
@@ -69,13 +70,17 @@ def job_wrapper(
func,
inner_args,
inner_kwargs,
- stdout_file: Path,
- stderr_file: Path,
+ stdout_file: Path = None,
+ stderr_file: Path = None,
cuda_devices=None,
):
- with stdout_file.open("w") as stdout, stderr_file.open("w") as stderr:
- sys.stdout = stdout
- sys.stderr = stderr
+ stdout_ctx = stdout_file.open("w") if stdout_file is not None else nullcontext()
+ stderr_ctx = stderr_file.open("w") if stderr_file is not None else nullcontext()
+ with stdout_ctx as stdout, stderr_ctx as stderr:
+ if stdout_file is not None:
+ sys.stdout = stdout
+ if stderr_file is not None:
+ sys.stderr = stderr
if cuda_devices is not None:
os.environ[CUDA_VARNAME] = ",".join([str(i) for i in cuda_devices])
else:
@@ -84,10 +89,16 @@ def job_wrapper(
def launch_local(func, args, kwargs, job_id, log_folder, name, cuda_devices=None):
- stderr_file = log_folder / f"{job_id}_0_log.err"
- stdout_file = log_folder / f"{job_id}_0_log.out"
- with stdout_file.open("w") as f:
- f.write(f"{func} {args}\n")
+ if log_folder is None:
+ # pass input through to stdout if log_folder is None
+ stderr_file = None
+ stdout_file = None
+ print(f"{func} {args}")
+ else:
+ stderr_file = log_folder / f"{job_id}_0_log.err"
+ stdout_file = log_folder / f"{job_id}_0_log.out"
+ with stdout_file.open("w") as f:
+ f.write(f"{func} {args}\n")
kwargs = dict(
func=func,
@@ -104,9 +115,12 @@ def launch_local(func, args, kwargs, job_id, log_folder, name, cuda_devices=None
class ImmediateLocalExecutor:
- def __init__(self, folder: str):
- self.log_folder = Path(folder).resolve()
- self.log_folder.mkdir(exist_ok=True)
+ def __init__(self, folder: str | None):
+ if folder is None:
+ self.log_folder = None
+ else:
+ self.log_folder = Path(folder).resolve()
+ self.log_folder.mkdir(exist_ok=True)
self.parameters = {}
def update_parameters(self, **parameters):
@@ -236,8 +250,11 @@ def attempt_dispatch_job(self, job_rec, available, total, select_gpus="first"):
class ScheduledLocalExecutor:
def __init__(self, folder: str):
- self.log_folder = Path(folder)
- self.log_folder.mkdir(exist_ok=True)
+ if folder is None:
+ self.log_folder = None
+ else:
+ self.log_folder = Path(folder)
+ self.log_folder.mkdir(exist_ok=True)
self.parameters = {}
def update_parameters(self, **parameters):
diff --git a/infinigen/datagen/util/upload_util.py b/infinigen/datagen/util/upload_util.py
index 8941e758b..9d72aa550 100644
--- a/infinigen/datagen/util/upload_util.py
+++ b/infinigen/datagen/util/upload_util.py
@@ -121,6 +121,18 @@ def rclone_upload_file(src_file, dst_folder):
print(f"Uploaded {src_file}")
+def copy_upload_file(src_file, dst_folder, root_dir):
+ assert os.path.exists(src_file), src_file
+ if dst_folder.as_posix().startswith("infinigen/renders"):
+ dst_folder = dst_folder.as_posix().split("/")[-1]
+
+ if not os.path.exists(f"{root_dir}/{dst_folder}"):
+ os.makedirs(f"{root_dir}/{dst_folder}")
+
+ shutil.copy2(src_file, f"{root_dir}/{dst_folder}")
+ print(f"Copy {src_file}")
+
+
def get_commit_hash():
git = shutil.which("git")
if git is None:
@@ -180,6 +192,8 @@ def get_upload_func(method="smbclient"):
return rclone_upload_file
elif method == "smbclient":
return smb_client.upload
+ elif method.startswith("copyfile"):
+ return lambda x, y: copy_upload_file(x, y, root_dir=method.split(":")[-1])
else:
raise ValueError(f"Unrecognized {method=}")
diff --git a/infinigen/terrain/elements/landtiles.py b/infinigen/terrain/elements/landtiles.py
index 15847ab6b..467e405bd 100644
--- a/infinigen/terrain/elements/landtiles.py
+++ b/infinigen/terrain/elements/landtiles.py
@@ -34,6 +34,7 @@ def none_to_0(x):
return 0
return x
+
@gin.configurable
def get_land_process(land_processes, land_process_probs=None, snowfall_enabled=True):
if not isinstance(land_processes, list):
@@ -53,6 +54,7 @@ def get_land_process(land_processes, land_process_probs=None, snowfall_enabled=T
return None
return rg(("choice", land_processes_, land_process_probs_))
+
@gin.configurable
class LandTiles(Element):
name = ElementNames.LandTiles
diff --git a/infinigen/tools/process_mvs_data.py b/infinigen/tools/process_mvs_data.py
new file mode 100644
index 000000000..4d4e2b18e
--- /dev/null
+++ b/infinigen/tools/process_mvs_data.py
@@ -0,0 +1,198 @@
+# Copyright (C) 2024, Princeton University.
+# This source code is licensed under the BSD 3-Clause license found in the LICENSE file in the root directory of this source tree.
+
+# Authors: Zeyu Ma
+
+import argparse
+import os
+import shutil
+from pathlib import Path
+
+import cv2
+import numpy as np
+import submitit
+import torch
+import torch.nn.functional as F
+from tqdm import tqdm
+
+from infinigen.tools.suffixes import parse_suffix
+
+
+# these functions till check_cycle_consistency are from https://github.com/princeton-vl/SEA-RAFT
+def transform(T, p):
+ assert T.shape == (4, 4)
+ return np.einsum("H W j, i j -> H W i", p, T[:3, :3]) + T[:3, 3]
+
+
+def from_homog(x):
+ return x[..., :-1] / x[..., [-1]]
+
+
+def coords_grid(batch, ht, wd, device):
+ coords = torch.meshgrid(
+ torch.arange(ht, device=device), torch.arange(wd, device=device)
+ )
+ coords = torch.stack(coords[::-1], dim=0).float()
+ return coords[None].repeat(batch, 1, 1, 1)
+
+
+def reproject(depth1, pose1, pose2, K1, K2):
+ H, W = depth1.shape
+ x, y = np.meshgrid(np.arange(W), np.arange(H), indexing="xy")
+ img_1_coords = np.stack((x, y, np.ones_like(x)), axis=-1).astype(np.float64)
+ cam1_coords = np.einsum(
+ "H W, H W j, i j -> H W i", depth1, img_1_coords, np.linalg.inv(K1)
+ )
+ rel_pose = np.linalg.inv(pose2) @ pose1
+ cam2_coords = transform(rel_pose, cam1_coords)
+ return from_homog(np.einsum("H W j, i j -> H W i", cam2_coords, K2))
+
+
+def induced_flow(depth0, depth1, data):
+ H, W = depth0.shape
+ coords1 = reproject(depth0, data["T0"], data["T1"], data["K0"], data["K1"])
+
+ x, y = np.meshgrid(np.arange(W), np.arange(H), indexing="xy")
+ coords0 = np.stack([x, y], axis=-1)
+ flow_01 = coords1 - coords0
+
+ H, W = depth1.shape
+ coords1 = reproject(depth1, data["T1"], data["T0"], data["K1"], data["K0"])
+ x, y = np.meshgrid(np.arange(W), np.arange(H), indexing="xy")
+ coords0 = np.stack([x, y], axis=-1)
+ flow_10 = coords1 - coords0
+
+ return flow_01, flow_10
+
+
+def bilinear_sampler(img, coords, mode="bilinear", mask=False):
+ """Wrapper for grid_sample, uses pixel coordinates"""
+ H, W = img.shape[-2:]
+ xgrid, ygrid = coords.split([1, 1], dim=-1)
+ xgrid = 2 * xgrid / (W - 1) - 1
+ ygrid = 2 * ygrid / (H - 1) - 1
+
+ grid = torch.cat([xgrid, ygrid], dim=-1)
+ img = F.grid_sample(img, grid, align_corners=True)
+
+ if mask:
+ mask = (xgrid > -1) & (ygrid > -1) & (xgrid < 1) & (ygrid < 1)
+ return img, mask.float()
+
+ return img
+
+
+def check_cycle_consistency(flow_01, flow_10, threshold=1):
+ flow_01 = torch.from_numpy(flow_01).permute(2, 0, 1)[None]
+ flow_10 = torch.from_numpy(flow_10).permute(2, 0, 1)[None]
+ H, W = flow_01.shape[-2:]
+ coords = coords_grid(1, H, W, flow_01.device)
+ coords1 = coords + flow_01
+ flow_reprojected = bilinear_sampler(flow_10, coords1.permute(0, 2, 3, 1))
+ cycle = flow_reprojected + flow_01
+ cycle = torch.norm(cycle, dim=1)
+ mask = (cycle < threshold).float()
+ return mask[0].numpy()
+
+
+def compute_covisibility(depth0, depth1, camview0, camview1):
+ data = {}
+ data["K0"] = camview0["K"]
+ data["K1"] = camview1["K"]
+ data["T0"] = camview0["T"]
+ data["T1"] = camview1["T"]
+ flow_01, flow_10 = induced_flow(depth0, depth1, data)
+ mask = check_cycle_consistency(flow_01, flow_10)
+ return mask.mean()
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--source_folder", type=Path, default=None)
+ parser.add_argument("--target_folder", type=Path)
+ parser.add_argument("--postprocess_only", type=int, default=False)
+ args = parser.parse_args()
+
+ source_folder = args.source_folder
+ target_folder = args.target_folder
+
+ if not args.postprocess_only:
+ scenes = [
+ x for x in os.listdir(source_folder) if os.path.isdir(source_folder / x)
+ ]
+ for scene in tqdm(scenes):
+ image_dir = source_folder / scene / "frames/Image/camera_0"
+ if not os.path.exists(image_dir):
+ continue
+ images = [x for x in os.listdir(image_dir) if x.endswith(".png")]
+ for image in images:
+ im = cv2.imread(image_dir / image)
+ if im.mean() < 20:
+ continue
+ camera_path = (
+ source_folder
+ / scene
+ / f"frames/camview/camera_0/camview{image[5:-4]}.npz"
+ )
+ depth_path = (
+ source_folder
+ / scene
+ / f"frames/Depth/camera_0/Depth{image[5:-4]}.npy"
+ )
+ if not os.path.exists(camera_path):
+ continue
+ if not os.path.exists(depth_path):
+ continue
+ (target_folder / scene / "images").mkdir(parents=True, exist_ok=True)
+ (target_folder / scene / "cameras").mkdir(parents=True, exist_ok=True)
+ (target_folder / scene / "depths").mkdir(parents=True, exist_ok=True)
+ cam_id = parse_suffix(image)["cam_rig"]
+ shutil.copy(
+ image_dir / image,
+ target_folder / scene / "images" / f"{cam_id:04d}.png",
+ )
+ shutil.copy(
+ camera_path, target_folder / scene / "cameras" / f"{cam_id:04d}.npz"
+ )
+ shutil.copy(
+ depth_path, target_folder / scene / "depths" / f"{cam_id:04d}.npy"
+ )
+
+ scenes = os.listdir(target_folder)
+
+ def worker(scene):
+ cam_ids = [
+ x[:-4]
+ for x in os.listdir(target_folder / scene / "images")
+ if x.endswith(".png")
+ ]
+ with open(target_folder / scene / "pairs.txt", "w") as f:
+ for cam_id0 in cam_ids:
+ f.write(f"{cam_id0} ")
+ depth_path = target_folder / scene / f"depths/{cam_id0}.npy"
+ camera_path = target_folder / scene / f"cameras/{cam_id0}.npz"
+ depth0 = np.load(depth_path)
+ camview0 = np.load(camera_path)
+ for cam_id1 in cam_ids:
+ if cam_id1 == cam_id0:
+ continue
+ depth_path = target_folder / scene / f"depths/{cam_id1}.npy"
+ camera_path = target_folder / scene / f"cameras/{cam_id1}.npz"
+ depth1 = np.load(depth_path)
+ camview1 = np.load(camera_path)
+ cov = compute_covisibility(depth0, depth1, camview0, camview1)
+ f.write(f" {cam_id1} {cov}")
+ f.write("\n")
+ thumbnails = []
+ for image in os.listdir(target_folder / scene / "images"):
+ im = cv2.imread(target_folder / scene / "images" / image)
+ H, W = im.shape[:2]
+ thumbnails.append(cv2.resize(im, (W // 10, H // 10)))
+ thumbnails = np.concatenate(thumbnails, 1)
+ cv2.imwrite(target_folder / scene / "thumbnails.png", thumbnails)
+
+ log_folder = "~/sc/logs/%j"
+ executor = submitit.AutoExecutor(folder=log_folder)
+ executor.update_parameters(timeout_min=10, slurm_partition="allcs")
+ for scene in scenes:
+ job = executor.submit(worker, scene)
diff --git a/infinigen_examples/configs_indoor/base_indoors.gin b/infinigen_examples/configs_indoor/base_indoors.gin
index 90fd9ac0c..1cae98769 100644
--- a/infinigen_examples/configs_indoor/base_indoors.gin
+++ b/infinigen_examples/configs_indoor/base_indoors.gin
@@ -13,8 +13,6 @@ SimulatedAnnealingSolver.max_invalid_candidates = 5
RoomConstants.n_stories = 1
-configure_render_cycles.num_samples = 16000
-
animate_cameras.follow_poi_chance=0.0
camera.camera_pose_proposal.altitude = ("clip_gaussian", 1.5, 0.8, 0.5, 2.2)
camera.camera_pose_proposal.pitch = ("clip_gaussian", 90, 15, 60, 95)
@@ -70,4 +68,6 @@ compose_indoors.floating_objs_enabled = False
compose_indoors.num_floating = ('discrete_uniform', 15, 25)
compose_indoors.norm_floating_size = True
compose_indoors.enable_collision_floating = False
-compose_indoors.enable_collision_solved = False
\ No newline at end of file
+compose_indoors.enable_collision_solved = False
+
+configure_cameras.mvs_radius = ("uniform", 1, 2)
diff --git a/infinigen_examples/configs_nature/asset_demo.gin b/infinigen_examples/configs_nature/asset_demo.gin
index c19c8ea99..a157e164f 100644
--- a/infinigen_examples/configs_nature/asset_demo.gin
+++ b/infinigen_examples/configs_nature/asset_demo.gin
@@ -1,7 +1,7 @@
compose_nature.inview_distance = 30
-full/configure_render_cycles.min_samples = 50
-full/configure_render_cycles.num_samples = 300
+configure_render_cycles.min_samples = 50
+configure_render_cycles.num_samples = 300
configure_blender.motion_blur = False
render_image.use_dof = True
diff --git a/infinigen_examples/configs_nature/base.gin b/infinigen_examples/configs_nature/base.gin
index 9f4a58f51..a85931921 100644
--- a/infinigen_examples/configs_nature/base.gin
+++ b/infinigen_examples/configs_nature/base.gin
@@ -50,7 +50,8 @@ full/render_image.passes_to_save = [
['volume_direct', 'VolumeDir'],
['emit', 'Emit'],
['environment', 'Env'],
- ['ambient_occlusion', 'AO']
+ ['ambient_occlusion', 'AO'],
+ ['material_index', 'IndexMA'],
]
flat/render_image.passes_to_save = [
['z', 'Depth'],
@@ -78,7 +79,7 @@ camera.camera_pose_proposal.altitude = ("weighted_choice",
camera.camera_pose_proposal.pitch = ("clip_gaussian", 90, 30, 20, 160)
-# WARNING: Large camera rig translations or rotations require special handling.
+# WARNING: Large camera rig translations or rotations require special handling.
# if your cameras are not all approximately forward facing within a few centimeters, you must either:
# - configure the pipeline to generate assets / terrain for each camera separately, rather than sharing it between the whole rig
# - or, treat your camera rig as multiple camera rigs each with one camera, and implement code to positon them correctly
@@ -86,4 +87,4 @@ camera.spawn_camera_rigs.n_camera_rigs = 1
camera.spawn_camera_rigs.camera_rig_config = [
{'loc': (0, 0, 0), 'rot_euler': (0, 0, 0)},
{'loc': (0.075, 0, 0), 'rot_euler': (0, 0, 0)}
-]
\ No newline at end of file
+]
diff --git a/infinigen_examples/configs_nature/multiview_stereo.gin b/infinigen_examples/configs_nature/multiview_stereo.gin
new file mode 100644
index 000000000..c3ae64227
--- /dev/null
+++ b/infinigen_examples/configs_nature/multiview_stereo.gin
@@ -0,0 +1,9 @@
+compute_base_views.min_candidates_ratio = 1
+fine_terrain.mesher_backend = "OcMesher"
+
+configure_cameras.mvs_setting = True
+compose_nature.camera_selection_ranges_ratio = {}
+compose_nature.camera_selection_tags_ratio = {}
+compose_nature.animate_cameras_enabled=False
+
+compose_nature.inview_distance = 40
\ No newline at end of file
diff --git a/infinigen_examples/configs_nature/performance/dev.gin b/infinigen_examples/configs_nature/performance/dev.gin
index 413ff7e34..b9c555110 100644
--- a/infinigen_examples/configs_nature/performance/dev.gin
+++ b/infinigen_examples/configs_nature/performance/dev.gin
@@ -1,7 +1,7 @@
execute_tasks.generate_resolution = (960, 540)
-full/configure_render_cycles.min_samples = 32
-full/configure_render_cycles.num_samples = 512
+configure_render_cycles.min_samples = 32
+configure_render_cycles.num_samples = 512
OpaqueSphericalMesher.pixels_per_cube = 4
TransparentSphericalMesher.pixels_per_cube = 4
diff --git a/infinigen_examples/configs_nature/performance/simple.gin b/infinigen_examples/configs_nature/performance/simple.gin
index e5ec006cf..fc591221b 100644
--- a/infinigen_examples/configs_nature/performance/simple.gin
+++ b/infinigen_examples/configs_nature/performance/simple.gin
@@ -2,4 +2,4 @@ include 'infinigen_examples/configs_nature/performance/dev.gin'
include 'infinigen_examples/configs_nature/disable_assets/no_creatures.gin'
include 'infinigen_examples/configs_nature/performance/fast_terrain_assets.gin'
run_erosion.n_iters = [1,1]
-full/configure_render_cycles.num_samples = 100
\ No newline at end of file
+configure_render_cycles.num_samples = 100
\ No newline at end of file
diff --git a/infinigen_examples/generate_asset_demo.py b/infinigen_examples/generate_asset_demo.py
index bba4504aa..49afcd0bb 100644
--- a/infinigen_examples/generate_asset_demo.py
+++ b/infinigen_examples/generate_asset_demo.py
@@ -112,7 +112,7 @@ def compose_scene(
kole_clouds.add_kole_clouds()
camera_rigs = cam_util.spawn_camera_rigs()
- cam = cam_util.get_camera(0, 0)
+ cam = camera_rigs[0].children[0]
# find a flat spot on the terrain to do the demo\
terrain = Terrain(
@@ -172,7 +172,7 @@ def compose_scene(
# apply a procedural backdrop on all visible parts of the terrain
terrain_inview, *_ = split_inview(
- terrain_mesh, cam=cam, dist_max=params["inview_distance"], vis_margin=2
+ terrain_mesh, cameras=[cam], dist_max=params["inview_distance"], vis_margin=2
)
if background is None:
pass
diff --git a/infinigen_examples/generate_individual_assets.py b/infinigen_examples/generate_individual_assets.py
index d21c7608a..03fa7ea55 100644
--- a/infinigen_examples/generate_individual_assets.py
+++ b/infinigen_examples/generate_individual_assets.py
@@ -485,8 +485,7 @@ def main(args):
if len(factories) == 1 and factories[0].endswith(".txt"):
factories = [
- f.split(".")[-1]
- for f in load_txt_list(factories[0], skip_sharp=False)
+ f.split(".")[-1] for f in load_txt_list(factories[0], skip_sharp=False)
]
else:
assert not any(f.endswith(".txt") for f in factories)
diff --git a/infinigen_examples/generate_indoors.py b/infinigen_examples/generate_indoors.py
index 129d9a8c3..dbf20108b 100644
--- a/infinigen_examples/generate_indoors.py
+++ b/infinigen_examples/generate_indoors.py
@@ -252,6 +252,7 @@ def pose_cameras():
camera_rigs,
scene_preprocessed=scene_preprocessed,
init_surfaces=solved_floor_surface,
+ nonroom_objs=nonroom_objs,
)
butil.delete(solved_floor_surface)
return scene_preprocessed
@@ -309,7 +310,7 @@ def place_floating():
placer = FloatingObjectPlacement(
generators=facs,
- camera=cam_util.get_camera(0, 0),
+ camera=camera_rigs[0].children[0],
background_objs=list(pholder_cutters.objects) + list(pholder_rooms.objects),
collision_objs=list(pholder_objs.objects),
)
@@ -388,8 +389,6 @@ def place_floating():
# state.print()
state.to_json(output_folder / "solve_state.json")
- cam = cam_util.get_camera(0, 0)
-
def turn_off_lights():
for o in bpy.data.objects:
if o.type == "LIGHT" and not o.data.cycles.is_portal:
@@ -430,7 +429,7 @@ def invisible_room_ceilings():
create_outdoor_backdrop,
terrain,
house_bbox=house_bbox,
- cam=cam,
+ cameras=[rig.children[0] for rig in camera_rigs],
p=p,
params=overrides,
use_chance=False,
diff --git a/infinigen_examples/generate_nature.py b/infinigen_examples/generate_nature.py
index 8f689dc61..d04a1b790 100644
--- a/infinigen_examples/generate_nature.py
+++ b/infinigen_examples/generate_nature.py
@@ -286,24 +286,32 @@ def camera_preprocess():
p.run_stage(
"pose_cameras",
lambda: cam_util.configure_cameras(
- camera_rigs, scene_preprocessed, init_bounding_box=bbox
+ camera_rigs,
+ scene_preprocessed,
+ init_bounding_box=bbox,
+ terrain_mesh=terrain_mesh,
),
use_chance=False,
)
- cam = cam_util.get_camera(0, 0)
+ primary_cams = [rig.children[0] for rig in camera_rigs]
- p.run_stage("lighting", lighting.sky_lighting.add_lighting, cam, use_chance=False)
+ p.run_stage(
+ "lighting",
+ lighting.sky_lighting.add_lighting,
+ primary_cams[0],
+ use_chance=False,
+ )
# determine a small area of the terrain for the creatures to run around on
# must happen before camera is animated, as camera may want to follow them around
terrain_center, *_ = split_in_view.split_inview(
terrain_mesh,
- cam=cam,
- start=0,
- end=0,
- outofview=False,
- vis_margin=5,
+ primary_cams,
dist_max=params["center_distance"],
+ vis_margin=5,
+ frame_start=0,
+ frame_end=0,
+ outofview=False,
hide_render=True,
suffix="center",
)
@@ -364,10 +372,9 @@ def animate_cameras():
with logging_util.Timer("Compute coarse terrain frustrums"):
terrain_inview, *_ = split_in_view.split_inview(
terrain_mesh,
+ primary_cams,
verbose=True,
outofview=False,
- print_areas=True,
- cam=cam,
vis_margin=2,
dist_max=params["inview_distance"],
hide_render=True,
@@ -375,10 +382,9 @@ def animate_cameras():
)
terrain_near, *_ = split_in_view.split_inview(
terrain_mesh,
+ primary_cams,
verbose=True,
outofview=False,
- print_areas=True,
- cam=cam,
vis_margin=2,
dist_max=params["near_distance"],
hide_render=True,
@@ -731,9 +737,12 @@ def add_tilted_river():
@gin.configurable
-def populate_scene(output_folder, scene_seed, **params):
+def populate_scene(
+ output_folder: Path, scene_seed: int, camera_rigs: list[bpy.types.Object], **params
+):
p = RandomStageExecutor(scene_seed, output_folder, params)
- camera = [cam_util.get_camera(i, j) for i, j in cam_util.get_cameras_ids()]
+
+ primary_cams = [rig.children[0] for rig in camera_rigs]
season = p.run_stage(
"choose_season", trees.random_season, use_chance=False, default=[]
@@ -747,7 +756,7 @@ def populate_scene(output_folder, scene_seed, **params):
use_chance=False,
default=[],
fn=lambda: placement.populate_all(
- trees.TreeFactory, camera, season=season, vis_cull=4
+ trees.TreeFactory, primary_cams, season=season, vis_cull=4
),
) # ,
# meshing_camera=camera, adapt_mesh_method='subdivide', cam_meshing_max_dist=8))
@@ -755,40 +764,44 @@ def populate_scene(output_folder, scene_seed, **params):
"populate_boulders",
use_chance=False,
default=[],
- fn=lambda: placement.populate_all(rocks.BoulderFactory, camera, vis_cull=3),
+ fn=lambda: placement.populate_all(
+ rocks.BoulderFactory, primary_cams, vis_cull=3
+ ),
) # ,
# meshing_camera=camera, adapt_mesh_method='subdivide', cam_meshing_max_dist=8))
populated["bushes"] = p.run_stage(
"populate_bushes",
use_chance=False,
fn=lambda: placement.populate_all(
- trees.BushFactory, camera, vis_cull=1, adapt_mesh_method="subdivide"
+ trees.BushFactory, primary_cams, vis_cull=1, adapt_mesh_method="subdivide"
),
)
p.run_stage(
"populate_kelp",
use_chance=False,
fn=lambda: placement.populate_all(
- monocot.KelpMonocotFactory, camera, vis_cull=5
+ monocot.KelpMonocotFactory, primary_cams, vis_cull=5
),
)
populated["cactus"] = p.run_stage(
"populate_cactus",
use_chance=False,
- fn=lambda: placement.populate_all(cactus.CactusFactory, camera, vis_cull=6),
+ fn=lambda: placement.populate_all(
+ cactus.CactusFactory, primary_cams, vis_cull=6
+ ),
)
p.run_stage(
"populate_clouds",
use_chance=False,
fn=lambda: placement.populate_all(
- cloud.CloudFactory, camera, dist_cull=None, vis_cull=None
+ cloud.CloudFactory, primary_cams, dist_cull=None, vis_cull=None
),
)
p.run_stage(
"populate_glowing_rocks",
use_chance=False,
fn=lambda: placement.populate_all(
- rocks.GlowingRocksFactory, camera, dist_cull=None, vis_cull=None
+ rocks.GlowingRocksFactory, primary_cams, dist_cull=None, vis_cull=None
),
)
@@ -798,7 +811,7 @@ def populate_scene(output_folder, scene_seed, **params):
default=[],
fn=lambda: placement.populate_all(
fluid.CachedTreeFactory,
- camera,
+ primary_cams,
season=season,
vis_cull=4,
dist_cull=70,
@@ -811,7 +824,7 @@ def populate_scene(output_folder, scene_seed, **params):
default=[],
fn=lambda: placement.populate_all(
fluid.CachedBoulderFactory,
- camera,
+ primary_cams,
vis_cull=3,
dist_cull=70,
cache_system=fire_cache_system,
@@ -822,7 +835,7 @@ def populate_scene(output_folder, scene_seed, **params):
use_chance=False,
fn=lambda: placement.populate_all(
fluid.CachedBushFactory,
- camera,
+ primary_cams,
vis_cull=1,
adapt_mesh_method="subdivide",
cache_system=fire_cache_system,
@@ -833,7 +846,7 @@ def populate_scene(output_folder, scene_seed, **params):
use_chance=False,
fn=lambda: placement.populate_all(
fluid.CachedCactusFactory,
- camera,
+ primary_cams,
vis_cull=6,
cache_system=fire_cache_system,
),
@@ -906,7 +919,7 @@ def apply_snow_layer(surface_cls):
p.run_stage(
f"populate_{k}",
use_chance=False,
- fn=lambda: placement.populate_all(fac, camera=None),
+ fn=lambda: placement.populate_all(fac, cameras=None),
)
fire_warmup = params.get("fire_warmup", 50)
diff --git a/infinigen_examples/util/generate_indoors_util.py b/infinigen_examples/util/generate_indoors_util.py
index 2e60daf4f..c85717927 100644
--- a/infinigen_examples/util/generate_indoors_util.py
+++ b/infinigen_examples/util/generate_indoors_util.py
@@ -44,7 +44,7 @@ def within_bbox_2d(verts, bbox):
def create_outdoor_backdrop(
terrain: Terrain,
house_bbox: tuple,
- cam,
+ cameras: list[bpy.types.Object],
p: pipeline.RandomStageExecutor,
params: dict,
):
@@ -86,10 +86,9 @@ def create_outdoor_backdrop(
terrain_inview, *_ = split_in_view.split_inview(
main_terrain,
+ cameras,
verbose=True,
outofview=False,
- print_areas=True,
- cam=cam,
vis_margin=2,
dist_max=params["near_distance"],
hide_render=True,